- _C1.npy`. To execute the inference, run the `predict.py` script with the input file list as an argument.",
+ "type": "comment"
+ },
+ "4628": {
+ "file_id": 393,
+ "content": " --config configs/segmentation/ms_tcn/ms_tcn_gtea.yaml \\\n --model_file inference/MSTCN/MSTCN.pdmodel \\\n --params_file inference/MSTCN/MSTCN.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nexample of logs:\n```bash\nresult write in : ./inference/infer_results/S1_Cheese_C1.txt\nresult write in : ./inference/infer_results/S1_CofHoney_C1.txt\nresult write in : ./inference/infer_results/S1_Coffee_C1.txt\nresult write in : ./inference/infer_results/S1_Hotdog_C1.txt\nresult write in : ./inference/infer_results/S1_Pealate_C1.txt\nresult write in : ./inference/infer_results/S1_Peanut_C1.txt\nresult write in : ./inference/infer_results/S1_Tea_C1.txt\n```\n## Reference\n- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://arxiv.org/pdf/1903.01945.pdf), Y. Abu Farha and J. Gall.",
+ "type": "code",
+ "location": "/english_documents/model_zoo/segmentation/mstcn.md:109-130"
+ },
+ "4629": {
+ "file_id": 393,
+ "content": "The code is specifying the configuration file, model file, and parameter file for running the MSTCN (Multi-Stage Temporal Convolutional Network) segmentation model. It also sets the GPU usage to True and TensorRT to False.\nExample logs show the results being written into respective text files in the inference/infer_results folder.",
+ "type": "comment"
+ },
+ "4630": {
+ "file_id": 394,
+ "content": "/english_documents/quick_start.md",
+ "type": "filepath"
+ },
+ "4631": {
+ "file_id": 394,
+ "content": "PaddleVideo Quick Start guide covers installation, usage details, and action recognition model for classifying video files. Highlights top-5 classes with high confidence using an example command, also suggesting alternative OpenCV installation method.",
+ "type": "summary"
+ },
+ "4632": {
+ "file_id": 394,
+ "content": "English | [简体中文](../zh-CN/quick_start.md)\n# PaddleVide Quick Start\n- [1. Installation](#1)\n - [1.1 Install PaddlePaddle](#11)\n - [1.2 Install PaddleVideo Whl Package](#12)\n- [2. Easy-to-Use](#2)\n - [2.1 Use by Command Line](#21)\n - [2.2 Use by Python Code](#22)\n- [3. Arguments description](#3)\n- [4.QA](#4)\n## 1. Installation\n\n### 1.1 Install PaddlePaddle\n- If you have CUDA 9 or CUDA 10 installed on your machine, please run the following command to install\n ```bash\n python3.7 -m pip install paddlepaddle-gpu -i https://mirror.baidu.com/pypi/simple\n ```\n- If you have no available GPU on your machine, please run the following command to install the CPU version\n ```bash\n python3.7 -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple\n ```\nFor more software version requirements, please refer to the instructions in [Installation Document](https://www.paddlepaddle.org.cn/install/quick) for operation.\n\n### 1.2 Install PaddleVideo Whl Package\n- option1: use pypi（recommand）",
+ "type": "code",
+ "location": "/english_documents/quick_start.md:1-36"
+ },
+ "4633": {
+ "file_id": 394,
+ "content": "Code is an English version of the Quick Start guide for PaddleVideo. It provides information on how to install the necessary packages, use PaddleVideo by command line and Python code, describes arguments, and answers frequently asked questions. The code also includes instructions for installing PaddlePaddle with or without a GPU, as well as the option to install the PaddleVideo Whl Package from pypi.",
+ "type": "comment"
+ },
+ "4634": {
+ "file_id": 394,
+ "content": "```bash\npip3.7 install ppvideo==2.3.0\n```\n- option2: build and install locally\n```bash\npython3.7 setup.py bdist_wheel\npython3.7 -m pip install dist/ppvideo-2.3.0-py3-none-any.whl\n```\n## 2. Easy-to-Use\n\n### 2.1 Use by Command Line\nRun shell command：\n```bash\nppvideo --model_name='ppTSM_v2' --use_gpu=False --video_file='data/example.avi'\n```\n- This command use `PP-TSM_v2` model to infer `data/example.avi` file in `CPU`.\n- The length of the example video is about 10s. When inference, the video is first divided into 16 segments according to the time axis, then extract one frame from each segment. Finally all frames are combined and feeded into the network.\nResults：\n```\nCurrent video file: data/example.avi\n top-1 classes: [5]\n top-1 scores: [1.]\n top-1 label names: ['archery']\n```\nAs you can see, use `PP-TSM_v2` trained on Kinetics-400 to predict `data/example.avi` video，top1 prediction class_id is `5`, scores is `1.0`, class name is `archery`.\n\n### 2.2 Use by Python Code",
+ "type": "code",
+ "location": "/english_documents/quick_start.md:38-76"
+ },
+ "4635": {
+ "file_id": 394,
+ "content": "Install package using pip:\n```bash\npip3.7 install ppvideo==2.3.0\n```\nAlternatively, build and install locally:\n```bash\npython3.7 setup.py bdist_wheel\npython3.7 -m pip install dist/ppvideo-2.3.0-py3-none-any.whl\n```\nCommand to use by command line:\n```bash\nppvideo --model_name='ppTSM_v2' --use_gpu=False --video_file='data/example.avi'\n```\nThis command uses PP-TSM_v2 model on CPU for inference on data/example.avi file, divided into 16 segments and frames combined before feeding into network. Results show top-1 prediction class_id as 5, scores as 1.0, and class name as 'archery'.",
+ "type": "comment"
+ },
+ "4636": {
+ "file_id": 394,
+ "content": "Run python code：\n```python\nfrom ppvideo import PaddleVideo\nclas = PaddleVideo(model_name='ppTSM_v2', use_gpu=False)\nvideo_file='data/example.avi'\nclas.predict(video_file)\n```\n- This code use `PP-TSM_v2` model to infer `data/example.avi` file in `CPU`.\nResults:\n```\nCurrent video file: data/example.avi\n top-1 classes: [5]\n top-1 scores: [1.]\n top-1 label names: ['archery']\n```\nAs you can see, use `PP-TSM_v2` trained on Kinetics-400 to predict `data/example.avi` video，top1 prediction class_id is `5`, scores is `1.0`, class name is `archery`.\n\n## 3. Arguments description\n| name | type | description |\n| :---: | :---: | :--- |\n| model_name | str | optional, model name, `'ppTSM'` or `'ppTSM_v2'`. If None, please specify the path of your inference model by args `model_file` and `params_file`. |\n| video_file | str | required, Video file path, supported format: single video file path, or folder containing multiple videos. |\n| use_gpu | bool | whether to use GPU，default True。 |\n| nu",
+ "type": "code",
+ "location": "/english_documents/quick_start.md:78-107"
+ },
+ "4637": {
+ "file_id": 394,
+ "content": "This code uses the PaddleVideo library with the PP-TSM_v2 model for video inference on a CPU. It predicts the top-1 class, score, and label name of the provided 'data/example.avi' video file. The model is trained on Kinetics-400 dataset. Arguments include model name (PP-TSM or PP-TSM_v2), video file path, GPU usage, and other optional parameters.",
+ "type": "comment"
+ },
+ "4638": {
+ "file_id": 394,
+ "content": "m_seg | int | The number of segments used in the TSM model, which is also the number of frames extracted from the video. 8 for `ppTSM`, 16 for `ppTSM_v2`, default 16. |\n| short_size | int | short size of frame, default 256.|\n| target_size | int | target size of frame, default 224.|\n| model_file | str | optional，inference model(`.pdmodel`)path. |\n| params_file | str | optional, inference modle(`.pdiparams`) path. |\n| batch_size | int | Batch size, default 1.|\n| use_fp16 | bool | whether to use float16，default False.|\n| use_tensorrt | bool| whether to use Tensorrt, default False.|\n| gpu_mem | int | use GPU memory, default 8000.|\n| enable_mkldnn | bool | whether to use MKLDNN, default False.|\n| top_k | int | top_k, default 1. |\n| label_name_path | str | This file consists the relation of class_id and class_name. Default use `data/k400/Kinetics-400_label_list.txt` of Kinetics-400. You can replace it with your own label file. |\ncommand example1：\n```bash\nppvideo --model_name='ppTSM_v2' --num_seg=16 --video_file=\"data/mp4\" --batch_size=2 --top_k=5",
+ "type": "code",
+ "location": "/english_documents/quick_start.md:107-122"
+ },
+ "4639": {
+ "file_id": 394,
+ "content": "The code defines several parameters for the PaddleVideo model including the number of segments, short and target frame sizes, model file paths, batch size, use of float16, TensorRT, MKLDNN, top_k, and label name path. It also provides a command example usage of the model with specified parameters.",
+ "type": "comment"
+ },
+ "4640": {
+ "file_id": 394,
+ "content": "```\nResults：\n```txt\nCurrent video file: data/mp4/example3.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]\n top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\nCurrent video file: data/mp4/example2.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]\n top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\nCurrent video file: data/mp4/example.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]\n top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\nCurrent video file: data/mp4/example1.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]",
+ "type": "code",
+ "location": "/english_documents/quick_start.md:123-142"
+ },
+ "4641": {
+ "file_id": 394,
+ "content": "The code displays the top-5 classes, scores, and label names for five different video files. It shows that the classifier consistently identifies the same top-5 classes with high confidence for each video file, indicating a reliable classification performance.",
+ "type": "comment"
+ },
+ "4642": {
+ "file_id": 394,
+ "content": " top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\n```\ncommand example1：\n```bash\nppvideo --model_name='ppTSM' --num_seg=8 --video_file=\"data/mp4\" --batch_size=2 --top_k=5\n```\n\n## 4. QA\n1. opecv-python Installation maybe slow, you can try:\n```\npython3.7 -m pip install opencv-python==4.2.0.32 -i https://pypi.doubanio.com/simple\n```",
+ "type": "code",
+ "location": "/english_documents/quick_start.md:143-157"
+ },
+ "4643": {
+ "file_id": 394,
+ "content": "The code provides a list of top-5 label names for PaddleVideo's action recognition model. The command example demonstrates how to run the model with specific parameters, such as the model name, number of video segments, input video file, and batch size. Additionally, it suggests an alternative installation method for OpenCV-python if the regular installation is slow.",
+ "type": "comment"
+ },
+ "4644": {
+ "file_id": 395,
+ "content": "/english_documents/tools.md",
+ "type": "filepath"
+ },
+ "4645": {
+ "file_id": 395,
+ "content": "This code provides usage instructions for various tools in PaddleVideo. It shows how to retrieve model parameters, calculate FLOPs, and test an exported model (coming soon). The code examples use Python 3.7 and require specific configuration files.",
+ "type": "summary"
+ },
+ "4646": {
+ "file_id": 395,
+ "content": "[简体中文](../zh-CN/tools.md) | English\n# Tools\nThis page includes the usage of some useful tools in PaddleVideo.\n## Params\nTo get the params of a model.\n```shell\npython3.7 tools/summary.py -c configs/recognization/tsm/tsm.yaml\n```\n## FLOPS\nto print FLOPs.\n```shell\npython3.7 tools/summary.py -c configs/recognization/tsm/tsm.yaml --FLOPs\n```\n## Test the export model coming soon",
+ "type": "code",
+ "location": "/english_documents/tools.md:1-22"
+ },
+ "4647": {
+ "file_id": 395,
+ "content": "This code provides usage instructions for various tools in PaddleVideo. It shows how to retrieve model parameters, calculate FLOPs, and test an exported model (coming soon). The code examples use Python 3.7 and require specific configuration files.",
+ "type": "comment"
+ },
+ "4648": {
+ "file_id": 396,
+ "content": "/english_documents/tutorials/Action Recognition Datasets",
+ "type": "filepath"
+ },
+ "4649": {
+ "file_id": 396,
+ "content": "This code provides a list of useful action recognition datasets along with their respective links for further reference. These datasets are essential for training and evaluating action recognition models, each serving its specific purpose in the field of computer vision.",
+ "type": "summary"
+ },
+ "4650": {
+ "file_id": 396,
+ "content": "Usefull Action Recognition Datasets.\n AVA, https://arxiv.org/abs/1705.08421\n Kinetics, https://arxiv.org/abs/1705.06950\n YouTube-8M, https://arxiv.org/abs/1609.08675\n ActivityNet, http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.html\n Moments in Time, https://arxiv.org/pdf/1801.03150.pdf\n Charades, https://arxiv.org/abs/1604.01753\n EPIC-Kitchens, https://arxiv.org/abs/1804.02748\n THUMOS, https://arxiv.org/abs/1604.06182\n UCF-101, http://crcv.ucf.edu/papers/UCF101_CRCV-TR-12-01.pdf\n HMDB51, http://serre-lab.clps.brown.edu/wp-content/uploads/2012/08/Kuehne_etal_iccv11.pdf",
+ "type": "code",
+ "location": "/english_documents/tutorials/Action Recognition Datasets:1-12"
+ },
+ "4651": {
+ "file_id": 396,
+ "content": "This code provides a list of useful action recognition datasets along with their respective links for further reference. These datasets are essential for training and evaluating action recognition models, each serving its specific purpose in the field of computer vision.",
+ "type": "comment"
+ },
+ "4652": {
+ "file_id": 397,
+ "content": "/english_documents/tutorials/Action Recognition Papers",
+ "type": "filepath"
+ },
+ "4653": {
+ "file_id": 397,
+ "content": "This code provides a list of papers on action recognition and video classification, including TSN, SlowFast Networks, X3D, ECO, 3D ResNet, etc. The paper \"Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors\" presents an efficient method for recognizing actions from video sequences using deep convolutional descriptors and trajectory pooling.",
+ "type": "summary"
+ },
+ "4654": {
+ "file_id": 397,
+ "content": "Useful Papers on Action Recognition and Video Classification.\nTSN: Temporal Segment Networks: Towards Good Practices for Deep Action Recognition, ECCV 2016\nTSM: Temporal Shift Module for Efficient Video Understanding, ICCV 2019\nSlowFast Networks for Video Recognition, ICCV 2019\nNon-local Neural Networks, CVPR 2018\nA Multigrid Method for Efficiently Training Video Models, CVPR2020\nX3D: Progressive Network Expansion for Efficient Video Recognition, CVPR2020\nECO: Efficient Convolutional Network for Online Video Understanding, ECCV 2018\n3D Resnet: Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs, CVPR 2018\nTPN: Temporal Pyramid Network for Action Recognition, CVPR 2020\nEvaNet: Evolving Space-Time Neural Architectures for Videos, ICCV 2019\nRepFlow: Representation Flow for Action Recognition, CVPR 2019\nMARS: Motion-Augmented RGB Stream for Action Recognition, CVPR 2019\nStNet: Local and Global Spatial-Temporal Modeling for Human Action Recognition, AAAI 2019\nAttention Cluster: Purely Attention Based Local Feature Integration for Video Classification",
+ "type": "code",
+ "location": "/english_documents/tutorials/Action Recognition Papers:1-16"
+ },
+ "4655": {
+ "file_id": 397,
+ "content": "This code contains a list of useful papers related to action recognition and video classification, including TSN, TSM, SlowFast Networks, Non-local Neural Networks, X3D, ECO, 3D ResNet, TPN, EvaNet, RepFlow, MARS, StNet, and Attention Cluster.",
+ "type": "comment"
+ },
+ "4656": {
+ "file_id": 397,
+ "content": "NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification\nC-TCN: Action localization Model by Baidu, the Champion model of ActivityNet 2018\nNeural Graph Matching Networks for Fewshot 3D Action Recognition - M. Guo et al., ECCV2018. \nTemporal 3D ConvNets using Temporal Transition Layer - A. Diba et al., CVPRW2018. \nTemporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification - A. Diba et al., arXiv2017. \nAttentional Pooling for Action Recognition - R. Girdhar and D. Ramanan, NIPS2017. \nFully Context-Aware Video Prediction - Byeon et al, arXiv2017. \nHidden Two-Stream Convolutional Networks for Action Recognition - Y. Zhu et al, arXiv2017. \nDynamic Image Networks for Action Recognition - H. Bilen et al, CVPR2016. \nLong-term Recurrent Convolutional Networks for Visual Recognition and Description - J. Donahue et al, CVPR2015. \nDescribing Videos by Exploiting Temporal Structure - L. Yao et al, ICCV2015. \nReal-time Action Recognition with Enhanced Motion Vector CNNs - B. Zhang et al, CVPR2016. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/Action Recognition Papers:17-28"
+ },
+ "4657": {
+ "file_id": 397,
+ "content": "This code contains references to various research papers in the field of action recognition and video classification, highlighting different models and architectures for these tasks.",
+ "type": "comment"
+ },
+ "4658": {
+ "file_id": 397,
+ "content": "Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors - L. Wang et al, CVPR2015. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/Action Recognition Papers:29-29"
+ },
+ "4659": {
+ "file_id": 397,
+ "content": "This code refers to a paper titled \"Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors\" published in CVPR 2015 by authors L. Wang et al. This paper presents a method for action recognition using deep convolutional descriptors and trajectory pooling, offering an efficient approach to analyze and recognize actions from video sequences.",
+ "type": "comment"
+ },
+ "4660": {
+ "file_id": 398,
+ "content": "/english_documents/tutorials/Spatio-Temporal Action Detection Papers",
+ "type": "filepath"
+ },
+ "4661": {
+ "file_id": 398,
+ "content": "Summary: A comprehensive list of notable Spatio-Temporal Action Detection papers and authors from 2015-2017, covering major conferences like ICCV, BMVC, ECCV, and arXiv.",
+ "type": "summary"
+ },
+ "4662": {
+ "file_id": 398,
+ "content": "Usefull Spatio-Temporal Action Detection Papers.\n A Better Baseline for AVA - R. Girdhar et al., ActivityNet Workshop, CVPR2018.\n Real-Time End-to-End Action Detection with Two-Stream Networks - A. El-Nouby and G. Taylor, arXiv2018.\n Human Action Localization with Sparse Spatial Supervision - P. Weinzaepfel et al., arXiv2017.\n Unsupervised Action Discovery and Localization in Videos - K. Soomro and M. Shah, ICCV2017.\n Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions - P. Mettes and C. G. M. Snoek, ICCV2017.\n Action Tubelet Detector for Spatio-Temporal Action Localization - V. Kalogeiton et al, ICCV2017. \n Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos - R. Hou et al, ICCV2017. \n Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection - M. Zolfaghari et al, ICCV2017. \n TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal - H. Zhu et al., ICCV2017.",
+ "type": "code",
+ "location": "/english_documents/tutorials/Spatio-Temporal Action Detection Papers:1-13"
+ },
+ "4663": {
+ "file_id": 398,
+ "content": "The code provides a list of useful Spatio-Temporal Action Detection papers and their corresponding authors, year of publication, and conference or journal.",
+ "type": "comment"
+ },
+ "4664": {
+ "file_id": 398,
+ "content": " Online Real time Multiple Spatiotemporal Action Localisation and Prediction - G. Singh et al, ICCV2017. \n AMTnet: Action-Micro-Tube regression by end-to-end trainable deep architecture - S. Saha et al, ICCV2017.\n Am I Done? Predicting Action Progress in Videos - F. Becattini et al, BMVC2017.\n Generic Tubelet Proposals for Action Localization - J. He et al, arXiv2017.\n Incremental Tube Construction for Human Action Detection - H. S. Behl et al, arXiv2017.\n Multi-region two-stream R-CNN for action detection - X. Peng and C. Schmid. ECCV2016. \n Spot On: Action Localization from Pointly-Supervised Proposals - P. Mettes et al, ECCV2016.\n Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos - S. Saha et al, BMVC2016. \n Learning to track for spatio-temporal action localization - P. Weinzaepfel et al. ICCV2015.\n Action detection by implicit intentional motion clustering - W. Chen and J. Corso, ICCV2015.\n Finding Action Tubes - G. Gkioxari and J. Malik CVPR2015. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/Spatio-Temporal Action Detection Papers:14-24"
+ },
+ "4665": {
+ "file_id": 398,
+ "content": "This code provides a list of notable research papers related to action detection and localization in videos, published between 2015 and 2017. The papers are from various conferences such as ICCV, BMVC, ECCV, and arXiv, demonstrating the advancements made in this field during that period.",
+ "type": "comment"
+ },
+ "4666": {
+ "file_id": 398,
+ "content": " APT: Action localization proposals from dense trajectories - J. Gemert et al, BMVC2015. \n Spatio-Temporal Object Detection Proposals - D. Oneata et al, ECCV2014.\n Action localization with tubelets from motion - M. Jain et al, CVPR2014.\n Spatiotemporal deformable part models for action detection - Y. Tian et al, CVPR2013. \n Action localization in videos through context walk - K. Soomro et al, ICCV2015.\n Fast Action Proposals for Human Action Detection and Search - G. Yu and J. Yuan, CVPR2015. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/Spatio-Temporal Action Detection Papers:25-30"
+ },
+ "4667": {
+ "file_id": 398,
+ "content": "List of papers on action localization and detection in videos.",
+ "type": "comment"
+ },
+ "4668": {
+ "file_id": 399,
+ "content": "/english_documents/tutorials/TSM.md",
+ "type": "filepath"
+ },
+ "4669": {
+ "file_id": 399,
+ "content": "The Temporal Shift Module (TSM) is an efficient video understanding technique that balances performance and efficiency, capturing spatial-temporal features. Suitable for both online and offline videos, it focuses on temporal information and has a simple 2-line implementation.",
+ "type": "summary"
+ },
+ "4670": {
+ "file_id": 399,
+ "content": "# 1. Background&Motivation\nAt present, the video data on the Internet is increasing rapidly, and the time users spend watching short videos and small videos is also increasing rapidly. How to analyze, process and classify the massive video resources quickly and accurately is an urgent problem to be solved. The video understanding technology can analyze the video content in multiple dimensions, understand the video semantics, and automatically classify and label the video, which greatly saves the efficiency of manual audit and costs. At the same time, accurate user recommendation is realized to improve the experience effect.\nIn this paper, we will introduce the classic model **TSM (Temporal Shift Module)** in the field of video understanding, which is proposed by **MIT** and **IBM Watson AI Lab** `Ji Lin, Chuang Gan and Songhan, etc`, to achieve the balance between effeiciency and performance and improve video understanding ability.\nThe most relevant video understanding model to TSM is the **Temporal Segment Network (TSN)** published by Limin Wang",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:1-5"
+ },
+ "4671": {
+ "file_id": 399,
+ "content": "This code snippet provides background and motivation for TSM (Temporal Shift Module), a classic model in video understanding proposed by MIT and IBM Watson AI Lab. The TSM aims to balance efficiency and performance while improving the ability to analyze video content in various dimensions. It is related to the Temporal Segment Network (TSN) published by Limin Wang.",
+ "type": "comment"
+ },
+ "4672": {
+ "file_id": 399,
+ "content": "a series of works represented such as I3D, S3D and P3D, which carry out end-to-end joint spatial-temporal modeling through 3D convolution. Although this series of works can capture spatial-temporal features, compared with TSN, the transition from 2D convolution to 3D convolution inevitably introduces extra computation. TSM cleverly uses the idea of temporal dimension feature map shift, theoretically achieving the purpose of feature fusion and joint modeling among different frames with zero extra computing overhead compared with TSN.\n**Paper Address:** [Temporal Shift Module for Efficient Video Understanding](https://arxiv.org/pdf/1811.08383v2.pdf)\nLet's have a look at the following example: if the video is played from left to right and then from right to left respectively, the subjects will give different but correct interpretation of the video, indicating that the understanding of the video is strongly dependent on the temporal information of the video. Yes !, It is the motivation why TSM is proposed.",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:6-10"
+ },
+ "4673": {
+ "file_id": 399,
+ "content": "This code describes the Temporal Shift Module (TSM), which is a method for efficient video understanding that avoids extra computation by using temporal dimension feature map shift. It is based on the concept of capturing spatial-temporal features, with a focus on temporal information in videos. This approach aims to achieve feature fusion and joint modeling among different frames without adding extra computational overhead compared to TSN.",
+ "type": "comment"
+ },
+ "4674": {
+ "file_id": 399,
+ "content": "
\n![](\"../../images/temporal.png\")
\n
\nIt looks interesting, next,let's dive into the core modules of TSM.\n# 2. Dark technologies used in TSM\nOn the basis of traditional image analysis, video analysis needs researchers to supplement the modeling structure of temporal information. At present, 2D CNN and 3D CNN are the two most commonly used methods in video understanding: using 2D CNN model requires less computation but will lose part of the time information; While using 3D CNN has a good effect but a large amount of computation. Faced with such a situation, Ji Lin, Chuang Gan and Song Han et al. from MIT and IBM Watson AI Lab proposed the Temp Shift Module (TSM) Module. By embedding the time displacement module into 2D CNN, they can easily achieve the same video understanding ability as 3D CNN without adding any additional calculation and parameters.\n\n![](\"../../images/tsm_intr.png\")
",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:11-21"
+ },
+ "4675": {
+ "file_id": 399,
+ "content": "The code is an introduction to Temporal Shift Module (TSM) in video understanding, highlighting the trade-offs between 2D and 3D CNN methods, and how TSM embeds time displacement into 2D CNN for equivalent performance without additional computation or parameters.",
+ "type": "comment"
+ },
+ "4676": {
+ "file_id": 399,
+ "content": "
\nThe rows and columns of the matrix in the figure above represent the temporal and channel dimensions of the feature graph, respectively. In TSM module, some channels are moved forward one step int the temporal dimension, and some channels are moved backward one step in the temporal dimension, and the gaps after the displacement are zeroed. In this way, context interaction on the temporal dimension is introduced into the feature graph. The channel movement operation can make the current frame contain the channel information of the two adjacent frames. In this way, the 2D convolution operation can directly extract the spatial-temporal information of the video just like the 3D convolution.\nIt improves the modeling ability of the model in time dimension. based on this basis, the researchers further subdivided the module into TSM module suitable for online video and TSM module suitable for offline video.\n\n![](\"../../images/tsm_architecture.png\")
",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:22-27"
+ },
+ "4677": {
+ "file_id": 399,
+ "content": "This code describes the TSM (Temporal Segment Networks) module, which introduces context interaction on the temporal dimension in feature graphs. It does this by moving some channels forward and backward one step in the temporal dimension, filling gaps with zeros. The channel movement allows 2D convolution to extract spatial-temporal information like 3D convolution. This improves model ability in time dimension and has TSM modules suitable for online and offline videos.",
+ "type": "comment"
+ },
+ "4678": {
+ "file_id": 399,
+ "content": "
\nBi-Direction TSM module can obtain past and future spatial and temporal information, which is suitable for offline video with high throughput. However, UNI-Direction TSM module is only suitable for low delay online video recognition compared with the present and past spatio-temporal information.\nIn addition, the author also considered the insertion position of TSM modules and compared two TSM insertion methods: **Residual TSM** and **in-place TSM**. The author found that **Residual TSM** could achieve better performance than **in-place TSM**, At the same time, author explained that **in-place TSM** may affect the extraction of spatial information.\n\n![](\"../../images/residual_tsm.png\")
\n
\nTSM module looks **So Easy!!**, the next question is how to implement ?\n# 3. The core codes of TSM\nNow that the principle is clear, let's look at how the code works. First let's have a look the torch version tsm. Unfortunately, the Torch fr",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:28-40"
+ },
+ "4679": {
+ "file_id": 399,
+ "content": "Bi-Direction TSM module handles past and future spatial and temporal information, suitable for high throughput offline videos. UNI-Direction TSM is more appropriate for low delay online video recognition. Residual TSM performs better than in-place TSM but may affect spatial information extraction. Torch version tsm implementation to follow.",
+ "type": "comment"
+ },
+ "4680": {
+ "file_id": 399,
+ "content": "amework does not provide an API for TSM, so we will have to do it by ourselves. The code is shown below:\n\n![](\"../../images/torch_tsm.png\")
\n
\nThis means that you only need to add four lines of code to TSN's codebase then you can **double the accuracy in Something-Something datasets!!** what a simple and efficient model!\nBut...，\n**paddlepaddle** framework take the needs of the majority of users into account and have achieve TSM OP,then users can use it easily.\n\n![](\"../../images/tsm_op.png\")
\n
\nSo you no longer have to achieve it by yourself, **it cab be called directly!!! , it can be called directly!!! , it can be called directly!!!** The important thing must say three times.\nDo you think that it is the end of the this topic? **Too young Too simple !!!**\nWe have also optimized it to increase speed by 5 times while reducing memory consumption. See the acceleration documentation [accelerate.md](./accelerate.md) for more information.",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:40-58"
+ },
+ "4681": {
+ "file_id": 399,
+ "content": "The code demonstrates a TSM model implementation in PaddlePaddle framework, allowing users to achieve Temporal Shift Module (TSM) operations without writing additional code. It significantly improves accuracy and efficiency on Something-Something datasets. The provided images visually explain the TSM implementation and the optimized version (TSM OP). Additionally, the documentation refers users to the acceleration documentation for further information on speeding up the model while reducing memory consumption.",
+ "type": "comment"
+ },
+ "4682": {
+ "file_id": 399,
+ "content": "Let's have a look at how TSM is implemented using **paddlepaddle**:\n`import paddle.nn.functional as F`\n`shifts = F.temporal_shift(inputs, self.num_seg, 1.0 / self.num_seg)`\n**Only two lines codes !!!**, isn't it easy ?\n# Reference\n[1] [Lin Ji , Gan Chuang , Han Song . TSM: Temporal Shift Module for Efficient Video Understanding. arXiv:1811.08383,2018](https://arxiv.org/pdf/1811.08383v2.pdf).\n[2] [Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoo Tang,and Luc Van Gool. Temporal segment networks for action recognition in videos? In Proceedings of the European Conference on Computer Vision,pages 20–36. Springer, 2016](https://arxiv.org/abs/1608.00859).",
+ "type": "code",
+ "location": "/english_documents/tutorials/TSM.md:60-73"
+ },
+ "4683": {
+ "file_id": 399,
+ "content": "The provided code demonstrates an implementation of the Temporal Shift Module (TSM) in PaddlePaddle. It only requires two lines of code and uses the `temporal_shift` function from `paddle.nn.functional`. This makes it easy to implement the TSM for efficient video understanding, as referenced by Lin Ji et al. and Limin Wang et al.",
+ "type": "comment"
+ },
+ "4684": {
+ "file_id": 400,
+ "content": "/english_documents/tutorials/Temporal Action Detection Papers",
+ "type": "filepath"
+ },
+ "4685": {
+ "file_id": 400,
+ "content": "This code lists three research papers on action detection and localization in videos: Fast Action Proposals, Bag-of-fragments, and Action Localization through Context Walk.",
+ "type": "summary"
+ },
+ "4686": {
+ "file_id": 400,
+ "content": "Usefull Temporal Action Detection Papers. \n Rethinking the Faster R-CNN Architecture for Temporal Action Localization - Yu-Wei Chao et al., CVPR2018\n Weakly Supervised Action Localization by Sparse Temporal Pooling Network - Phuc Nguyen et al., CVPR 2018\n Temporal Deformable Residual Networks for Action Segmentation in Videos - P. Lei and S. Todrovic., CVPR2018.\n End-to-End, Single-Stream Temporal Action Detection in Untrimmed Videos - Shayamal Buch et al., BMVC 2017\n Cascaded Boundary Regression for Temporal Action Detection - Jiyang Gao et al., BMVC 2017\n Temporal Tessellation: A Unified Approach for Video Analysis - Kaufman et al., ICCV2017. \n Temporal Action Detection with Structured Segment Networks - Y. Zhao et al., ICCV2017. \n Temporal Context Network for Activity Localization in Videos - X. Dai et al., ICCV2017.\n Detecting the Moment of Completion: Temporal Models for Localising Action Completion - F. Heidarivincheh et al., arXiv2017.\n CDC: Convolutional-De-",
+ "type": "code",
+ "location": "/english_documents/tutorials/Temporal Action Detection Papers:1-12"
+ },
+ "4687": {
+ "file_id": 400,
+ "content": "This code is a list of useful temporal action detection papers, each with their respective authors and conference/journal they were published in.",
+ "type": "comment"
+ },
+ "4688": {
+ "file_id": 400,
+ "content": "Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos - Z. Shou et al, CVPR2017.\n SST: Single-Stream Temporal Action Proposals - S. Buch et al, CVPR2017.\n R-C3D: Region Convolutional 3D Network for Temporal Activity Detection - H. Xu et al, arXiv2017. [code] [project web] [PyTorch]\n DAPs: Deep Action Proposals for Action Understanding - V. Escorcia et al, ECCV2016. \n Online Action Detection using Joint Classification-Regression Recurrent Neural Networks - Y. Li et al, ECCV2016. \n Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs - Z. Shou et al, CVPR2016. \n Fast Temporal Activity Proposals for Efficient Detection of Human Actions in Untrimmed Videos - F. Heilbron et al, CVPR2016. \n Actionness Estimation Using Hybrid Fully Convolutional Networks - L. Wang et al, CVPR2016. \n Learning Activity Progression in LSTMs for Activity Detection and Early Detection - S. Ma et al, CVPR2016.\n End-to-end Learning of Action Detection from Frame Glimpses in Videos - S. Yeung et al, CVPR2016. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/Temporal Action Detection Papers:12-21"
+ },
+ "4689": {
+ "file_id": 400,
+ "content": "This code contains references to various papers on temporal action detection, localization, and understanding in untrimmed videos. It includes papers from different authors and years, with some including PyTorch implementation and project web links.",
+ "type": "comment"
+ },
+ "4690": {
+ "file_id": 400,
+ "content": " Fast Action Proposals for Human Action Detection and Search - G. Yu and J. Yuan, CVPR2015. \n Bag-of-fragments: Selecting and encoding video fragments for event detection and recounting - P. Mettes et al, ICMR2015.\n Action localization in videos through context walk - K. Soomro et al, ICCV2015.",
+ "type": "code",
+ "location": "/english_documents/tutorials/Temporal Action Detection Papers:22-24"
+ },
+ "4691": {
+ "file_id": 400,
+ "content": "This code provides references to three research papers related to action detection and localization in videos: \n1. Fast Action Proposals by Yu & Yuan (CVPR2015), \n2. Bag-of-fragments by Mettes et al. (ICMR2015), and \n3. Action Localization through Context Walk by Soomro et al. (ICCV2015).",
+ "type": "comment"
+ },
+ "4692": {
+ "file_id": 401,
+ "content": "/english_documents/tutorials/accelerate.md",
+ "type": "filepath"
+ },
+ "4693": {
+ "file_id": 401,
+ "content": "This code snippet is providing two links to tutorials, one in English and the other in Simplified Chinese (简体中文). The English tutorial can be accessed at \"../../zh-CN/tutorials/accelerate.md\" and the Chinese one at the current location \"PaddleVideo/english_documents/tutorials/accelerate.md\".",
+ "type": "summary"
+ },
+ "4694": {
+ "file_id": 401,
+ "content": "[简体中文](../../zh-CN/tutorials/accelerate.md) | English",
+ "type": "code",
+ "location": "/english_documents/tutorials/accelerate.md:1-1"
+ },
+ "4695": {
+ "file_id": 401,
+ "content": "This code snippet is providing two links to tutorials, one in English and the other in Simplified Chinese (简体中文). The English tutorial can be accessed at \"../../zh-CN/tutorials/accelerate.md\" and the Chinese one at the current location \"PaddleVideo/english_documents/tutorials/accelerate.md\".",
+ "type": "comment"
+ },
+ "4696": {
+ "file_id": 402,
+ "content": "/english_documents/tutorials/config.md",
+ "type": "filepath"
+ },
+ "4697": {
+ "file_id": 402,
+ "content": "The code demonstrates how PaddleVideo uses Inversion of Control and Dependency Injection for improved modularity, resolving coupling issues through factory classes and configuration files. It creates class instances based on configs and applies design patterns for dependency injection, using a config file for architecture, dataset, pipeline, and optimizer configurations.",
+ "type": "summary"
+ },
+ "4698": {
+ "file_id": 402,
+ "content": "# Configs design\n---\nThis page shows how PaddleVideo use the basic IOC/DI technology to decouple and control the whole framework. It is flexible to increase modularity of this system and make it extensible. At last, we will explain the details of config yaml and script args.\n## Design\nFirst, when we create a new class, it is common to new a instance like:\n```python\nclass TSM():\n pass\nmodel = TSM(init_attributes)\n```\nwhen more classes are created, the coupling relationship between the calling and called method will increase sharply, obviously, we can create a factory class to solve it, like that:\n```python\nif model_name == \"TSM\":\n model = TSM()\nelif model_name == \"TSN\":\n model = TSN()\nelif ...\n```\nand\n```python\noptimizer_cfg = dict(name:\"MOMENTUM\", params: XXX)\nif optimizer_cfg.name = \"MOMENTUM\":\n optimizer = MOMENTUM(optimizer_cfg.pop(name))\nelif:\n ...\n```\nmore and more conditions have to be created though. like widly used in the Java or other platforms, we apply ```inversion of control``` and ```Dependency Inversion``` to decuople.",
+ "type": "code",
+ "location": "/english_documents/tutorials/config.md:1-37"
+ },
+ "4699": {
+ "file_id": 402,
+ "content": "This code discusses the use of Inversion of Control (IOC) and Dependency Injection (DI) in PaddleVideo, a framework for video processing. It explains how these techniques help decouple and control the framework, increasing modularity and extensibility. The code demonstrates how traditional class instantiation can lead to coupling issues, and how IOC/DI can solve them by creating factory classes and using configuration files.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/47.json b/docs/data/47.json
new file mode 100644
index 000000000..46365e1a5
--- /dev/null
+++ b/docs/data/47.json
@@ -0,0 +1,541 @@
+{
+ "4700": {
+ "file_id": 402,
+ "content": "Second, to implenment DI, we build two components:\n- Register, to regist a class\n- Builder, to new an instance\n1. Register\nWe implenment a getter and a setter function to map string to an instance.\n[source code](../../paddlevideo/utils/registry.py)\n```python\n#excerpt from source code.\nclass Registry():\n def __init__(self, name):\n self._name = name\n self._obj_map = {}\n #mapping name -> object\n def register(self, obj, name):\n self._obj_map[name] = obj\n #get object\n def get(self, name):\n ret = self._obj_map.get(name)\n return ret\n```\nIt provides name -> object mapping. For example, To register an object:\n```python\n BACKBONES = Registry('backbone')\n class ResNet:\n pass\n BACKBONES.register(ResNet)\n```\nOr, use a decorator\n```python\n BACKBONES = Registry('backbone') #new a Register\n @BACKBONES.register() #regist resnet as a backbone.\n class ResNet:\n pass\n```\n2. Builder\nTo obtain a registed module.\n```python\n # Usage: To build a module.\n backbone_name = \"ResNet\"",
+ "type": "code",
+ "location": "/english_documents/tutorials/config.md:39-89"
+ },
+ "4701": {
+ "file_id": 402,
+ "content": "This code demonstrates the implementation of Dependency Injection (DI) using a Register and Builder. The Register provides name-to-object mapping, allowing objects to be registered with a specific name. The Builder facilitates obtaining registered modules by accepting a module's name and returning the corresponding instance.",
+ "type": "comment"
+ },
+ "4702": {
+ "file_id": 402,
+ "content": " b = BACKBONES.get(backbone_name)()\n```\nso that we can new(register) an instance in **where it declared**, not **where it called**, a basic DI sub-system has been created now.\nWe apply this design on many places, such as: PIPELINE, BACKBONE, HEAD, LOSS, METRIC and so on.\nFinally, We build all of the framework components from config yaml which matches the source code one by one, **It means the attributes in a configuration field is same as the init atrributes of the mathced class**, and to indicate a specified class, we always use ```name``` to mark it. like:\n```yaml\nhead:\n name: \"TSMHead\" # class name\n num_classes: 400 # TSMHead class init attributes\n ...\n```\n---\n## config yaml details\nWe separate the config to several parts, in high level:\n- **MODEL:** Architecture configuration, such as HEAD module, BACKBONE module.\n- **DATASET:** DATASET and dataloader configuration.\n- **PIPELINE:** pipeline of processing configuration.\n- **OPTIMIZER:** Optimizer configuration.\nand some unique global configurations, like",
+ "type": "code",
+ "location": "/english_documents/tutorials/config.md:90-117"
+ },
+ "4703": {
+ "file_id": 402,
+ "content": "This code snippet is creating an instance of a class based on its name specified in the configuration file. It applies this design to various components like PIPELINE, BACKBONE, HEAD, LOSS, and METRIC for dependency injection. The attributes in the configuration field match the initialization attributes of the corresponding class. The config file separates architecture, dataset, pipeline, and optimizer configurations, along with global settings.",
+ "type": "comment"
+ },
+ "4704": {
+ "file_id": 402,
+ "content": "- model_name\n- log_interval\n- epochs\n- resume_epoch\n- log_level\n...\nTraining script args\n- **--validate**: switch validate mode on or not\n- **--test**: switch test mode on or not\n- **--weights**: weights path\n- **-c**: config yaml path\n- **-o**: override args, one can use it like: -o DATASET.batch_size=16",
+ "type": "code",
+ "location": "/english_documents/tutorials/config.md:118-131"
+ },
+ "4705": {
+ "file_id": 402,
+ "content": "This code snippet is describing command-line arguments for a training script. The user can switch validate or test mode on/off, provide weights and config paths, and override specific args using \"-o\" option. It also mentions the available commands for each argument.",
+ "type": "comment"
+ },
+ "4706": {
+ "file_id": 403,
+ "content": "/english_documents/tutorials/customized_usage.md",
+ "type": "filepath"
+ },
+ "4707": {
+ "file_id": 403,
+ "content": "This code provides information about customizing different aspects of the PaddleVideo framework, including dataset, network, solvers, metrics, and debug tools. It discusses finetuning, adding new augmentations and batch augments in the pipeline, modular design, changing frameworks, initializing functions, loss functions, step/epoch decay, creating customized solvers, adding new data processing, records, and metrics, as well as using debug levels and FAQ.",
+ "type": "summary"
+ },
+ "4708": {
+ "file_id": 403,
+ "content": "[简体中文](../../zh-CN/tutorials/customized_usage.md) | English\n# Customized Usage\n## Customized Dataset\n1. finetune\nPlease refer to [finetune](../start.md#model_finetune) if only change a \"regular\" dataset.\n2. customized pipeline\n - add new augments\n - add new batch augments\n **Note**: Be care of checking the difference of different modes.\n## Customized Network\n1. module function\nPlease refer to [modular desigh](modular_design.md) for more information.\n2. customized framework\n - change framework\n - change initialized function\n - customized loss\n## Customized Solvers\n1. step decay and epoch decay\n2. customized solvers\n## Customized metrics\n - add new data processing\n - add new record\n - add new metrics\n## Debug tools\n1. Debug level\n2. FAQ",
+ "type": "code",
+ "location": "/english_documents/tutorials/customized_usage.md:1-44"
+ },
+ "4709": {
+ "file_id": 403,
+ "content": "This code provides information about customizing different aspects of the PaddleVideo framework, including dataset, network, solvers, metrics, and debug tools. It discusses finetuning, adding new augmentations and batch augments in the pipeline, modular design, changing frameworks, initializing functions, loss functions, step/epoch decay, creating customized solvers, adding new data processing, records, and metrics, as well as using debug levels and FAQ.",
+ "type": "comment"
+ },
+ "4710": {
+ "file_id": 404,
+ "content": "/english_documents/tutorials/demos",
+ "type": "filepath"
+ },
+ "4711": {
+ "file_id": 404,
+ "content": "The code outlines six different tasks in action recognition and detection using various algorithms, including TSN (Two-Stream Convolutional Networks), TSM (Temporal Shift Module), SlowFast Networks, LSTM (Long Short-Term Memory), and BNM (Boundary-aware Multi-scale Network). The tasks include single-class action recognition, multi-class action recognition, action localization, spatio-temporal action detection, 3000-class tagging application, and highlights detection application.",
+ "type": "summary"
+ },
+ "4712": {
+ "file_id": 404,
+ "content": "some useful demo todo. \n1、single-class action recognition， tsn/tsm/slowfast \n2、multi-class action recognition，lstm \n3、action localization，bmn \n4、spatio temporal action detection，todo \n5、3000-class tagging application(videotag)：tsn+lstm \n6、Highlights detection application：bmn+tsn+lstm ",
+ "type": "code",
+ "location": "/english_documents/tutorials/demos:1-8"
+ },
+ "4713": {
+ "file_id": 404,
+ "content": "The code outlines six different tasks in action recognition and detection using various algorithms, including TSN (Two-Stream Convolutional Networks), TSM (Temporal Shift Module), SlowFast Networks, LSTM (Long Short-Term Memory), and BNM (Boundary-aware Multi-scale Network). The tasks include single-class action recognition, multi-class action recognition, action localization, spatio-temporal action detection, 3000-class tagging application, and highlights detection application.",
+ "type": "comment"
+ },
+ "4714": {
+ "file_id": 405,
+ "content": "/english_documents/tutorials/deployment.md",
+ "type": "filepath"
+ },
+ "4715": {
+ "file_id": 405,
+ "content": "This code explains how to convert dygraph models to static models for inference and deployment using PaddleInference, and provides examples on video inference testing with predict.py and benchmarking. Support for C++ infer and PaddleHub Serving deploy are coming soon.",
+ "type": "summary"
+ },
+ "4716": {
+ "file_id": 405,
+ "content": "[简体中文](../../zh-CN/tutorials/deployment.md) | English\n# Inference\n## How to convert dygraph model to static model?\nTo infer and deploy a model, we need export an inference model, or called to_static: `convert dygraph model to static model`, at first.\n```python\npython3.7 tools/export_model.py -c config_file -o output_path -p params_file\n```\nNote: In `export_model.py`, It will build a model again, and then loading the prarams. But some init params in the infer phase is different from the train phase.\nwe add `num_seg` for TSM in advanced, please add more params or modify them if it is necessary.\nplease refer to [official documents](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/04_dygraph_to_static/index_cn.html) for more information.\n## How to test the export model?\nPaddleVideo supports a test script to test the exported model.\n```python\npython3.7 tools/test_export_model.py -p params_file -i inference_folder -c config_file\n```\nWe just print the output shape, please feel free to ex",
+ "type": "code",
+ "location": "/english_documents/tutorials/deployment.md:1-24"
+ },
+ "4717": {
+ "file_id": 405,
+ "content": "The code provides instructions on how to convert a dygraph model to a static model for inference and deployment, as well as testing the exported model using PaddleVideo's test script. The conversion is done using the \"export_model.py\" script with appropriate arguments, and some additional parameters are added for TSM. Refer to official documents for more information.",
+ "type": "comment"
+ },
+ "4718": {
+ "file_id": 405,
+ "content": "tend it. Avtually, only test a video file by PaddleInference can make sure the exported model is right.\n## How to use PaddleInference?\nPaddleVideo supports ```tools/predict.py``` to infer\n```python\npython3.7 tools/predict.py -v example.avi --model_file \"./inference/example.pdmodel\" --params_file \"./inference/example.pdiparams\" --enable_benchmark=False --model=\"example\" --num_seg=8\n ```\n## How to test inference speed?\nPaddleVideo support a script to test inference speed\n```python\npython3.7 tools/predict.py --enable_benchmark=True --model_file=模型文件 --params_file=参数文件\n```\n## How to use C++ infer?\n coming soon\n# Deployment\n## How to use PaddleHub Serving deploy?\n coming soon\n## How to use PaddleLite deploy?\n coming soon",
+ "type": "code",
+ "location": "/english_documents/tutorials/deployment.md:24-48"
+ },
+ "4719": {
+ "file_id": 405,
+ "content": "This code explains how to use the PaddleInference tool for testing video inference, providing examples on using predict.py and enabling benchmarking. It also mentions that support for C++ infer is coming soon, as well as instructions on using PaddleHub Serving deploy and PaddleLite deploy, which will be added later.",
+ "type": "comment"
+ },
+ "4720": {
+ "file_id": 406,
+ "content": "/english_documents/tutorials/modular_design.md",
+ "type": "filepath"
+ },
+ "4721": {
+ "file_id": 406,
+ "content": "This code provides a link to the Chinese version of the modular design tutorial and the English version, allowing users to switch between languages based on their preference.",
+ "type": "summary"
+ },
+ "4722": {
+ "file_id": 406,
+ "content": "[简体中文](../../zh-CN/tutorials/modular_design.md) | English",
+ "type": "code",
+ "location": "/english_documents/tutorials/modular_design.md:1-1"
+ },
+ "4723": {
+ "file_id": 406,
+ "content": "This code provides a link to the Chinese version of the modular design tutorial and the English version, allowing users to switch between languages based on their preference.",
+ "type": "comment"
+ },
+ "4724": {
+ "file_id": 407,
+ "content": "/english_documents/tutorials/pp-tsm.md",
+ "type": "filepath"
+ },
+ "4725": {
+ "file_id": 407,
+ "content": "This code introduces the PP-TSM, a high-performance and efficient video recognition model optimized based on TSM in PaddleVideo. It outlines various strategies like ImageNet pretraining, data augmentation, and optimizer improvements to enhance performance and achieve fast inference speed on V101 GPU with top-1 accuracy on UCF101 and Kinetics400 datasets.",
+ "type": "summary"
+ },
+ "4726": {
+ "file_id": 407,
+ "content": "# High performance recognition 2D architecture PP-TSM\nPP-TSM：An Effective and Efficient video-recognition model \nPP-TSM is an optimized model based on TSM in PaddleVideo, \nwhose performance (top-1 on UCF101 and Kinetics400) and inference spped \nare better than TSM paper(https://arxiv.org/abs/1811.08383 ) and \nother open source TSM，PaddlePaddle2.0(available on pip now) or \nDaily Version( https://www.paddlepaddle.org.cn/documentation/docs/zh/install/Tables.html#whl-dev ) \nis required to run PP-TSM. \nWhen only use ImageNet for pretrain and only use 8X1 sample， \nPP-TSM’s top1 reached to 89.5% and 73.5% on UCF101 and Kinetics400, \nand inference speed of FP32 on single V100 is 147 VPS on Kinectics400 dataset. \ninference speed of FP16 with TensorRT on single V100 isTODO. \nAs far as we know, under the same conditions, \ntop1=73.5% on Kinetics400 is the best performance for 2D video model until now. \nPP-TSM improved performance and speed of TSM with following methods: \n1、Model Tweaks: ResNet50vd ，+2.5% ",
+ "type": "code",
+ "location": "/english_documents/tutorials/pp-tsm.md:1-22"
+ },
+ "4727": {
+ "file_id": 407,
+ "content": "This code describes the PP-TSM, a high-performance and efficient video recognition model optimized based on TSM in PaddleVideo. It mentions its better performance and inference speed compared to TSM paper and other open source TSM models. Requires PaddlePaddle2.0 for execution. When using ImageNet for pretrain and 8X1 sample, it achieves high top-1 accuracy on UCF101 and Kinetics400 datasets with fast inference speed on V100 GPU.",
+ "type": "comment"
+ },
+ "4728": {
+ "file_id": 407,
+ "content": "2、ImageNet pretrain weights based on Knowledge Distillation ， +1.3% \n3、beter batch size ，+0.2% \n4、beter L2 ，+0.3% \n5、label_smoothing ，+0.2% \n6、beter lr decay ，+0.15% \n7、Data augmentation ，+0.3% \n8、beter epoch num ，+0.15% \n9、bn strategy ，+0.4% \n10、integrated PaddleInference \n11、more strategies todo: Knowledge Distillation、optimizer and so on. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/pp-tsm.md:23-32"
+ },
+ "4729": {
+ "file_id": 407,
+ "content": "This code outlines several strategies implemented to improve the performance of a model, including ImageNet pretraining, better batch size and L2 values, label smoothing, better learning rate decay, data augmentation, and updated epoch numbers. The code also mentions using Knowledge Distillation, optimizer improvements, and plans for integrating PaddleInference.",
+ "type": "comment"
+ },
+ "4730": {
+ "file_id": 408,
+ "content": "/english_documents/tutorials/summarize.md",
+ "type": "filepath"
+ },
+ "4731": {
+ "file_id": 408,
+ "content": "Video classification tasks involve recognizing actions through RGB images and skeleton data. Concepts include temporal action localization, dense-captioning events, popular datasets, feature extraction, motion representation, and classification using deep learning methods since 2014.",
+ "type": "summary"
+ },
+ "4732": {
+ "file_id": 408,
+ "content": "[简体中文](../../zh-CN/tutorials/summarize.md) | English\n# Introduction for video classification(action recognition)\n## Wide range of application scenarios\nVideo classification has a wide range of applications in many fields, such as online video platforms such as short videos, offline such as security, transportation, quality inspection and other fields。\n## Multiple subtasks\nSimilar to image tasks, video tasks can also be divided into two categories: **classification (recognition) and detection**, and these two types of tasks can be specifically subdivided by combining different scenes：\n+ Task1：Trimmed Action Recognition. Users input a trimmed video,which contains only single action,then a video tag will be output by model as depicted in fig below:\n\n![](\"../../images/action_classification.png\")
\n Action Classification\n
\n In terms of the data modality used, classification tasks can be further subdivided into classification based on si",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:1-18"
+ },
+ "4733": {
+ "file_id": 408,
+ "content": "Introduction to video classification (action recognition) with various applications in different fields, including online platforms and offline sectors like security, transportation, and quality inspection. Tasks include classification/recognition and detection, further subdivided by combining different scenes.",
+ "type": "comment"
+ },
+ "4734": {
+ "file_id": 408,
+ "content": "ngle modality data, classification based on multi-modality data, classification based on RGB images and classification based on human skeleton, etc, as shown in the figure below:\n \n ![](\"../../images/multimodality.png\")
\n multi-modality\n
\nIn terms of the perspective of video, it can also be divided into first-person action recognition, \nthird-person action recognition, single perspective action recognition and multi-perspective fusion action recognition. \nUsers who are interested in these fields can refer to relevant literatures.\n+ Task2：Untrimmed Video Classification. \nUnlike trimmed videos, untrimmed videos often contain multiple actions and have a long time span. \nThere are a lot of movements that we may need not paying attention to. Through the global analysis of the input long video, and then make a soft classify to mutiple categories.\n+ Task3：Temporal Action Proposal. It is similar to the ROI extraction in the image detection task. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:18-32"
+ },
+ "4735": {
+ "file_id": 408,
+ "content": "This code is describing different types of classification tasks in video analysis. It covers multi-modality data, RGB images, human skeleton data, and various perspectives such as first-person, third-person, and multiple perspectives. Additionally, it mentions untrimmed videos, temporal action proposals, and ROI extraction in image detection tasks.",
+ "type": "comment"
+ },
+ "4736": {
+ "file_id": 408,
+ "content": "The task is to find the video clips that may contain action in a long video with a lot of actions.\n+ Task4：Temporal Action Localization. Compared with the temporal action proposal task as mentioned above, \ntemporal action localization task is more consistent with detection task in the field of imgae, \nit requires not only to find the video segments with possible actions from the video but also to classify them,\nas shown in the figure below\n \n![](\"../../images/action_detection.png\")
\n Action Detection\n
\n+ Task5：Dense-Captioning Events. The reason why it is called dense captioning events is mainly \nbecause that this task requires video action description on the basis of temporal action localization \n(detection). That is to say, the task needs to locate the actions in a **untrimmed** video,in **temporal \ndimension** and describe the behavior of the **whole video** after obtaining many video segments which contain actions.\n## Introduction of datasets",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:33-49"
+ },
+ "4737": {
+ "file_id": 408,
+ "content": "Task 4: Temporal Action Localization - find video segments with possible actions, classify them.\nTask 5: Dense-Captioning Events - describe untrimmed videos' actions in temporal dimension.",
+ "type": "comment"
+ },
+ "4738": {
+ "file_id": 408,
+ "content": "### Classification datasets\nThe training and validation of the model cannot be done without comprehensive, \nlarge and well annotated datasets. With the deepening of research on video action recognition, \nmore and more datasets are applied to the research in this field. \nTypical datasets are as follows:\n+ KTH[1](#1)\nKTH dataset is an early small action recognition dataset, \nincluding 599 videos of 6 types of actions (walking, jumping, running, punching, waving and clapping). \nThe background is relatively still, except for the zoom in and out of the camera, \nthe camera movement is relatively slight. Since this data set is relatively small, \nit is easy to overfit when training heavy 3D networks, \nso most current researches are not based on this it.\n+ UCF10[2](#2)\nUCF101 is a medium-size dataset in which most videos are from YouTube. \nIt contains 13,320 videos with 101 types of actions. \nEach type of action is performed by 25 people, each of whom performs 4-7 sets of actions. \nThe UCF101 and HMDB51 datasets used to be the benchmarks to evaluate the effectiveness of action ",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:51-72"
+ },
+ "4739": {
+ "file_id": 408,
+ "content": "The code provides a brief overview of popular video action recognition datasets, such as KTH and UCF101. It mentions that the datasets are essential for training and validating models, but overfitting may occur with larger 3D networks on smaller datasets like KTH.",
+ "type": "comment"
+ },
+ "4740": {
+ "file_id": 408,
+ "content": "recognition model for a long time before the Kinetics dataset was released.\n+ HMDB51[3](#3)\nBrown University's proposed dataset named HMDB51 was released in 2011. \nMost of the videos come from movies, \nbut some come from public databases and online video libraries such as YouTube. \nThe datasets contains 6849 samples divided into 51 classes, \neach of which contains at least 101 samples.\n+ Kinetics[4](#4)\nKinetics is the most important large-scale action recognition dataset, which was proposed by Google's DeepMind team in 2017. The video data also comes from YouTube, with 400 categories (now expanded to 700 categories) and more than 300,000 videos (now expanded to 600,000 videos), each lasting about 10 seconds. \nThe action categories are mainly divided into three categories: \"human\", \"human and animal\", \"human and human interaction\". Kinetics can train 3D-RESNET up to 152 layers without over-fitting, \nwhich solves the problem that the previous training dataset is too small to train deep 3D network. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:73-87"
+ },
+ "4741": {
+ "file_id": 408,
+ "content": "HMDB51 is a dataset proposed by Brown University in 2011, consisting of movie and online video sources. It contains 6849 samples across 51 classes with at least 101 samples each. Kinetics is the largest action recognition dataset, created by Google's DeepMind team in 2017. It uses YouTube videos, now expanded to 600k videos in 700 categories. The categories are divided into human, human and animal, and human and human interaction. Kinetics can train deep networks like 3D-RESNET up to 152 layers without overfitting, solving the issue of small training datasets.",
+ "type": "comment"
+ },
+ "4742": {
+ "file_id": 408,
+ "content": "Kinetics has replaced UCF101 and HMDB51 as the benchmark in the field of action recognition. \nAt present, most studies use this dataset for evaluation and pre-training.\n+ Something-Something[5](#5)\nSomethingV1 contains 108,499 annotated videos (V2 has expanded to 220,847), each of which last two to six seconds. These videos contain 174 kinds of actions. Different from the previous dataset, \nthe identification of this data set requires stronger time information, \nso this dataset has a very important reference value in testing the temporal modeling ability of the model.\nIn addition to the above datasets, there are Charades[6](#6) dataset for complex Action recognition, Breakfast Action[7](#7), and Sports 1M[8](#8).\n### Detection datasets\n+ THUMOS 2014\nThis dataset is from THUMOS Challenge 2014, Its training set is UCF101, validation set and test set include 1010 and 1574 undivided video clips respectively. In the action detection task, only 20 kinds of unsegmented videos of actions were labeled with sequential action fragments, ",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:88-104"
+ },
+ "4743": {
+ "file_id": 408,
+ "content": "Kinetics is the benchmark for action recognition, replacing UCF101 and HMDB51. Most studies use this dataset for evaluation and pre-training. SomethingV1 has 108,499 annotated videos with 174 kinds of actions, requiring strong temporal modeling ability. Other datasets include Charades (complex action recognition), Breakfast Action, Sports 1M, THUMOS 2014 (action detection).",
+ "type": "comment"
+ },
+ "4744": {
+ "file_id": 408,
+ "content": "including 200 validation sets (3007 action fragments) and 213 test sets (3358 action fragments).\n+ MEXaction2\nThe Mexaction2 dataset contains two types of action: horse riding and bullfighting. \nThe dataset consists of three parts: YouTube videos, horseback riding videos in UCF101, and INA videos. \nYouTube clips and horseback riding videos in UCF101 are short segmented video clips that are used as training sets. \nThe INA video is a long unsegmented video with a total length of 77 hours, \nand it is divided into three parts: training, validation and test. \nThere are 1336 action segments in the training set, 310 in the validation set and 329 in the test set. \nMoreover, the Mexaction2 dataset is characterized by very long unsegmented video lengths, \nand marked action segments only account for a very low proportion of the total video length.\n+ ActivityNet\nAt present the largest database, also contains two tasks of classification and detection. \nThis dataset only provides a YouTube link to the video, not a direct download of the video, ",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:105-121"
+ },
+ "4745": {
+ "file_id": 408,
+ "content": "The code describes two datasets, Mexaction2 and ActivityNet. Mexaction2 has horse riding and bullfighting actions, split into training, validation, and test sets. It includes YouTube clips, UCF101 horseback riding videos, and an unsegmented 77-hour INA video with low marked action proportions. ActivityNet is the largest database, including classification and detection tasks, but only provides YouTube links without direct downloads.",
+ "type": "comment"
+ },
+ "4746": {
+ "file_id": 408,
+ "content": "so you also need to use the YouTube download tool in Python to automatically download the videos. \nThe dataset contains 200 action categories, 20,000 (training + verification + test set) videos, \nand a total of about 700 hours of video.\n## Introduction of classic models\nAs shown in the figure, \nthe action recognition framework mainly includes three steps: \nfeature extraction, motion representation and classification. \nHow to extract spatiotemporal features of video is the core problem of action recognition and video classification.\n \n![](\"../../images/action_framework.png\")
\nFramework of action recognition\n
\nAccording to different methods, action recognition (video classification) methods can be generally summarized into two stages: \nmanual feature-based method and deep learning-based method. \nTypical motion descriptors in the manual feature-based method stage include DTP and IDT, \nwhich are also the most excellent motion descriptors accepted by most researchers before deep-learning is applied in this field. ",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:122-138"
+ },
+ "4747": {
+ "file_id": 408,
+ "content": "The code discusses the process of feature extraction, motion representation and classification in action recognition. It highlights two stages - manual feature-based method and deep learning-based method. It mentions DTP and IDT as typical motion descriptors used before deep-learning was applied. The code also shows a framework diagram for action recognition.",
+ "type": "comment"
+ },
+ "4748": {
+ "file_id": 408,
+ "content": "Interinterested readers may refer to the relevant references at the end of this paper. \nSince 2014, deep learning methods have been gradually applied to the field of video classification. \nAt present, deep learning-based methods have become a hotspot of research in both academic and the practice, and the effect is far beyond the motion features of manual design. \nSince 2014, many classic network structures have been put forward by the researchers regarding the problem of how to represent motion characteristics, \nas shown in the figure below:\n \n![](\"../../images/classic_model.png\")
\nClassic Models\n
\nAt present,Paddlevideo has contained several classic models such as:TSN[9](#9),TSM[10](#10),slowfast[11](#11),et al.In the future,\nwe will analyze the classic models and papers in these fields. Please look forward to it\n## Introduction of competetion\n+ [ActivityNet](http://activity-net.org/challenges/2020/challenge.html)",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:139-154"
+ },
+ "4749": {
+ "file_id": 408,
+ "content": "The code discusses the application of deep learning methods in video classification since 2014, highlighting their effectiveness beyond manual motion design. It mentions various classic network structures proposed by researchers for representing motion characteristics and includes images to illustrate these models. The code also introduces PaddleVideo's inclusion of such models like TSN, TSM, slowfast, etc., and anticipates future analysis of these classical models and papers in the field. Additionally, it references an ActivityNet competition for further context.",
+ "type": "comment"
+ },
+ "4750": {
+ "file_id": 408,
+ "content": "ActivityNet is a large-scale action recognition competition. Since 2016, \nit has been held simultaneously with CVPR every year. Up to this year, \nit has been held for 4 consecutive sessions. It focuses on identifying everyday, high-level, goal-oriented activities from \nuser-generated videos taken from the Internet video portal YouTube. \nAt present, ActivityNet competition has become the most influential competition in the field of action recognition.\n## Reference\n\n[1] Schuldt C, Laptev I, Caputo B.Recognizing Human Actions: A Local SVM Approach Proceedings of International Conference on Pattern Recognition. Piscataway, NJ: IEEE, 2004:23-26\n
\n
\n\n[2] Soomro K, Zamir A R, Shah M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv:1212.0402,2012.\n
\n
\n\n[3] Kuehne H, Jhuang H, Garrote E, et al. HMDB: a large video database for human motion recognition Proceedings of IEEE International Conference on Computer Vision. Piscataway, NJ: IEEE, 2011:2556-2563.",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:156-173"
+ },
+ "4751": {
+ "file_id": 408,
+ "content": "This code snippet provides information about the ActivityNet competition, which is a large-scale action recognition event held annually since 2016. It focuses on identifying everyday activities from user-generated YouTube videos and has become the most influential in the field of action recognition. The code also includes references to relevant research papers for further reading.",
+ "type": "comment"
+ },
+ "4752": {
+ "file_id": 408,
+ "content": "
\n
\n\n[4] Carreira J , Zisserman A . Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017:6299-6308.\n
\n
\n\n[5] Goyal R, Kahou S E, Michalski V. The “something something” video database for learning and evaluating visual common sense. arXiv:1706.04261,2017.\n
\n
\n\n[6] Sigurdsson G A , Varol Gül, Wang Xiaolong, et al. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. arXiv: 604.01753,2016\n
\n
\n\n[7] Kuehne H, Arslan A, Serre T. The Language of Actions Recovering the Syntax and Semantics of Goal-Directed Human Activities Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2014.\n
\n
\n\n[8] Karpathy A , Toderici G , Shetty S , et al. Large-Scale Video Classification with Convolutional Neural Networks Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2014:1725-1732.",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:174-193"
+ },
+ "4753": {
+ "file_id": 408,
+ "content": "This code provides references to various research papers related to action recognition and video classification, such as the \"Quo Vadis, Action Recognition?\" paper by Carreira and Zisserman, and the \"Hollywood in Homes\" paper by Sigurdsson et al. These references are from well-known conferences like IEEE Conference on Computer Vision and Pattern Recognition (CVPR) and arXiv preprints.",
+ "type": "comment"
+ },
+ "4754": {
+ "file_id": 408,
+ "content": "
\n
\n\n[9] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoo Tang,and Luc Van Gool. Temporal segment networks for action recognition in videos? In Proceedings of the European Conference on Computer Vision,pages 20–36. Springer, 2016.\n
\n
\n\n[10] Lin Ji , Gan Chuang , Han Song . TSM: Temporal Shift Module for Efficient Video Understanding. arXiv:1811.08383,2018.\n
\n
\n\n[11] Feichtenhofer C , Fan Haoqi , Malik J , et al. SlowFast Networks for Video Recognition. arXiv:1812.03982,2018.\n
",
+ "type": "code",
+ "location": "/english_documents/tutorials/summarize.md:194-206"
+ },
+ "4755": {
+ "file_id": 408,
+ "content": "The code represents a list of references for papers related to video recognition. Each reference has an identifier (id) and the corresponding paper details like authors, title, and publication information.",
+ "type": "comment"
+ },
+ "4756": {
+ "file_id": 409,
+ "content": "/english_documents/usage.md",
+ "type": "filepath"
+ },
+ "4757": {
+ "file_id": 409,
+ "content": "This code sets up PaddleVideo on Linux, supports multi-card training and testing with PaddlePaddle. It provides log format for phases, resumes sessions, fine-tunes with pretrained params and best accuracy achieved. The code launches PaddleVideo in distributed mode with 4 GPUs, tests, exports model, introduces `use_gpu` parameter, and benchmark results are available in benchmark document.",
+ "type": "summary"
+ },
+ "4758": {
+ "file_id": 409,
+ "content": "[简体中文](../zh-CN/usage.md) | English\n# Usage\n---\nPlease refer to [installation documents](./install.md) to prepare the enviroment, and follow the steps mentioned in the [data preparation documents](./dataset/) to construct dataset, we will take you through the basic functions supported by PaddleVideo, all of it takes the ucf101 dataset with frame format as example.\nPaddleVideo only support linux operation system and GPU running time environment now.\nDefault detination folder of PaddleVideo files. running the [example config](../../configs/example.yaml) as example.\n```\nPaddleVideo\n ├── paddlevideo\n ├── ... #other source codes\n ├── output #ouput destination\n | ├── example\n | | ├── example_best.pdparams #path_to_weights\n | | └── ... \n | └── ... \n ├── log #log file destination.\n | ├── worker.0\n | ├── worker.1\n | └── ... \n └── inference #inference files destination.\n ├── .pdiparams file\n ├── .pdimodel file\n └── .pdiparmas.info file",
+ "type": "code",
+ "location": "/english_documents/usage.md:1-28"
+ },
+ "4759": {
+ "file_id": 409,
+ "content": "This code provides instructions for setting up the environment, preparing data using the PaddleVideo library, and explains its supported functions. It also mentions that it only supports Linux operation systems with GPU environments and gives an example of how to run the library. The code outlines the default destination folders for output, log files, and inference files.",
+ "type": "comment"
+ },
+ "4760": {
+ "file_id": 409,
+ "content": "```\n\n## 1. Train and Test\nStart running multi-cards training scripts or test scripts by `paddle.distributed.launch`, or run the `run.sh` directly.\n```bash\nsh run.sh\n```\nWe put all the start commands in advanced in the ```run.sh```, please uncomment the selected one to run.\n\n### 1.1 Train\nSwitch `--validate` on to validating while training.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n --validate \\\n -c ./configs/example.yaml\n```\nIndicating `-c` to set configuration, and one can flexible add `-o` in the script to update it.\n```bash\npython -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --validate \\\n -o DATASET.batch_size=16\n```\nIndicating `-o DATASET.batch_size=16` can update batch size to 16, please refer to [configuration](tutorials/config.md#config-yaml-details) for more information.\nAfter starting training, log files will generated, ",
+ "type": "code",
+ "location": "/english_documents/usage.md:29-71"
+ },
+ "4761": {
+ "file_id": 409,
+ "content": "This code demonstrates how to train and test a model using PaddlePaddle, a popular deep learning framework. The training process involves running multi-card training scripts or tests by executing the `paddle.distributed.launch` command with appropriate arguments such as GPU selection, script path, and optional configuration file. The configuration file allows for flexible updates like changing batch sizes on the fly. After starting the training, log files are generated for tracking progress and analysis.",
+ "type": "comment"
+ },
+ "4762": {
+ "file_id": 409,
+ "content": "and its format is shown as below, it will output to both the screen and files. Default destination of log is under the `.log/` folder, and stored in the files named like `worker.0`, `worker.1` ...\n[train phase] current time, current epoch/ total epoch, batch id, metrics, elapse time, ips, etc.:\n [12/28 17:31:26] epoch:[ 1/80 ] train step:0 loss: 0.04656 lr: 0.000100 top1: 1.00000 top5: 1.00000 elapse: 0.326 reader: 0.001s ips: 98.22489 instance/sec.\n[eval phase] current time, current epoch/ total epoch, batch id, metrics, elapse time, ips, etc.:\n [12/28 17:31:32] epoch:[ 80/80 ] val step:0 loss: 0.20538 top1: 0.88281 top5: 0.99219 elapse: 1.589 reader: 0.000s ips: 20.14003 instance/sec.\n[epoch end] current time, metrics, elapse time, ips, etc.\n [12/28 17:31:38] END epoch:80 val loss_avg: 0.52208 top1_avg: 0.84398 top5_avg: 0.97393 elapse_avg: 0.234 reader_avg: 0.000 elapse_sum: 7.021s ips: 136.73686 instance/sec.\n[the best Acc] \n [12/28 17:28:42] Already save the best model (top1 acc)0.8494",
+ "type": "code",
+ "location": "/english_documents/usage.md:71-89"
+ },
+ "4763": {
+ "file_id": 409,
+ "content": "The code shows log output format for training and validation phases, including time, epoch, batch ID, metrics, elapse time (execution time), and ips (instances per second). It also displays the best accuracy achieved during training.",
+ "type": "comment"
+ },
+ "4764": {
+ "file_id": 409,
+ "content": "\n### 1.2 Resume\nIndicate `-o resume_epoch` to resume, It will training from ```resume_epoch``` epoch, PaddleVideo will auto load optimizers parameters and checkpoints from `./output` folder, as it is the default output destination.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --validate \\\n -o resume_epoch=5\n```\n\n### 1.3 Finetune\nIndicate `--weights` to load pretrained parameters, PaddleVideo will auto treat it as a finetune mission.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --validate \\\n --weights=./outputs/example/path_to_weights\n```\nNote: PaddleVideo will NOT load shape unmatched parameters.\n\n### 1.4 Test\nSwitch `--test` on to start test mode, and indicate `--weights` to load pretrained model.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3",
+ "type": "code",
+ "location": "/english_documents/usage.md:91-132"
+ },
+ "4765": {
+ "file_id": 409,
+ "content": "The code provides instructions on how to use PaddleVideo for three different tasks: resuming a training session, finetuning with pretrained parameters, and testing. In the resume task, the user should indicate \"-o resume_epoch\" to continue from a specific epoch, while in finetuning, \"--weights\" is used to load pretrained parameters. The test mode is activated using \"--test\". PaddleVideo will not load unmatched parameters.",
+ "type": "comment"
+ },
+ "4766": {
+ "file_id": 409,
+ "content": "python3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --test \\\n --weights=./output/example/path_to_weights\n```\n\n## 2. Infer\nFirst, export model.\nIndicate `-c` to set configuration, `-p` to load pretrained model, `-o` to set inference files destination.\n```bash\npython tools/export_model.py \\\n -c ./configs/example.yaml \\\n -p ./output/example/path_to_weights \\\n -o ./inference\n```\nIt will generate `model_name.pdmodel` , `model_name.pdiparams` and `model_name.pdiparames.info`.\nSecond, start PaddleInference engine to infer a video.\n```bash\npython tools/predict.py \\\n --input_file \"data/example.avi\" \\\n --model_file \"./inference/example.pdmodel\" \\\n --params_file \"./inference/example.pdiparams\" \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nAttributes:\n+ `input_file`: input file path or input directory, which contains input files(s).\n+ `model_file`: pdmodel file path.\n+ `params_file`: pdiparams file path.\n+ `use_tensorrt`: use tensorrt to acclerate or not, default: False.",
+ "type": "code",
+ "location": "/english_documents/usage.md:134-174"
+ },
+ "4767": {
+ "file_id": 409,
+ "content": "This code is launching PaddleVideo in distributed mode with four GPUs, running the main.py script with a specified configuration file, and performing testing using weights from a particular path. Then it exports the model for inference by specifying the configuration file, pretrained weights, and output directory. Lastly, it uses the PaddleInference engine to infer a video using the exported model files, input video file, and optional TensorRT acceleration.",
+ "type": "comment"
+ },
+ "4768": {
+ "file_id": 409,
+ "content": "+ `use_gpu`: use gpu to infer or not, default: True.\nbenchmark results are shown in th [benchmark](./benchmark.md).",
+ "type": "code",
+ "location": "/english_documents/usage.md:175-177"
+ },
+ "4769": {
+ "file_id": 409,
+ "content": "This code snippet is referring to the `use_gpu` parameter in PaddleVideo, which enables or disables GPU usage for inferencing. The default setting is set to True and benchmark results are available in the [benchmark](./benchmark.md) document.",
+ "type": "comment"
+ },
+ "4770": {
+ "file_id": 410,
+ "content": "/main.py",
+ "type": "filepath"
+ },
+ "4771": {
+ "file_id": 410,
+ "content": "This code imports libraries, defines functions for PaddleVideo model training, and allows users to specify command-line arguments. It uses the Apache License 2.0 and enables parallel execution with distributed environments. The method to be executed is determined by the command line arguments.",
+ "type": "summary"
+ },
+ "4772": {
+ "file_id": 410,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport argparse\nimport random\nimport numpy as np\nimport paddle\nfrom paddlevideo.tasks import (test_model, train_dali, train_model,\n train_model_multigrid)\nfrom paddlevideo.utils import get_config, get_dist_info\ndef parse_args():\n parser = argparse.ArgumentParser(\"PaddleVideo train script\")\n parser.add_argument('-c',\n '--config',\n type=str,",
+ "type": "code",
+ "location": "/main.py:1-29"
+ },
+ "4773": {
+ "file_id": 410,
+ "content": "This code imports necessary libraries and defines functions for training a PaddleVideo model. It also handles command line arguments using argparse. The script is licensed under the Apache License, Version 2.0.",
+ "type": "comment"
+ },
+ "4774": {
+ "file_id": 410,
+ "content": " default='configs/example.yaml',\n help='config file path')\n parser.add_argument('-o',\n '--override',\n action='append',\n default=[],\n help='config options to be overridden')\n parser.add_argument('--test',\n action='store_true',\n help='whether to test a model')\n parser.add_argument('--train_dali',\n action='store_true',\n help='whether to use dali to speed up training')\n parser.add_argument('--multigrid',\n action='store_true',\n help='whether to use multigrid training')\n parser.add_argument('-w',\n '--weights',\n type=str,\n help='weights for finetuning or testing')\n parser.add_argument('--fleet',\n action='store_true',\n help='whether to use fleet run distributed training')",
+ "type": "code",
+ "location": "/main.py:30-52"
+ },
+ "4775": {
+ "file_id": 410,
+ "content": "This code segment is defining command line arguments using the 'argparse' library for a PaddleVideo program. It allows users to set configuration file paths, override config options, test models, enable DALI for training, use multigrid training, specify weights for finetuning or testing, and utilize fleet run distributed training.",
+ "type": "comment"
+ },
+ "4776": {
+ "file_id": 410,
+ "content": " parser.add_argument('--amp',\n action='store_true',\n help='whether to open amp training.')\n parser.add_argument(\n '--amp_level',\n type=str,\n default=None,\n help=\"optimize level when open amp training, can only be 'O1' or 'O2'.\")\n parser.add_argument(\n '--validate',\n action='store_true',\n help='whether to evaluate the checkpoint during training')\n parser.add_argument(\n '--seed',\n type=int,\n default=1234,\n help='fixed all random seeds when the program is running')\n parser.add_argument(\n '--max_iters',\n type=int,\n default=None,\n help='max iterations when training(this arg only used in test_tipc)')\n parser.add_argument(\n '-p',\n '--profiler_options',\n type=str,\n default=None,\n help='The option of profiler, which should be in format '\n '\\\"key1=value1;key2=value2;key3=value3\\\".')\n args = parser.parse_args()\n return args",
+ "type": "code",
+ "location": "/main.py:53-84"
+ },
+ "4777": {
+ "file_id": 410,
+ "content": "The code adds command-line arguments for AMP (automatic mixed precision) training, validation, random seed, maximum iterations, and profiler options. It then parses these arguments to customize the program's behavior during training.",
+ "type": "comment"
+ },
+ "4778": {
+ "file_id": 410,
+ "content": "def main():\n args = parse_args()\n cfg = get_config(args.config, overrides=args.override)\n # enable to use npu if paddle is built with npu\n if paddle.is_compiled_with_custom_device('npu') :\n cfg.__setattr__(\"use_npu\", True)\n elif paddle.device.is_compiled_with_xpu():\n cfg.__setattr__(\"use_xpu\", True)\n # set seed if specified\n seed = args.seed\n if seed is not None:\n assert isinstance(\n seed, int), f\"seed must be a integer when specified, but got {seed}\"\n random.seed(seed)\n np.random.seed(seed)\n paddle.seed(seed)\n # set amp_level if amp is enabled\n if args.amp:\n if args.amp_level is None:\n args.amp_level = 'O1' # set defaualt amp_level to 'O1'\n else:\n assert args.amp_level in [\n 'O1', 'O2'\n ], f\"amp_level must be 'O1' or 'O2' when amp enabled, but got {args.amp_level}.\"\n _, world_size = get_dist_info()\n parallel = world_size != 1\n if parallel:\n paddle.distributed.init_parallel_env()",
+ "type": "code",
+ "location": "/main.py:87-118"
+ },
+ "4779": {
+ "file_id": 410,
+ "content": "This code snippet defines a `main` function that parses arguments, configures settings based on provided overrides and device availability (NPU or XPU), sets seed for random number generation if specified, and enables parallel execution using Paddle's distributed environment.",
+ "type": "comment"
+ },
+ "4780": {
+ "file_id": 410,
+ "content": " if args.test:\n test_model(cfg, weights=args.weights, parallel=parallel)\n elif args.train_dali:\n train_dali(cfg, weights=args.weights, parallel=parallel)\n elif args.multigrid:\n train_model_multigrid(cfg,\n world_size=world_size,\n validate=args.validate)\n else:\n train_model(cfg,\n weights=args.weights,\n parallel=parallel,\n validate=args.validate,\n use_fleet=args.fleet,\n use_amp=args.amp,\n amp_level=args.amp_level,\n max_iters=args.max_iters,\n profiler_options=args.profiler_options)\nif __name__ == '__main__':\n main()",
+ "type": "code",
+ "location": "/main.py:120-141"
+ },
+ "4781": {
+ "file_id": 410,
+ "content": "This code determines the method to be executed based on command line arguments. If '--test' is given, it executes 'test_model'. If '--train_dali' is given, it executes 'train_dali'. If '--multigrid' is given, it executes 'train_model_multigrid'. Otherwise, it executes 'train_model', passing the necessary parameters to perform model training or validation.",
+ "type": "comment"
+ },
+ "4782": {
+ "file_id": 411,
+ "content": "/paddlevideo/__init__.py",
+ "type": "filepath"
+ },
+ "4783": {
+ "file_id": 411,
+ "content": "This code snippet is importing the paddlevideo_version from the version module. This suggests that this file is serving as an initialization point for the PaddleVideo library, potentially setting up necessary imports or defining constants and functions to be used throughout the library.",
+ "type": "summary"
+ },
+ "4784": {
+ "file_id": 411,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .version import paddlevideo_version",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/__init__.py:1-15"
+ },
+ "4785": {
+ "file_id": 411,
+ "content": "This code snippet is importing the paddlevideo_version from the version module. This suggests that this file is serving as an initialization point for the PaddleVideo library, potentially setting up necessary imports or defining constants and functions to be used throughout the library.",
+ "type": "comment"
+ },
+ "4786": {
+ "file_id": 412,
+ "content": "/paddlevideo/loader/__init__.py",
+ "type": "filepath"
+ },
+ "4787": {
+ "file_id": 412,
+ "content": "This code is part of the PaddleVideo library, containing imports and definitions for functions related to dataset building and data loading. It allows users to build datasets, dataloaders, batch pipelines, and utilize the TSN_Dali_loader and get_input_data functions.",
+ "type": "summary"
+ },
+ "4788": {
+ "file_id": 412,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .builder import build_dataset, build_dataloader, build_batch_pipeline\nfrom .dataset import VideoDataset\nfrom .dali_loader import TSN_Dali_loader, get_input_data\n__all__ = [\n 'build_dataset', 'build_dataloader', 'build_batch_pipeline', 'VideoDataset',\n 'TSN_Dali_loader', 'get_input_data'\n]",
+ "type": "code",
+ "location": "/paddlevideo/loader/__init__.py:1-22"
+ },
+ "4789": {
+ "file_id": 412,
+ "content": "This code is part of the PaddleVideo library, containing imports and definitions for functions related to dataset building and data loading. It allows users to build datasets, dataloaders, batch pipelines, and utilize the TSN_Dali_loader and get_input_data functions.",
+ "type": "comment"
+ },
+ "4790": {
+ "file_id": 413,
+ "content": "/paddlevideo/loader/builder.py",
+ "type": "filepath"
+ },
+ "4791": {
+ "file_id": 413,
+ "content": "The code constructs a PaddleVideo pipeline for preprocessing data and builds a data loader for distributed model training, handling variable batch sizes and using mix_collate_fn to collate data.",
+ "type": "summary"
+ },
+ "4792": {
+ "file_id": 413,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport signal\nimport os\nimport paddle\nfrom paddle.io import DataLoader, DistributedBatchSampler\nfrom .registry import DATASETS, PIPELINES\nfrom ..utils.build_utils import build\nfrom .pipelines.compose import Compose\nfrom paddlevideo.utils import get_logger\nfrom paddlevideo.utils.multigrid import DistributedShortSampler\nimport numpy as np\nlogger = get_logger(\"paddlevideo\")\ndef build_pipeline(cfg):\n \"\"\"Build pipeline.",
+ "type": "code",
+ "location": "/paddlevideo/loader/builder.py:1-29"
+ },
+ "4793": {
+ "file_id": 413,
+ "content": "The code is building a pipeline for PaddleVideo. It imports necessary libraries and classes, utilizes function to build the pipeline, logs information using get_logger from paddlevideo.utils, and adheres to Apache License 2.0. The purpose of this code seems to be related to data preprocessing and possibly model training in a distributed environment.",
+ "type": "comment"
+ },
+ "4794": {
+ "file_id": 413,
+ "content": " Args:\n cfg (dict): root config dict.\n \"\"\"\n if cfg == None:\n return\n return Compose(cfg)\ndef build_dataset(cfg):\n \"\"\"Build dataset.\n Args:\n cfg (dict): root config dict.\n Returns:\n dataset: dataset.\n \"\"\"\n #XXX: ugly code here!\n cfg_dataset, cfg_pipeline = cfg\n cfg_dataset.pipeline = build_pipeline(cfg_pipeline)\n dataset = build(cfg_dataset, DATASETS, key=\"format\")\n return dataset\ndef build_batch_pipeline(cfg):\n batch_pipeline = build(cfg, PIPELINES)\n return batch_pipeline\ndef build_dataloader(dataset,\n batch_size,\n num_workers,\n places,\n shuffle=True,\n drop_last=True,\n multigrid=False,\n collate_fn_cfg=None,\n **kwargs):\n \"\"\"Build Paddle Dataloader.\n XXX explain how the dataloader work!\n Args:\n dataset (paddle.dataset): A PaddlePaddle dataset object.\n batch_size (int): batch size on single card.",
+ "type": "code",
+ "location": "/paddlevideo/loader/builder.py:30-74"
+ },
+ "4795": {
+ "file_id": 413,
+ "content": "build_dataset: Builds a dataset using provided config dictionary, building pipeline first.\nbuild_batch_pipeline: Constructs the batch pipeline using config from the PIPELINES module.\nbuild_dataloader: Creates Paddle Dataloader object using specified parameters and dataset.",
+ "type": "comment"
+ },
+ "4796": {
+ "file_id": 413,
+ "content": " num_worker (int): num_worker\n shuffle(bool): whether to shuffle the data at every epoch.\n \"\"\"\n if multigrid:\n sampler = DistributedShortSampler(dataset,\n batch_sizes=batch_size,\n shuffle=True,\n drop_last=True)\n else:\n sampler = DistributedBatchSampler(dataset,\n batch_size=batch_size,\n shuffle=shuffle,\n drop_last=drop_last)\n #NOTE(shipping): when switch the mix operator on, such as: mixup, cutmix.\n # batch like: [[img, label, attibute, ...], [imgs, label, attribute, ...], ...] will recollate to:\n # [[img, img, ...], [label, label, ...], [attribute, attribute, ...], ...] as using numpy.transpose.\n def mix_collate_fn(batch):\n pipeline = build_batch_pipeline(collate_fn_cfg)\n batch = pipeline(batch)\n slots = []",
+ "type": "code",
+ "location": "/paddlevideo/loader/builder.py:75-96"
+ },
+ "4797": {
+ "file_id": 413,
+ "content": "This code appears to be part of a data loading and processing function for a machine learning or deep learning model. It uses a sampler to manage the data, with options for shuffling and dropping the last batch if needed. The mix_collate_fn function is defined to collate the data in a specific way using a pipeline built from collate_fn_cfg.",
+ "type": "comment"
+ },
+ "4798": {
+ "file_id": 413,
+ "content": " for items in batch:\n for i, item in enumerate(items):\n if len(slots) < len(items):\n slots.append([item])\n else:\n slots[i].append(item)\n return [np.stack(slot, axis=0) for slot in slots]\n #if collate_fn_cfg is not None:\n #ugly code here. collate_fn is mix op config\n # collate_fn = mix_collate_fn(collate_fn_cfg)\n data_loader = DataLoader(\n dataset,\n batch_sampler=sampler,\n places=places,\n num_workers=num_workers,\n collate_fn=mix_collate_fn if collate_fn_cfg is not None else None,\n return_list=True,\n **kwargs)\n return data_loader\ndef term_mp(sig_num, frame):\n \"\"\" kill all child processes\n \"\"\"\n pid = os.getpid()\n pgid = os.getpgid(os.getpid())\n logger.info(\"main proc {} exit, kill process group \" \"{}\".format(pid, pgid))\n os.killpg(pgid, signal.SIGKILL)\n return\nsignal.signal(signal.SIGINT, term_mp)\nsignal.signal(signal.SIGTERM, term_mp)",
+ "type": "code",
+ "location": "/paddlevideo/loader/builder.py:97-132"
+ },
+ "4799": {
+ "file_id": 413,
+ "content": "This code appears to create a data loader that can handle batches of varying lengths. It iterates through each batch and organizes the items into slots based on their length, either creating a new slot for longer items or appending them to existing ones. The DataLoader class is then instantiated with this collate_fn for processing the dataset, using the provided parameters. Additionally, signal handlers are set up to handle SIGINT and SIGTERM signals to terminate the process group if needed.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/48.json b/docs/data/48.json
new file mode 100644
index 000000000..189ad2c00
--- /dev/null
+++ b/docs/data/48.json
@@ -0,0 +1,543 @@
+{
+ "4800": {
+ "file_id": 414,
+ "content": "/paddlevideo/loader/dali_loader.py",
+ "type": "filepath"
+ },
+ "4801": {
+ "file_id": 414,
+ "content": "The code imports necessary libraries, sets up a DALI reader, defines TSN_Dali_loader class, initializes parallel video preprocessing, handles potential import errors, and returns output and label using PaddleOps for normalization.",
+ "type": "summary"
+ },
+ "4802": {
+ "file_id": 414,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport math\nimport paddle\nfrom paddle.distributed import ParallelEnv\nimport paddle.distributed as dist\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ntry:\n from nvidia.dali.pipeline import Pipeline\n import nvidia.dali.ops as ops\n import nvidia.dali.types as types\n import tempfile\n from nvidia.dali.plugin.paddle import DALIGenericIterator\nexcept:\n Pipeline = object",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:1-32"
+ },
+ "4803": {
+ "file_id": 414,
+ "content": "This code imports necessary libraries, sets up logger, and attempts to import DALI pipeline and related functions for creating a generic iterator for PaddlePaddle. If any of these imports fail, it falls back by setting the respective variable as an object.",
+ "type": "comment"
+ },
+ "4804": {
+ "file_id": 414,
+ "content": "def get_input_data(data):\n return paddle.to_tensor(data[0]['image']), paddle.to_tensor(\n data[0]['label'])\nclass TSN_Dali_loader(object):\n def __init__(self, cfg):\n self.batch_size = cfg.batch_size\n self.file_path = cfg.file_path\n self.num_seg = cfg.num_seg\n self.seglen = cfg.seglen\n self.short_size = cfg.short_size\n self.target_size = cfg.target_size\n # set num_shards and shard_id when distributed training is implemented\n self.num_shards = dist.get_world_size()\n self.shard_id = ParallelEnv().local_rank\n self.dali_mean = cfg.mean * (self.num_seg * self.seglen)\n self.dali_std = cfg.std * (self.num_seg * self.seglen)\n def build_dali_reader(self):\n \"\"\"\n build dali training reader\n \"\"\"\n def reader_():\n with open(self.file_path) as flist:\n full_lines = [line for line in flist]\n if (not hasattr(reader_, 'seed')):\n reader_.seed = 0\n random.Random(reader_.seed).shuffle(full_lines)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:35-65"
+ },
+ "4805": {
+ "file_id": 414,
+ "content": "The code defines a class `TSN_Dali_loader` that initializes attributes related to batch size, file path, number of segments, segment length, input and target image sizes. It also sets variables for distributed training, data normalization, and builds a DALI reader for training data using shuffled full lines from the file path.",
+ "type": "comment"
+ },
+ "4806": {
+ "file_id": 414,
+ "content": " logger.info(f\"reader shuffle seed: {reader_.seed}.\")\n if reader_.seed is not None:\n reader_.seed += 1\n per_node_lines = int(\n math.ceil(len(full_lines) * 1.0 / self.num_shards))\n total_lines = per_node_lines * self.num_shards\n # aligned full_lines so that it can evenly divisible\n full_lines += full_lines[:(total_lines - len(full_lines))]\n assert len(full_lines) == total_lines\n # trainer get own sample\n lines = full_lines[self.shard_id:total_lines:self.num_shards]\n assert len(lines) == per_node_lines\n logger.info(\n f\"shard_id: {self.shard_id}, trainer_count: {self.num_shards}\"\n )\n logger.info(\n f\"read videos from {self.shard_id * per_node_lines}, \"\n f\"length: {per_node_lines}, \"\n f\"lines length: {len(lines)}, \"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:66-88"
+ },
+ "4807": {
+ "file_id": 414,
+ "content": "This code snippet initializes a reader and distributes the data evenly across multiple shards. It calculates the number of lines to be assigned to each shard based on the total number of lines and the number of shards. It then ensures that the full_lines list is an even multiple of the total_lines by appending additional items if necessary. The snippet asserts that the length of full_lines equals total_lines, assigns lines to trainers based on their shard ID, and logs information about the distribution of data among shards.",
+ "type": "comment"
+ },
+ "4808": {
+ "file_id": 414,
+ "content": " f\"total: {len(full_lines)}\")\n video_files = ''.join([item for item in lines])\n tf = tempfile.NamedTemporaryFile()\n tf.write(str.encode(video_files))\n tf.flush()\n video_files = tf.name\n device_id = ParallelEnv().local_rank\n logger.info(f'---------- device_id: {device_id} -----------')\n pipe = VideoPipe(batch_size=self.batch_size,\n num_threads=1,\n device_id=device_id,\n file_list=video_files,\n sequence_length=self.num_seg * self.seglen,\n num_seg=self.num_seg,\n seg_length=self.seglen,\n resize_shorter_scale=self.short_size,\n crop_target_size=self.target_size,\n is_training=True,\n num_shards=self.num_shards,\n shard_id=self.shard_id,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:89-111"
+ },
+ "4809": {
+ "file_id": 414,
+ "content": "This code initializes a PaddlePaddle VideoPipe instance, loading and preprocessing video files in parallel for training. It sets the batch size, number of threads, device ID, file list, sequence length, number of segments, segment length, resize shorter scale, crop target size, whether it's in training mode, and the number of shards and shard ID.",
+ "type": "comment"
+ },
+ "4810": {
+ "file_id": 414,
+ "content": " dali_mean=self.dali_mean,\n dali_std=self.dali_std)\n logger.info(\n 'initializing dataset, it will take several minutes if it is too large .... '\n )\n video_loader = DALIGenericIterator([pipe], ['image', 'label'],\n len(lines),\n dynamic_shape=True,\n auto_reset=True)\n return video_loader\n dali_reader = reader_()\n return dali_reader\nclass VideoPipe(Pipeline):\n def __init__(self,\n batch_size,\n num_threads,\n device_id,\n file_list,\n sequence_length,\n num_seg,\n seg_length,\n resize_shorter_scale,\n crop_target_size,\n is_training=False,\n initial_prefetch_size=20,\n num_shards=1,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:112-142"
+ },
+ "4811": {
+ "file_id": 414,
+ "content": "This code initializes a DALI (Data Augmentation Library for Images) generic iterator to load video data from a file list, and returns it. It uses a VideoPipe class to define the pipeline configuration, including parameters such as batch size, number of threads, device ID, sequence length, and more.",
+ "type": "comment"
+ },
+ "4812": {
+ "file_id": 414,
+ "content": " shard_id=0,\n dali_mean=0.,\n dali_std=1.0):\n super(VideoPipe, self).__init__(batch_size, num_threads, device_id)\n self.input = ops.VideoReader(device=\"gpu\",\n file_list=file_list,\n sequence_length=sequence_length,\n num_seg=num_seg,\n seg_length=seg_length,\n is_training=is_training,\n num_shards=num_shards,\n shard_id=shard_id,\n random_shuffle=is_training,\n initial_fill=initial_prefetch_size)\n # the sequece data read by ops.VideoReader is of shape [F, H, W, C]\n # Because the ops.Resize does not support sequence data,\n # it will be transposed into [H, W, F, C],\n # then reshaped to [H, W, FC], and then resized like a 2-D image.",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:143-160"
+ },
+ "4813": {
+ "file_id": 414,
+ "content": "This code initializes a VideoPipe object with the given parameters, including file list, sequence length, and number of segments. It uses ops.VideoReader to read video data from the file list in the specified format. Due to the limitations of resize function, it transposes and reshapes the data before performing resizing operation on the 2-D image.",
+ "type": "comment"
+ },
+ "4814": {
+ "file_id": 414,
+ "content": " self.transpose = ops.Transpose(device=\"gpu\", perm=[1, 2, 0, 3])\n self.reshape = ops.Reshape(device=\"gpu\",\n rel_shape=[1.0, 1.0, -1],\n layout='HWC')\n self.resize = ops.Resize(device=\"gpu\",\n resize_shorter=resize_shorter_scale)\n # crops and mirror are applied by ops.CropMirrorNormalize.\n # Normalization will be implemented in paddle due to the difficulty of dimension broadcast,\n # It is not sure whether dimension broadcast can be implemented correctly by dali, just take the Paddle Op instead.\n self.pos_rng_x = ops.Uniform(range=(0.0, 1.0))\n self.pos_rng_y = ops.Uniform(range=(0.0, 1.0))\n self.mirror_generator = ops.Uniform(range=(0.0, 1.0))\n self.cast_mirror = ops.Cast(dtype=types.DALIDataType.INT32)\n self.crop_mirror_norm = ops.CropMirrorNormalize(\n device=\"gpu\",\n crop=[crop_target_size, crop_target_size],",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:161-176"
+ },
+ "4815": {
+ "file_id": 414,
+ "content": "The code creates a DALI loader for image processing, with transpose, reshape, resize operations, and implements crop and mirror normalization. It also includes uniform distribution generators for position and mirror. The normalization will be implemented using PaddleOps due to the difficulty of dimension broadcasting in DALI.",
+ "type": "comment"
+ },
+ "4816": {
+ "file_id": 414,
+ "content": " mean=dali_mean,\n std=dali_std)\n self.reshape_back = ops.Reshape(\n device=\"gpu\",\n shape=[num_seg, seg_length * 3, crop_target_size, crop_target_size],\n layout='FCHW')\n self.cast_label = ops.Cast(device=\"gpu\", dtype=types.DALIDataType.INT64)\n def define_graph(self):\n output, label = self.input(name=\"Reader\")\n output = self.transpose(output)\n output = self.reshape(output)\n output = self.resize(output)\n output = output / 255.\n pos_x = self.pos_rng_x()\n pos_y = self.pos_rng_y()\n mirror_flag = self.mirror_generator()\n mirror_flag = (mirror_flag > 0.5)\n mirror_flag = self.cast_mirror(mirror_flag)\n output = self.crop_mirror_norm(output,\n crop_pos_x=pos_x,\n crop_pos_y=pos_y,\n mirror=mirror_flag)\n output = self.reshape_back(output)\n label = self.cast_label(label)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:177-202"
+ },
+ "4817": {
+ "file_id": 414,
+ "content": "The code defines a DALI loader and its associated operations for image processing. It includes mean and std for normalization, reshaping, casting to int64, transpose, resize, normalization by dividing by 255, generating positional information, cropping with mirror flag, and finally reshaping the output.",
+ "type": "comment"
+ },
+ "4818": {
+ "file_id": 414,
+ "content": " return output, label\n def __len__(self):\n return self.epoch_size()",
+ "type": "code",
+ "location": "/paddlevideo/loader/dali_loader.py:203-206"
+ },
+ "4819": {
+ "file_id": 414,
+ "content": "The code defines a method that returns an output and label, and another method for determining the length of the loader.",
+ "type": "comment"
+ },
+ "4820": {
+ "file_id": 415,
+ "content": "/paddlevideo/loader/dataset/MRI.py",
+ "type": "filepath"
+ },
+ "4821": {
+ "file_id": 415,
+ "content": "The MRI.py file in PaddleVideo library provides an action recognition dataset loader, utilizing a MRIDataset class for transform operations on raw frames and includes license information, copyright notices, and data structure registration. It reads data, stores components in a list, handles missing files through retry and exception handling, and logs errors. The code snippet returns a numpy array for images and another for labels from the 'results' dictionary, likely used in a function that processes data from MRI datasets where 'imgs' contains image data and 'labels' stores their corresponding labels or annotations.",
+ "type": "summary"
+ },
+ "4822": {
+ "file_id": 415,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass MRIDataset(BaseDataset):\n \"\"\"Rawframe dataset for action recognition.\n The dataset loads raw frames from frame files, and apply specified transform operatation them.\n The indecx file is",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI.py:1-31"
+ },
+ "4823": {
+ "file_id": 415,
+ "content": "The code snippet is from the MRI.py file within the PaddleVideo library, which appears to be a loader dataset for action recognition tasks. It imports necessary libraries and defines the MRIDataset class that inherits from BaseDataset. This class loads raw frames from frame files and applies specified transform operations on them. The index file is used by the dataset loader. The code also includes license information, copyright notices, and data structure registration.",
+ "type": "comment"
+ },
+ "4824": {
+ "file_id": 415,
+ "content": " a text file with multiple lines, and each line indicates the directory of frames of a video, toatl frames of the video, and its label, which split with a whitespace.\n Example of an index file:\n .. code-block:: txt\n file_path-1 150 1\n file_path-2 160 1\n file_path-3 170 2\n file_path-4 180 2\n Args:\n file_path (str): Path to the index file.\n pipeline(XXX):\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n suffix (str): suffix of file. Default: 'img_{:05}.jpg'.\n \"\"\"\n def __init__(self,\n file_path,\n pipeline,\n num_retries=5,\n data_prefix=None,\n test_mode=False,\n suffix='img_{:05}.jpg'):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI.py:31-61"
+ },
+ "4825": {
+ "file_id": 415,
+ "content": "This function initializes the MRI dataset object, taking the file path to the index file as well as other optional arguments. The load_file method is used to load the index file and retrieve video information.",
+ "type": "comment"
+ },
+ "4826": {
+ "file_id": 415,
+ "content": " info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split()\n frame_dir, frames_len, labels = line_split\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n info.append(\n dict(\n frame_dir=frame_dir,\n #suffix=self.suffix,\n frames_len=frames_len,\n labels=int(labels)))\n return info\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training/valid gisven index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI.py:62-86"
+ },
+ "4827": {
+ "file_id": 415,
+ "content": "This code reads data from a file, splits it into different components like frame directory, frames length, and labels, and stores it in a list. It also handles missing files by retrying multiple times using exception handling.",
+ "type": "comment"
+ },
+ "4828": {
+ "file_id": 415,
+ "content": " logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return np.array(results['imgs']), np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for test given index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI.py:87-108"
+ },
+ "4829": {
+ "file_id": 415,
+ "content": "The code is attempting to load frames for testing by trying multiple times in case of an exception caused by missing frames. It uses a logger to inform about the error and tries again with different frames until successful or reaching the maximum retries.",
+ "type": "comment"
+ },
+ "4830": {
+ "file_id": 415,
+ "content": " return np.array(results['imgs']), np.array([results['labels']])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI.py:109-109"
+ },
+ "4831": {
+ "file_id": 415,
+ "content": "The code snippet returns a numpy array for images and another for labels from the 'results' dictionary. It is likely used in a function that processes data from MRI datasets, where 'imgs' contains image data and 'labels' stores their corresponding labels or annotations.",
+ "type": "comment"
+ },
+ "4832": {
+ "file_id": 416,
+ "content": "/paddlevideo/loader/dataset/MRI_SlowFast.py",
+ "type": "filepath"
+ },
+ "4833": {
+ "file_id": 416,
+ "content": "This code imports libraries, creates action recognition and data loading classes, and processes video data for training or validation using a pipeline, handling exceptions through retries and logging. It is part of a function that returns arrays of images and labels.",
+ "type": "summary"
+ },
+ "4834": {
+ "file_id": 416,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass SFMRIDataset(BaseDataset):\n \"\"\"Rawframe dataset for action recognition.\n The dataset loads raw frames from frame files, and apply specified transform operatation them.\n The indecx file ",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI_SlowFast.py:1-31"
+ },
+ "4835": {
+ "file_id": 416,
+ "content": "This code snippet is importing necessary libraries and registering a new dataset class named SFMRIDataset for action recognition. It uses raw frames from frame files, applies specified transform operations, and loads an index file. The copyright and license information are also included in the beginning of the file.",
+ "type": "comment"
+ },
+ "4836": {
+ "file_id": 416,
+ "content": "is a text file with multiple lines, and each line indicates the directory of frames of a video, toatl frames of the video, and its label, which split with a whitespace.\n Example of an index file:\n .. code-block:: txt\n file_path-1 150 1\n file_path-2 160 1\n file_path-3 170 2\n file_path-4 180 2\n Args:\n file_path (str): Path to the index file.\n pipeline(XXX):\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n suffix (str): suffix of file. Default: 'img_{:05}.jpg'.\n \"\"\"\n def __init__(self,\n file_path,\n pipeline,\n num_retries=5,\n data_prefix=None,\n test_mode=False,\n suffix='img_{:05}.jpg'):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI_SlowFast.py:31-61"
+ },
+ "4837": {
+ "file_id": 416,
+ "content": "This code is creating a class for loading an index file containing video information, including the directory of frames, total frames, and label. The constructor takes arguments like the file path, pipeline, data prefix, test mode, and suffix. The load_file function loads the index file to retrieve the video details.",
+ "type": "comment"
+ },
+ "4838": {
+ "file_id": 416,
+ "content": " info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split()\n frame_dir, frames_len, labels = line_split\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n info.append(\n dict(\n frame_dir=frame_dir,\n #suffix=self.suffix,\n frames_len=frames_len,\n labels=int(labels)))\n return info\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training/valid gisven index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI_SlowFast.py:62-86"
+ },
+ "4839": {
+ "file_id": 416,
+ "content": "The code reads information from a file and stores it in a list of dictionaries. It then attempts to prepare the frames for training or validation by applying a pipeline, handling potential exceptions within a specified number of retries.",
+ "type": "comment"
+ },
+ "4840": {
+ "file_id": 416,
+ "content": " logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return np.array(results['imgs'][0]), np.array(\n results['imgs'][1]), np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for test given index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI_SlowFast.py:87-108"
+ },
+ "4841": {
+ "file_id": 416,
+ "content": "This code handles error cases when loading data by retrying the operation if an exception occurs. It uses a logger to provide information on the error, the number of retries, and whether or not to try again with a different index. The 'prepare_test' function is responsible for preparing frames for testing.",
+ "type": "comment"
+ },
+ "4842": {
+ "file_id": 416,
+ "content": " continue\n return np.array(results['imgs'][0]), np.array(\n results['imgs'][1]), np.array([results['labels']])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/MRI_SlowFast.py:109-111"
+ },
+ "4843": {
+ "file_id": 416,
+ "content": "The code is part of a function that returns three arrays: the first image from the 'imgs' list, the second image, and the labels. If there are more images available, the function continues processing them; if not, it returns the stored images and labels.",
+ "type": "comment"
+ },
+ "4844": {
+ "file_id": 417,
+ "content": "/paddlevideo/loader/dataset/__init__.py",
+ "type": "filepath"
+ },
+ "4845": {
+ "file_id": 417,
+ "content": "The code imports various dataset classes from PaddleVideo library for video understanding tasks, and adds them to the `__all__` list for accessibility. These datasets include VideoDataset, FrameDataset, and more, with licensing information provided.",
+ "type": "summary"
+ },
+ "4846": {
+ "file_id": 417,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .actbert_dataset import ActBertDataset\nfrom .ava_dataset import AVADataset\nfrom .bmn_dataset import BMNDataset\nfrom .davis_dataset import DavisDataset\nfrom .feature import FeatureDataset\nfrom .frame import FrameDataset, FrameDataset_Sport\nfrom .MRI import MRIDataset\nfrom .MRI_SlowFast import SFMRIDataset\nfrom .msrvtt import MSRVTTDataset\nfrom .actbert_dataset import ActBertDataset\nfrom .asrf_dataset import ASRFDataset",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/__init__.py:1-25"
+ },
+ "4847": {
+ "file_id": 417,
+ "content": "This code is importing various dataset classes from different modules in the PaddleVideo library. These datasets are used for video understanding tasks, such as action recognition, activity classification, and video captioning. The code also includes licensing information and mentions that these datasets can be accessed on an \"AS IS\" basis.",
+ "type": "comment"
+ },
+ "4848": {
+ "file_id": 417,
+ "content": "from .ms_tcn_dataset import MSTCNDataset\nfrom .oxford import MonoDataset\nfrom .skeleton import SkeletonDataset\nfrom .slowfast_video import SFVideoDataset\nfrom .video import VideoDataset\nfrom .ucf101_skeleton import UCF101SkeletonDataset\nfrom .ucf24_dataset import UCF24Dataset\n__all__ = [\n 'VideoDataset', 'FrameDataset', 'SFVideoDataset', 'BMNDataset',\n 'FeatureDataset', 'SkeletonDataset', 'AVADataset', 'MonoDataset',\n 'MSRVTTDataset', 'ActBertDataset', 'DavisDataset', 'MRIDataset',\n 'SFMRIDataset', 'FrameDataset_Sport', 'MSTCNDataset', 'ASRFDataset',\n 'UCF101SkeletonDataset', 'UCF24Dataset'\n]",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/__init__.py:26-41"
+ },
+ "4849": {
+ "file_id": 417,
+ "content": "This code imports several dataset classes and adds them to the `__all__` list, making them accessible within this module. The datasets include VideoDataset, FrameDataset, SFVideoDataset, BMNDataset, FeatureDataset, SkeletonDataset, AVADataset, MonoDataset, MSRVTTDataset, ActBertDataset, DavisDataset, MRIDataset, SFMRIDataset, FrameDataset_Sport, MSTCNDataset, ASRFDataset, UCF101SkeletonDataset, and UCF24Dataset.",
+ "type": "comment"
+ },
+ "4850": {
+ "file_id": 418,
+ "content": "/paddlevideo/loader/dataset/actbert_dataset.py",
+ "type": "filepath"
+ },
+ "4851": {
+ "file_id": 418,
+ "content": "The code sets up ActBERT dataset in PaddlePaddle's video processing library, initializing the dataset with necessary libraries and packages. It defines two methods: \"prepare_train\" for preparing frames for training and a placeholder \"prepare_test\".",
+ "type": "summary"
+ },
+ "4852": {
+ "file_id": 418,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\ntry:\n import lmdb\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [lmdb] package and it's dependencies is required for ActBERT.\"\n )\nimport pickle\nimport json\ntry:\n from paddlenlp.transformers import BertTokenizer\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [paddlenlp] package and it's dependencies is required for ActBERT.\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/actbert_dataset.py:1-31"
+ },
+ "4853": {
+ "file_id": 418,
+ "content": "This code is importing necessary libraries and packages, checking for missing dependencies, and setting up the ActBERT dataset in PaddlePaddle's video processing library.",
+ "type": "comment"
+ },
+ "4854": {
+ "file_id": 418,
+ "content": " )\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass ActBertDataset(BaseDataset):\n \"\"\"ActBert dataset.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n bert_model=\"bert-base-uncased\",\n data_prefix=None,\n test_mode=False,\n ):\n self.bert_model = bert_model\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n feature_data = np.load(self.file_path, allow_pickle=True)\n self.tokenizer = BertTokenizer.from_pretrained(self.bert_model,\n do_lower_case=True)\n self.info = []\n for item in feature_data:\n self.info.append(dict(feature=item, tokenizer=self.tokenizer))\n return self.info\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training/valid given index. \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/actbert_dataset.py:32-66"
+ },
+ "4855": {
+ "file_id": 418,
+ "content": "Class ActBertDataset is a dataset for PaddleVideo, initialized with file path, pipeline, bert_model, data_prefix and test mode. It loads the index file to get video information, uses the tokenizer from pre-trained bert model, and stores information in the info list. The load_file method is used to load the feature data and prepare the dataset for training or validation.",
+ "type": "comment"
+ },
+ "4856": {
+ "file_id": 418,
+ "content": " results = copy.deepcopy(self.info[idx])\n #print('==results==', results)\n results = self.pipeline(results)\n return results['features']\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for test given index. \"\"\"\n pass",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/actbert_dataset.py:67-74"
+ },
+ "4857": {
+ "file_id": 418,
+ "content": "This code defines two methods: \"prepare_train\" and \"prepare_test\". The former prepares the frames for training given an index by creating a deep copy of info at that index, applies the pipeline to it, and returns the features from the result. The latter is a placeholder method with no implementation.",
+ "type": "comment"
+ },
+ "4858": {
+ "file_id": 419,
+ "content": "/paddlevideo/loader/dataset/asrf_dataset.py",
+ "type": "filepath"
+ },
+ "4859": {
+ "file_id": 419,
+ "content": "This PaddleVideo library code initializes a dataset class for action segmentation videos, includes methods to load data for training/validation, and loads video features, labels, and boundaries using a pipeline.",
+ "type": "summary"
+ },
+ "4860": {
+ "file_id": 419,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport os\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass ASRFDataset(BaseDataset):\n \"\"\"Video dataset for action segmentation.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n feature_path,\n label_path,\n boundary_path,\n **kwargs,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/asrf_dataset.py:1-38"
+ },
+ "4861": {
+ "file_id": 419,
+ "content": "The code imports necessary libraries and defines the ASRFDataset class for action segmentation video datasets. It registers this dataset with the DATASETS registry and initializes the dataset with specified file paths and pipeline parameters.",
+ "type": "comment"
+ },
+ "4862": {
+ "file_id": 419,
+ "content": " ):\n super().__init__(file_path, pipeline, **kwargs)\n self.label_path = label_path\n self.boundary_path = boundary_path\n self.feature_path = feature_path\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n file_ptr = open(self.file_path, 'r')\n info = file_ptr.read().split('\\n')[:-1]\n file_ptr.close()\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID: Prepare data for training/valid given the index.\"\"\"\n results = {}\n video_name = self.info[idx]\n # load video feature\n file_name = video_name.split('.')[0] + \".npy\"\n feat_file_path = os.path.join(self.feature_path, file_name)\n #TODO: check path\n video_feat = np.load(feat_file_path)\n # load label\n file_name = video_name.split('.')[0] + \".npy\"\n label_file_path = os.path.join(self.label_path, file_name)\n label = np.load(label_file_path).astype(np.int64)\n # load boundary\n file_name = video_name.split('.')[0] + \".npy\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/asrf_dataset.py:39-68"
+ },
+ "4863": {
+ "file_id": 419,
+ "content": "The code initializes an instance of a dataset class with file paths for labels, boundaries, and features. It defines methods to load index files containing video information and prepare data for training/validation, including loading video features, labels, and boundaries based on the given index.",
+ "type": "comment"
+ },
+ "4864": {
+ "file_id": 419,
+ "content": " boundary_file_path = os.path.join(self.boundary_path, file_name)\n boundary = np.expand_dims(np.load(boundary_file_path),axis=0).astype(np.float32)\n results['video_feat'] = copy.deepcopy(video_feat)\n results['video_label'] = copy.deepcopy(label)\n results['video_boundary'] = copy.deepcopy(boundary)\n results = self.pipeline(results)\n return results['video_feat'], results['video_label'], results['video_boundary']\n def prepare_test(self, idx):\n \"\"\"TEST: Prepare the data for test given the index.\"\"\"\n results = {}\n video_name = self.info[idx]\n # load video feature\n file_name = video_name.split('.')[0] + \".npy\"\n feat_file_path = os.path.join(self.feature_path, file_name)\n #TODO: check path\n video_feat = np.load(feat_file_path)\n # load label\n file_name = video_name.split('.')[0] + \".npy\"\n label_file_path = os.path.join(self.label_path, file_name)\n label = np.load(label_file_path).astype(np.int64)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/asrf_dataset.py:69-92"
+ },
+ "4865": {
+ "file_id": 419,
+ "content": "The code above is from a dataset loader class in the PaddleVideo library. It loads video features, labels, and boundaries for either training or testing data. The prepare_test function loads video features and labels given an index. The code uses numpy to load data from specified file paths and deepcopy the results for further processing by the pipeline function.",
+ "type": "comment"
+ },
+ "4866": {
+ "file_id": 419,
+ "content": " # load boundary\n file_name = video_name.split('.')[0] + \".npy\"\n boundary_file_path = os.path.join(self.boundary_path, file_name)\n boundary = np.expand_dims(np.load(boundary_file_path),axis=0).astype(np.float32)\n results['video_feat'] = copy.deepcopy(video_feat)\n results['video_label'] = copy.deepcopy(label)\n results['video_boundary'] = copy.deepcopy(boundary)\n results = self.pipeline(results)\n return results['video_feat'], results['video_label'], results['video_boundary']",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/asrf_dataset.py:94-104"
+ },
+ "4867": {
+ "file_id": 419,
+ "content": "This code snippet loads the boundary data for a video, reads it from a file using numpy's load function, and assigns it to a variable named 'boundary'. The code then creates a results dictionary, copies video features, labels, and boundaries into their respective keys in the results dictionary. Finally, it passes this dictionary through a pipeline and returns the video features, labels, and boundaries.",
+ "type": "comment"
+ },
+ "4868": {
+ "file_id": 420,
+ "content": "/paddlevideo/loader/dataset/ava_dataset.py",
+ "type": "filepath"
+ },
+ "4869": {
+ "file_id": 420,
+ "content": "The code introduces a spatial-temporal detection dataset class in PaddleVideo, initializes attributes and evaluation functions, loads records from paths, prepares training data by filtering proposals and annotations, pads elements to fixed lengths, and defines methods for padding 2D/1D features.",
+ "type": "summary"
+ },
+ "4870": {
+ "file_id": 420,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport sys\nimport os\nimport pickle\nfrom datetime import datetime\nfrom ...metrics.ava_utils import ava_evaluate_results\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom collections import defaultdict\n@DATASETS.register()\nclass AVADataset(BaseDataset):\n \"\"\"AVA dataset for spatial temporal detection.\n the dataset loads raw frames, bounding boxes, proposals and applies",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:1-32"
+ },
+ "4871": {
+ "file_id": 420,
+ "content": "This code snippet is the AVA dataset class for spatial-temporal detection, which is part of PaddleVideo. It imports necessary modules and registers the dataset in the DATASETS registry. The class inherits from BaseDataset and includes a function ava_evaluate_results for evaluation.",
+ "type": "comment"
+ },
+ "4872": {
+ "file_id": 420,
+ "content": " transformations to return the frame tensors and other information.\n \"\"\"\n _FPS = 30\n def __init__(self,\n pipeline,\n file_path=None,\n exclude_file=None,\n label_file=None,\n suffix='{:05}.jpg',\n proposal_file=None,\n person_det_score_thr=0.9,\n num_classes=81,\n data_prefix=None,\n test_mode=False,\n num_max_proposals=1000,\n timestamp_start=900,\n timestamp_end=1800):\n self.custom_classes = None\n self.exclude_file = exclude_file\n self.label_file = label_file\n self.proposal_file = proposal_file\n assert 0 <= person_det_score_thr <= 1, (\n 'The value of '\n 'person_det_score_thr should in [0, 1]. ')\n self.person_det_score_thr = person_det_score_thr\n self.num_classes = num_classes\n self.suffix = suffix\n self.num_max_proposals = num_max_proposals",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:33-62"
+ },
+ "4873": {
+ "file_id": 420,
+ "content": "This code is initializing a class with various parameters for the AvaDataset. It sets default values and performs checks on input values, such as ensuring 'person_det_score_thr' falls within 0 to 1 range. The code also initializes instance variables, including custom classes, exclude file path, label file path, proposal file path, and more.",
+ "type": "comment"
+ },
+ "4874": {
+ "file_id": 420,
+ "content": " self.timestamp_start = timestamp_start\n self.timestamp_end = timestamp_end\n super().__init__(\n file_path,\n pipeline,\n data_prefix,\n test_mode,\n )\n if self.proposal_file is not None:\n self.proposals = self._load(self.proposal_file)\n else:\n self.proposals = None\n if not test_mode:\n valid_indexes = self.filter_exclude_file()\n self.info = self.info = [self.info[i] for i in valid_indexes]\n def _load(self, path):\n f = open(path, 'rb')\n res = pickle.load(f)\n f.close()\n return res\n def parse_img_record(self, img_records):\n bboxes, labels, entity_ids = [], [], []\n while len(img_records) > 0:\n img_record = img_records[0]\n num_img_records = len(img_records)\n selected_records = list(\n filter(\n lambda x: np.array_equal(x['entity_box'], img_record[\n 'entity_box']), img_records))",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:63-93"
+ },
+ "4875": {
+ "file_id": 420,
+ "content": "The code snippet initializes class attributes and checks for proposal file. If the proposal file exists, it loads the proposals; otherwise, it sets them as None. It then filters out invalid indexes if not in test mode. The code also includes a method to load data from a given path using pickle and close the file afterward. Another method parses img_records by extracting bounding boxes, labels, and entity IDs.",
+ "type": "comment"
+ },
+ "4876": {
+ "file_id": 420,
+ "content": " num_selected_records = len(selected_records)\n img_records = list(\n filter(\n lambda x: not np.array_equal(x['entity_box'], img_record[\n 'entity_box']), img_records))\n assert len(img_records) + num_selected_records == num_img_records\n bboxes.append(img_record['entity_box'])\n valid_labels = np.array([\n selected_record['label'] for selected_record in selected_records\n ])\n label = np.zeros(self.num_classes, dtype=np.float32)\n label[valid_labels] = 1.\n labels.append(label)\n entity_ids.append(img_record['entity_id'])\n bboxes = np.stack(bboxes)\n labels = np.stack(labels)\n entity_ids = np.stack(entity_ids)\n return bboxes, labels, entity_ids\n def filter_exclude_file(self):\n valid_indexes = []\n if self.exclude_file is None:\n valid_indexes = list(range(len(self.info)))\n else:\n exclude_video_infos = [",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:94-122"
+ },
+ "4877": {
+ "file_id": 420,
+ "content": "This code is filtering out specific records from the dataset. It checks if the entity box of each record matches with a given img_record's entity box, excluding them if they do. If there are no exclude file information, it includes all the records in valid_indexes. Finally, it stacks and returns bboxes, labels, and entity_ids for further processing.",
+ "type": "comment"
+ },
+ "4878": {
+ "file_id": 420,
+ "content": " x.strip().split(',') for x in open(self.exclude_file)\n ]\n for i, video_info in enumerate(self.info):\n valid_indexes.append(i)\n for video_id, timestamp in exclude_video_infos:\n if (video_info['video_id'] == video_id\n and video_info['timestamp'] == int(timestamp)):\n valid_indexes.pop()\n break\n return valid_indexes\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n records_dict_by_img = defaultdict(list)\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split(',')\n video_id = line_split[0]\n timestamp = int(line_split[1])\n img_key = f'{video_id},{timestamp:04d}'\n entity_box = np.array(list(map(float, line_split[2:6])))\n label = int(line_split[6])\n entity_id = int(line_split[7])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:123-148"
+ },
+ "4879": {
+ "file_id": 420,
+ "content": "The code reads a file, splits each line into video ID, timestamp, and other data. It then checks for any exclusion videos based on the ID and timestamp. If found, it removes that index from the valid_indexes list. Finally, it returns the updated valid_indexes list. The load_file method reads the file, extracts information including video ID, timestamp, entity box, label, and entity ID for each line.",
+ "type": "comment"
+ },
+ "4880": {
+ "file_id": 420,
+ "content": " shot_info = (0, (self.timestamp_end - self.timestamp_start) *\n self._FPS)\n video_info = dict(video_id=video_id,\n timestamp=timestamp,\n entity_box=entity_box,\n label=label,\n entity_id=entity_id,\n shot_info=shot_info)\n records_dict_by_img[img_key].append(video_info)\n for img_key in records_dict_by_img:\n video_id, timestamp = img_key.split(',')\n bboxes, labels, entity_ids = self.parse_img_record(\n records_dict_by_img[img_key])\n ann = dict(gt_bboxes=bboxes,\n gt_labels=labels,\n entity_ids=entity_ids)\n frame_dir = video_id\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n video_info = dict(frame_dir=frame_dir,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:149-170"
+ },
+ "4881": {
+ "file_id": 420,
+ "content": "The code initializes `shot_info` based on the timestamp range and FPS, then creates a `video_info` dictionary containing various video details. It appends this information to the `records_dict_by_img` for each `img_key`. Next, it extracts video ID and timestamp from `img_key`, calls `parse_img_record()`, and stores the resulting bounding boxes, labels, and entity IDs in an `ann` dictionary. Finally, it sets the frame directory path and adds a new `video_info` dictionary for each video.",
+ "type": "comment"
+ },
+ "4882": {
+ "file_id": 420,
+ "content": " video_id=video_id,\n timestamp=int(timestamp),\n img_key=img_key,\n shot_info=shot_info,\n fps=self._FPS,\n ann=ann)\n info.append(video_info)\n return info\n def prepare_train(self, idx):\n results = copy.deepcopy(self.info[idx])\n img_key = results['img_key']\n results['suffix'] = self.suffix\n results['timestamp_start'] = self.timestamp_start\n results['timestamp_end'] = self.timestamp_end\n if self.proposals is not None:\n if img_key not in self.proposals:\n results['proposals'] = np.array([[0, 0, 1, 1]])\n results['scores'] = np.array([1])\n else:\n proposals = self.proposals[img_key]\n assert proposals.shape[-1] in [4, 5]\n if proposals.shape[-1] == 5:\n thr = min(self.person_det_score_thr, max(proposals[:, 4]))",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:171-197"
+ },
+ "4883": {
+ "file_id": 420,
+ "content": "The code initializes a video information object with the provided parameters, including the video ID, timestamp, image key, shot info, FPS, and annotations. It then appends this object to a list of video information. The prepare_train method takes an index, creates a copy of the corresponding video information from the list, adds suffix and timestamp information if applicable, and populates proposals with default values if not present in self.proposals.",
+ "type": "comment"
+ },
+ "4884": {
+ "file_id": 420,
+ "content": " positive_inds = (proposals[:, 4] >= thr)\n proposals = proposals[positive_inds]\n proposals = proposals[:self.num_max_proposals]\n results['proposals'] = proposals[:, :4]\n results['scores'] = proposals[:, 4]\n else:\n proposals = proposals[:self.num_max_proposals]\n results['proposals'] = proposals\n ann = results.pop('ann')\n results['gt_bboxes'] = ann['gt_bboxes']\n results['gt_labels'] = ann['gt_labels']\n results['entity_ids'] = ann['entity_ids']\n #ret = self.pipeline(results, \"\")\n ret = self.pipeline(results)\n #padding for dataloader\n len_proposals = ret['proposals'].shape[0]\n len_gt_bboxes = ret['gt_bboxes'].shape[0]\n len_gt_labels = ret['gt_labels'].shape[0]\n len_scores = ret['scores'].shape[0]\n len_entity_ids = ret['entity_ids'].shape[0]\n padding_len = 128\n ret['proposals'] = self.my_padding_2d(ret['proposals'], padding_len)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:198-221"
+ },
+ "4885": {
+ "file_id": 420,
+ "content": "This code is filtering and padding proposals and annotations for a dataset. It selects positive proposals based on a threshold, limits the number of proposals to the maximum allowed, and assigns the results to different categories. If there are no positive proposals, it simply limits the number and assigns them. After that, it retrieves ground truth bounding boxes, labels, and entity IDs from the 'ann' dictionary. Finally, the code pads the proposals, scores, and other elements with zeros to reach a fixed length of 128 using a custom padding function.",
+ "type": "comment"
+ },
+ "4886": {
+ "file_id": 420,
+ "content": " ret['gt_bboxes'] = self.my_padding_2d(ret['gt_bboxes'], padding_len)\n ret['gt_labels'] = self.my_padding_2d(ret['gt_labels'], padding_len)\n ret['scores'] = self.my_padding_1d(ret['scores'], padding_len)\n ret['entity_ids'] = self.my_padding_1d(ret['entity_ids'], padding_len)\n return ret['imgs'][0], ret['imgs'][1], ret['proposals'], ret[\n 'gt_bboxes'], ret['gt_labels'], ret['scores'], ret[\n 'entity_ids'], np.array(\n ret['img_shape'], dtype=int\n ), idx, len_proposals, len_gt_bboxes, len_gt_labels, len_scores, len_entity_ids\n def my_padding_2d(self, feat, max_len):\n feat_add = np.zeros((max_len - feat.shape[0], feat.shape[1]),\n dtype=np.float32)\n feat_pad = np.concatenate((feat, feat_add), axis=0)\n return feat_pad\n def my_padding_1d(self, feat, max_len):\n feat_add = np.zeros((max_len - feat.shape[0]), dtype=np.float32)\n feat_pad = np.concatenate((feat, feat_add), axis=0)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:222-240"
+ },
+ "4887": {
+ "file_id": 420,
+ "content": "This code snippet defines a class with methods for padding 2D and 1D features. The 'my_padding_2d' method takes a feature matrix and pads it with zeros to the maximum length specified, while the 'my_padding_1d' method does the same but for 1D features. These methods are then called in another function to pad various feature matrices before returning them along with other variables.",
+ "type": "comment"
+ },
+ "4888": {
+ "file_id": 420,
+ "content": " return feat_pad\n def prepare_test(self, idx):\n return self.prepare_train(idx)\n def evaluate(self, results):\n return ava_evaluate_results(self.info, len(self), results,\n self.custom_classes, self.label_file,\n self.file_path, self.exclude_file)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ava_dataset.py:241-249"
+ },
+ "4889": {
+ "file_id": 420,
+ "content": "The code defines three functions: 'prepare_train', 'prepare_test', and 'evaluate'. The 'prepare_train' function is used to prepare training data given an index, while the 'prepare_test' function returns the same as 'prepare_train'. The 'evaluate' function evaluates the results using 'ava_evaluate_results' by passing various arguments.",
+ "type": "comment"
+ },
+ "4890": {
+ "file_id": 421,
+ "content": "/paddlevideo/loader/dataset/base.py",
+ "type": "filepath"
+ },
+ "4891": {
+ "file_id": 421,
+ "content": "This class defines the BaseDataset for PaddlePaddle, with methods for loading data, preparing training and testing sets, and retrieving samples. It supports list format results due to limitations in Paddle.io.DataLoader.",
+ "type": "summary"
+ },
+ "4892": {
+ "file_id": 421,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport numpy as np\nfrom abc import ABC, abstractmethod\nimport paddle\nfrom paddle.io import Dataset\nclass BaseDataset(Dataset, ABC):\n \"\"\"Base class for datasets\n All datasets should subclass it.\n All subclass should overwrite:\n - Method: `load_file`, load info from index file.\n - Method: `prepare_train`, providing train data.\n - Method: `prepare_test`, providing test data.",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/base.py:1-32"
+ },
+ "4893": {
+ "file_id": 421,
+ "content": "This code is a Python class definition for the BaseDataset, which serves as the base class for all dataset types in PaddlePaddle. It requires subclasses to define load_file method for loading info from index files and provide train and test data using prepare_train and prepare_test methods respectively.",
+ "type": "comment"
+ },
+ "4894": {
+ "file_id": 421,
+ "content": " Args:\n file_path (str): index file path.\n pipeline (Sequence XXX)\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): whether to build test dataset. Default: False.\n \"\"\"\n def __init__(self, file_path, pipeline, data_prefix=None, test_mode=False):\n super().__init__()\n self.file_path = file_path\n self.data_prefix = osp.realpath(data_prefix) if \\\n data_prefix is not None and osp.isdir(data_prefix) else data_prefix\n self.test_mode = test_mode\n self.pipeline = pipeline\n self.info = self.load_file()\n @abstractmethod\n def load_file(self):\n \"\"\"load the video information from the index file path.\"\"\"\n pass\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n #Note: For now, paddle.io.DataLoader cannot support dict type retval, so convert to list here\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/base.py:34-59"
+ },
+ "4895": {
+ "file_id": 421,
+ "content": "This code initializes a base dataset class with file path, pipeline, data prefix, and test mode as arguments. It loads video information from the index file using load_file() method, supports training and validation, but cannot handle dict type results due to Paddle.io limitations.",
+ "type": "comment"
+ },
+ "4896": {
+ "file_id": 421,
+ "content": " #unsqueeze label to list\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"TEST: Prepare the data for test given the index.\"\"\"\n #Note: For now, paddle.io.DataLoader cannot support dict type retval, so convert to list here\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n #unsqueeze label to list\n return results['imgs'], np.array([results['labels']])\n def __len__(self):\n \"\"\"get the size of the dataset.\"\"\"\n return len(self.info)\n def __getitem__(self, idx):\n \"\"\" Get the sample for either training or testing given index\"\"\"\n if self.test_mode:\n return self.prepare_test(idx)\n else:\n return self.prepare_train(idx)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/base.py:60-80"
+ },
+ "4897": {
+ "file_id": 421,
+ "content": "This code defines a dataset class with methods for preparing data for training and testing. The `prepare_train` method returns the input images and labels for training, while the `prepare_test` method does the same for testing. The `__len__` method returns the size of the dataset, and the `__getitem__` method retrieves either a training or testing sample based on the mode. Due to an issue with Paddle.io.DataLoader not supporting dict type retval, the results are converted to list format.",
+ "type": "comment"
+ },
+ "4898": {
+ "file_id": 422,
+ "content": "/paddlevideo/loader/dataset/bmn_dataset.py",
+ "type": "filepath"
+ },
+ "4899": {
+ "file_id": 422,
+ "content": "The BMNDataset class handles video datasets for action localization, initializing with file path, pipeline, and subset information. It loads data, sorts by name, and returns features, ground truth IOU map, and start frame indices for training. The function also prepares test data given an index by processing through the pipeline and returning selected results.",
+ "type": "summary"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/49.json b/docs/data/49.json
new file mode 100644
index 000000000..0dde73320
--- /dev/null
+++ b/docs/data/49.json
@@ -0,0 +1,544 @@
+{
+ "4900": {
+ "file_id": 422,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport json\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass BMNDataset(BaseDataset):\n \"\"\"Video dataset for action localization.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n subset,\n **kwargs,\n ):\n self.subset = subset\n super().__init__(file_path, pipeline, **kwargs)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/bmn_dataset.py:1-36"
+ },
+ "4901": {
+ "file_id": 422,
+ "content": "This code snippet defines the BMNDataset class for video datasets used in action localization. It imports necessary modules, registers the class with the DATASETS registry, and initializes the dataset with file path, pipeline, and subset information. Logger is also defined for logging purposes.",
+ "type": "comment"
+ },
+ "4902": {
+ "file_id": 422,
+ "content": " def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n annos = json.load(open(self.file_path))\n for video_name in annos.keys():\n video_subset = annos[video_name][\"subset\"]\n if self.subset in video_subset:\n info.append(\n dict(\n video_name=video_name,\n video_info=annos[video_name],\n ))\n #sort by video_name\n sort_f = lambda elem: elem['video_name']\n info.sort(key=sort_f)\n #add video_idx to info\n for idx, elem in enumerate(info):\n info[idx]['video_idx'] = idx\n logger.info(\"{} subset video numbers: {}\".format(\n self.subset, len(info)))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID: Prepare data for training/valid given the index.\"\"\"\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n return results['video_feat'], results['gt_iou_map'], results['gt_start'],\\",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/bmn_dataset.py:38-64"
+ },
+ "4903": {
+ "file_id": 422,
+ "content": "This function is loading an index file to get video information and then sorts the data by video name. It also adds a video_idx attribute to each element in the list and returns the video features, ground truth IOU map, and start frame indices for training purposes.",
+ "type": "comment"
+ },
+ "4904": {
+ "file_id": 422,
+ "content": " results['gt_end']\n def prepare_test(self, idx):\n \"\"\"TEST: Prepare the data for test given the index.\"\"\"\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n return results['video_feat'], results['gt_iou_map'], results['gt_start'], \\\n results['gt_end'], results['video_idx']",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/bmn_dataset.py:65-72"
+ },
+ "4905": {
+ "file_id": 422,
+ "content": "This function prepares test data given an index by copying the dataset info, processing it through the pipeline, and returning selected results (video_feat, gt_iou_map, gt_start, gt_end, video_idx).",
+ "type": "comment"
+ },
+ "4906": {
+ "file_id": 423,
+ "content": "/paddlevideo/loader/dataset/davis_dataset.py",
+ "type": "filepath"
+ },
+ "4907": {
+ "file_id": 423,
+ "content": "The Python class VOS_Test extends BaseDataset for video object segmentation tasks and supports pipeline mode, color type options, and resizing. The Davis 2017 dataset is initialized in PaddleVideo and returns a sequence dataset with images, labels, and fixed resolution of 480 pixels.",
+ "type": "summary"
+ },
+ "4908": {
+ "file_id": 423,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport shutil\nfrom PIL import Image\nimport cv2\nfrom paddle.io import Dataset\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\nclass VOS_Test(Dataset):\n \"\"\"process frames in each video\n \"\"\"\n def __init__(self,\n image_root,\n label_root,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:1-37"
+ },
+ "4909": {
+ "file_id": 423,
+ "content": "This code snippet is from PaddleVideo's davis_dataset.py file and it appears to be a Python class named VOS_Test, which extends the BaseDataset class from the same module. The class is used for processing frames in each video of a dataset. It takes image_root and label_root as input parameters for accessing the required data. The logger is imported from paddle.utils to log any relevant information during execution. This dataset seems to be designed for video object segmentation (VOS) tasks, commonly used in computer vision applications.",
+ "type": "comment"
+ },
+ "4910": {
+ "file_id": 423,
+ "content": " seq_name,\n images,\n labels,\n pipeline=None,\n rgb=False,\n resolution=None):\n self.image_root = image_root\n self.label_root = label_root\n self.seq_name = seq_name\n self.images = images # image file list\n self.labels = labels\n self.obj_num = 1\n self.num_frame = len(self.images)\n self.pipeline = pipeline\n self.rgb = rgb\n self.resolution = resolution\n self.obj_nums = []\n temp_obj_num = 0\n for img_name in self.images:\n self.obj_nums.append(temp_obj_num)\n current_label_name = img_name.split('.')[0] + '.png'\n if current_label_name in self.labels:\n current_label = self.read_label(current_label_name)\n if temp_obj_num < np.unique(\n current_label)[-1]: #get object number from label_id\n temp_obj_num = np.unique(current_label)[-1]\n def __len__(self):",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:38-66"
+ },
+ "4911": {
+ "file_id": 423,
+ "content": "This code initializes the dataset with image and label file lists, image root and label root paths. It sets object number, total frames, pipeline mode, color type, resolution, creates an object numbers list, and assigns object numbers from labels.",
+ "type": "comment"
+ },
+ "4912": {
+ "file_id": 423,
+ "content": " return len(self.images)\n def read_image(self, idx):\n img_name = self.images[idx]\n img_path = os.path.join(self.image_root, self.seq_name, img_name)\n img = cv2.imread(img_path)\n img = np.array(img, dtype=np.float32)\n if self.rgb:\n img = img[:, :, [2, 1, 0]]\n return img\n def read_label(self, label_name):\n label_path = os.path.join(self.label_root, self.seq_name, label_name)\n label = Image.open(label_path)\n label = np.array(label, dtype=np.uint8)\n return label\n def __getitem__(self, idx):\n img_name = self.images[idx]\n current_img = self.read_image(idx)\n current_img = np.array(current_img)\n height, width, channels = current_img.shape\n if self.resolution is not None:\n width = int(np.ceil(float(width) * self.resolution / float(height)))\n height = int(self.resolution)\n current_label_name = img_name.split('.')[0] + '.png'\n obj_num = self.obj_nums[idx]",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:67-94"
+ },
+ "4913": {
+ "file_id": 423,
+ "content": "This code defines a class that loads data from the DAVIS dataset. It first returns the number of images in the dataset, then reads an image at a given index, and finally reads a corresponding label for the image. The class also allows resizing the images to a specified resolution if needed.",
+ "type": "comment"
+ },
+ "4914": {
+ "file_id": 423,
+ "content": " if current_label_name in self.labels:\n current_label = self.read_label(current_label_name)\n current_label = np.array(current_label)\n sample = {\n 'current_img': current_img,\n 'current_label': current_label\n }\n else:\n sample = {\n 'current_img': current_img\n } #only the first frame contains label\n sample['meta'] = {\n 'seq_name': self.seq_name,\n 'frame_num': self.num_frame,\n 'obj_num': obj_num,\n 'current_name': img_name,\n 'height': height,\n 'width': width,\n 'flip': False\n }\n if self.pipeline is not None:\n sample = self.pipeline(sample)\n for s in sample:\n s['current_img'] = np.array(s['current_img'])\n if 'current_label' in s.keys():\n s['current_label'] = s['current_label']\n return sample\n@DATASETS.register()\nclass DavisDataset(BaseDataset):",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:96-127"
+ },
+ "4915": {
+ "file_id": 423,
+ "content": "The function generates a sample for a dataset, including image and label data. It checks if the current_label_name is in labels, reads the label if present, creates a sample dictionary, adds metadata to the sample, applies a pipeline if one is specified, and converts 'current_img' to numpy array format.",
+ "type": "comment"
+ },
+ "4916": {
+ "file_id": 423,
+ "content": " \"\"\"Davis 2017 dataset.\n \"\"\"\n def __init__(\n self,\n file_path,\n result_root,\n pipeline,\n data_prefix=None,\n test_mode=False,\n year=2017,\n rgb=False,\n resolution='480p',\n ):\n self.rgb = rgb\n self.result_root = result_root\n self.resolution = resolution\n self.year = year\n self.spt = 'val' if test_mode else 'train'\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n self.image_root = os.path.join(self.file_path, 'JPEGImages',\n self.resolution)\n self.label_root = os.path.join(self.file_path, 'Annotations',\n self.resolution)\n seq_names = []\n with open(\n os.path.join(self.file_path, 'ImageSets', str(self.year),\n self.spt + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:128-158"
+ },
+ "4917": {
+ "file_id": 423,
+ "content": "The code represents the initialization and file loading process for the Davis 2017 dataset in PaddleVideo. The constructor takes various parameters like file path, result root, pipeline, data prefix, test mode, year, rgb, and resolution to initialize the class attributes. The load_file() method sets image and label roots based on the provided resolution and reads the sequence names from a specified file.",
+ "type": "comment"
+ },
+ "4918": {
+ "file_id": 423,
+ "content": " seq_names.extend(seqs_tmp)\n self.info = list(np.unique(seq_names))\n return self.info\n def prepare_test(self, idx):\n seq_name = self.info[idx] #video name\n images = list(\n np.sort(os.listdir(os.path.join(self.image_root, seq_name))))\n labels = [images[0].replace('jpg', 'png')] #we have first frame target\n # copy first frame target\n if not os.path.isfile(\n os.path.join(self.result_root, seq_name, labels[0])):\n if not os.path.exists(os.path.join(self.result_root, seq_name)):\n os.makedirs(os.path.join(self.result_root, seq_name))\n source_label_path = os.path.join(self.label_root, seq_name,\n labels[0])\n result_label_path = os.path.join(self.result_root, seq_name,\n labels[0])\n shutil.copy(source_label_path, result_label_path)\n seq_dataset = VOS_Test(self.image_root,\n self.label_root,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:159-182"
+ },
+ "4919": {
+ "file_id": 423,
+ "content": "This function prepares a test dataset for the VOS task. It retrieves the video name from the info list, then lists all image files in the corresponding directory and adds the first frame as the target label. If the target label does not exist in the result directory, it creates the necessary directories and copies the target label file to the correct location.",
+ "type": "comment"
+ },
+ "4920": {
+ "file_id": 423,
+ "content": " seq_name,\n images,\n labels,\n self.pipeline,\n rgb=self.rgb,\n resolution=480)\n return seq_dataset",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/davis_dataset.py:183-189"
+ },
+ "4921": {
+ "file_id": 423,
+ "content": "This code is returning a sequence dataset named seq_name with associated images and labels, processed by the pipeline function specified, potentially using RGB format if self.rgb is True, and a fixed resolution of 480 pixels.",
+ "type": "comment"
+ },
+ "4922": {
+ "file_id": 424,
+ "content": "/paddlevideo/loader/dataset/feature.py",
+ "type": "filepath"
+ },
+ "4923": {
+ "file_id": 424,
+ "content": "The Python class FeatureDataset, part of the PaddleVideo library, initializes attributes and provides methods for action recognition tasks. The code also includes a prepare_test function to prepare data for testing by applying a pipeline and checking 'iou_norm' results.",
+ "type": "summary"
+ },
+ "4924": {
+ "file_id": 424,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport os.path as osp\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\n@DATASETS.register()\nclass FeatureDataset(BaseDataset):\n \"\"\"Feature dataset for action recognition\n Example:(TODO)\n Args:(TODO)\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n data_prefix=None,\n test_mode=False,\n suffix=None,\n ):\n self.suffix = suffix\n super().__init__(file_path, pipeline, data_prefix, test_mode)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/feature.py:1-36"
+ },
+ "4925": {
+ "file_id": 424,
+ "content": "This code is a Python class named FeatureDataset, which is a subclass of BaseDataset. It appears to be part of the PaddleVideo library and is used for action recognition tasks. The class has an __init__ method that initializes various attributes such as file_path, pipeline, data_prefix, test_mode, and suffix. The class is registered in the DATASETS registry.",
+ "type": "comment"
+ },
+ "4926": {
+ "file_id": 424,
+ "content": " def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n filename = line.strip().split()[0]\n if self.data_prefix is not None:\n filename = osp.join(self.data_prefix, filename)\n if self.suffix is not None:\n filename = filename + self.suffix\n info.append(dict(filename=filename))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n if 'iou_norm' in results:\n return results['rgb_data'], results['rgb_len'], results[\n 'rgb_mask'], results['audio_data'], results[\n 'audio_len'], results['audio_mask'], results[\n 'labels'], results['iou_norm']\n else:\n return results['rgb_data'], results['rgb_len'], results[",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/feature.py:38-63"
+ },
+ "4927": {
+ "file_id": 424,
+ "content": "The code defines two methods. The `load_file` method reads an index file and retrieves video information by parsing each line, stripping whitespace, splitting the filename, and optionally appending a specified suffix or joining with a data prefix. It returns a list of dictionaries containing the filenames. The `prepare_train` method takes an index and prepares training/validation data using a specified pipeline function. If 'iou_norm' is present in the results, it returns multiple data types (e.g., rgb_data, rgb_len, rgb_mask, audio_data, audio_len, audio_mask) along with labels.",
+ "type": "comment"
+ },
+ "4928": {
+ "file_id": 424,
+ "content": " 'rgb_mask'], results['audio_data'], results[\n 'audio_len'], results['audio_mask'], results['labels']\n def prepare_test(self, idx):\n \"\"\"TEST. Prepare the data for testing given the index.\"\"\"\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n if 'iou_norm' in results:\n return results['rgb_data'], results['rgb_len'], results[\n 'rgb_mask'], results['audio_data'], results[\n 'audio_len'], results['audio_mask'], results[\n 'labels'], results['iou_norm']\n else:\n return results['rgb_data'], results['rgb_len'], results[\n 'rgb_mask'], results['audio_data'], results[\n 'audio_len'], results['audio_mask'], results['labels']",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/feature.py:64-80"
+ },
+ "4929": {
+ "file_id": 424,
+ "content": "The code defines a function prepare_test that prepares data for testing. It creates a deep copy of the dataset information at the given index, applies a pipeline to it and then checks if 'iou_norm' is in the results. If it is, it returns 7 elements including 'rgb_data', 'audio_data', 'labels' and 'iou_norm'. Otherwise, it returns 6 elements without 'iou_norm'.",
+ "type": "comment"
+ },
+ "4930": {
+ "file_id": 425,
+ "content": "/paddlevideo/loader/dataset/frame.py",
+ "type": "filepath"
+ },
+ "4931": {
+ "file_id": 425,
+ "content": "The PaddleVideo library's FrameDataset and FrameDataset_Sport classes load, transform, and process video data with error handling for missing or corrupted files under Apache License 2.0.",
+ "type": "summary"
+ },
+ "4932": {
+ "file_id": 425,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass FrameDataset(BaseDataset):\n \"\"\"Rawframe dataset for action recognition.\n The dataset loads raw frames from frame files, and apply specified transform operatation them.\n The indecx file ",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:1-31"
+ },
+ "4933": {
+ "file_id": 425,
+ "content": "This code is a Python class for a FrameDataset, which loads raw frames from frame files and applies specified transform operations. It is part of the PaddleVideo library and follows Apache License 2.0. The dataset index file is used to organize the loaded data. This class inherits from BaseDataset, suggesting it has some common functionalities.",
+ "type": "comment"
+ },
+ "4934": {
+ "file_id": 425,
+ "content": "is a text file with multiple lines, and each line indicates the directory of frames of a video, toatl frames of the video, and its label, which split with a whitespace.\n Example of an index file:\n .. code-block:: txt\n file_path-1 150 1\n file_path-2 160 1\n file_path-3 170 2\n file_path-4 180 2\n Args:\n file_path (str): Path to the index file.\n pipeline(XXX):\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n suffix (str): suffix of file. Default: 'img_{:05}.jpg'.\n \"\"\"\n def __init__(self,\n file_path,\n pipeline,\n num_retries=5,\n data_prefix=None,\n test_mode=False,\n suffix='img_{:05}.jpg'):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:31-61"
+ },
+ "4935": {
+ "file_id": 425,
+ "content": "This code initializes a class for loading video information from an index file. The index file contains the directory of frames, total frames, and label for each video. It supports pipeline and data_prefix, and has options for test mode and suffix format.",
+ "type": "comment"
+ },
+ "4936": {
+ "file_id": 425,
+ "content": " info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split()\n frame_dir, frames_len, labels = line_split\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n info.append(\n dict(frame_dir=frame_dir,\n suffix=self.suffix,\n frames_len=frames_len,\n labels=int(labels)))\n return info\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training/valid given index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:62-86"
+ },
+ "4937": {
+ "file_id": 425,
+ "content": "This code reads data from a file and returns information related to frames, such as the frame directory, suffix, number of frames, and labels. It also includes a try-catch block that attempts to prepare the frames for training or validation multiple times if an exception occurs while reading the frames files.",
+ "type": "comment"
+ },
+ "4938": {
+ "file_id": 425,
+ "content": " \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for test given index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:87-108"
+ },
+ "4939": {
+ "file_id": 425,
+ "content": "The code handles exceptions for loading missing frames in the dataset. It tries to load frames multiple times within a specified range and logs errors when needed. If an exception occurs, it randomly selects another index and continues the process until successful.",
+ "type": "comment"
+ },
+ "4940": {
+ "file_id": 425,
+ "content": "@DATASETS.register()\nclass FrameDataset_Sport(BaseDataset):\n \"\"\"Video dataset for action recognition\n The dataset loads raw videos and apply specified transforms on them.\n The index file is a file with multiple lines, and each line indicates\n a sample video with the filepath and label, which are split with a whitesapce.\n Example of a inde file:\n .. code-block:: txt\n path/000.mp4 1\n path/001.mp4 1\n path/002.mp4 2\n path/003.mp4 2\n Args:\n file_path(str): Path to the index file.\n pipeline(XXX): A sequence of data transforms.\n **kwargs: Keyword arguments for ```BaseDataset```.\n \"\"\"\n def __init__(self, file_path, pipeline, num_retries=5, suffix='', **kwargs):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, **kwargs)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n with open(self.file_path, 'r') as fin:",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:111-136"
+ },
+ "4941": {
+ "file_id": 425,
+ "content": "The code defines a FrameDataset_Sport class for loading raw videos and applying specified transforms. It uses an index file containing video file paths and labels, and takes arguments for file path, data transforms pipeline, retry attempts, and other BaseDataset kwargs. The load_file function reads the index file to obtain video information.",
+ "type": "comment"
+ },
+ "4942": {
+ "file_id": 425,
+ "content": " for line in fin:\n line_split = line.strip().split()\n frame_dir = line_split[0]\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n info.append(dict(frame_dir=frame_dir, suffix=self.suffix))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:137-158"
+ },
+ "4943": {
+ "file_id": 425,
+ "content": "This code reads lines from a file, each representing a frame directory and associated information. It then splits the line into components and appends the frame directory and suffix to the info list. The prepare_train function attempts to process data for training or validation, handling exceptions by retrying up to a specified number of times before selecting another index at random.",
+ "type": "comment"
+ },
+ "4944": {
+ "file_id": 425,
+ "content": " continue\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"TEST. Prepare the data for test given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/frame.py:159-177"
+ },
+ "4945": {
+ "file_id": 425,
+ "content": "The function `prepare_test` is attempting to prepare data for testing by iterating through a certain number of retries in case of exceptions caused by corrupted video files. If an exception occurs, it logs the error and tries again with a different random index from the list of info. Once successful, it returns the images and labels as arrays.",
+ "type": "comment"
+ },
+ "4946": {
+ "file_id": 426,
+ "content": "/paddlevideo/loader/dataset/ms_tcn_dataset.py",
+ "type": "filepath"
+ },
+ "4947": {
+ "file_id": 426,
+ "content": "The code initializes a class for MS-TCN dataset, loads video features and labels for training or testing, and converts label data to integers using a dictionary mapping.",
+ "type": "summary"
+ },
+ "4948": {
+ "file_id": 426,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport os\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass MSTCNDataset(BaseDataset):\n \"\"\"Video dataset for action segmentation.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n feature_path,\n gt_path,\n actions_map_file_path,\n **kwargs,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ms_tcn_dataset.py:1-38"
+ },
+ "4949": {
+ "file_id": 426,
+ "content": "Imports required modules and registers a class for the MS-TCN dataset, a video dataset for action segmentation. The class initializes with file paths and other parameters.",
+ "type": "comment"
+ },
+ "4950": {
+ "file_id": 426,
+ "content": " ):\n super().__init__(file_path, pipeline, **kwargs)\n self.gt_path = gt_path\n self.actions_map_file_path = actions_map_file_path\n self.feature_path = feature_path\n # actions dict generate\n file_ptr = open(self.actions_map_file_path, 'r')\n actions = file_ptr.read().split('\\n')[:-1]\n file_ptr.close()\n self.actions_dict = dict()\n for a in actions:\n self.actions_dict[a.split()[1]] = int(a.split()[0])\n self.num_classes = len(self.actions_dict.keys())\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n file_ptr = open(self.file_path, 'r')\n info = file_ptr.read().split('\\n')[:-1]\n file_ptr.close()\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID: Prepare data for training/valid given the index.\"\"\"\n results = {}\n video_name = self.info[idx]\n # load video feature\n file_name = video_name.split('.')[0] + \".npy\"\n feat_file_path = os.path.join(self.feature_path, file_name)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ms_tcn_dataset.py:39-68"
+ },
+ "4951": {
+ "file_id": 426,
+ "content": "This code initializes a class, likely for data loading in a video dataset. It takes file paths as parameters, reads an actions map file to create a dictionary of action classes and their corresponding labels. The class also has a method load_file() to read the index file, and a method prepare_train() to prepare training data given an index.",
+ "type": "comment"
+ },
+ "4952": {
+ "file_id": 426,
+ "content": " #TODO: check path\n video_feat = np.load(feat_file_path)\n # load label\n target_file_path = os.path.join(self.gt_path, video_name)\n file_ptr = open(target_file_path, 'r')\n content = file_ptr.read().split('\\n')[:-1]\n classes = np.zeros(min(np.shape(video_feat)[1], len(content)), dtype='int64')\n for i in range(len(classes)):\n classes[i] = self.actions_dict[content[i]]\n # classes = classes * (-100)\n results['video_feat'] = copy.deepcopy(video_feat)\n results['video_gt'] = copy.deepcopy(classes)\n results = self.pipeline(results)\n return results['video_feat'], results['video_gt']\n def prepare_test(self, idx):\n \"\"\"TEST: Prepare the data for test given the index.\"\"\"\n results = {}\n video_name = self.info[idx]\n # load video feature\n file_name = video_name.split('.')[0] + \".npy\"\n feat_file_path = os.path.join(self.feature_path, file_name)\n #TODO: check path\n video_feat = np.load(feat_file_path)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ms_tcn_dataset.py:69-95"
+ },
+ "4953": {
+ "file_id": 426,
+ "content": "This code is loading video features and labels from a dataset, likely for training or testing purposes. It first checks the path of the feature file, then loads both the video feature and label data from specified paths. The code converts the label data into integer format using a dictionary mapping and performs some potential preprocessing steps (not shown here). Finally, it returns the video feature and label data for further processing.",
+ "type": "comment"
+ },
+ "4954": {
+ "file_id": 426,
+ "content": " # load label\n target_file_path = os.path.join(self.gt_path, video_name)\n file_ptr = open(target_file_path, 'r')\n content = file_ptr.read().split('\\n')[:-1]\n classes = np.zeros(min(np.shape(video_feat)[1], len(content)))\n for i in range(len(classes)):\n classes[i] = self.actions_dict[content[i]]\n # classes = classes * (-100)\n results['video_feat'] = copy.deepcopy(video_feat)\n results['video_gt'] = copy.deepcopy(classes)\n results = self.pipeline(results)\n return results['video_feat'], results['video_gt']",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ms_tcn_dataset.py:97-110"
+ },
+ "4955": {
+ "file_id": 426,
+ "content": "This function loads labels for a video dataset. It reads the label file, converts content to class numbers using actions_dict, assigns class values to classes array, scales the classes, and returns the feature and ground truth data for the video.",
+ "type": "comment"
+ },
+ "4956": {
+ "file_id": 427,
+ "content": "/paddlevideo/loader/dataset/msrvtt.py",
+ "type": "filepath"
+ },
+ "4957": {
+ "file_id": 427,
+ "content": "The Python script prepares MSRVTTDataset by importing libraries, creating a class, tokenizing captions, retrieving features, and preparing sequences for processing. It processes image data, performs array operations, pads, resizes, calculates features, and converts to float32 for training/testing.",
+ "type": "summary"
+ },
+ "4958": {
+ "file_id": 427,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\ntry:\n import lmdb\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [lmdb] package and it's dependencies is required for ActBERT.\"\n )\nimport pickle\ntry:\n from paddlenlp.transformers import BertTokenizer\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [paddlenlp] package and it's dependencies is required for ActBERT.\"\n )",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:1-31"
+ },
+ "4959": {
+ "file_id": 427,
+ "content": "The code is a Python script that imports various libraries and packages, checks for the availability of 'lmdb' library, and tries to import 'BertTokenizer' from 'paddlenlp'. It also includes license information and copyright notice.",
+ "type": "comment"
+ },
+ "4960": {
+ "file_id": 427,
+ "content": "from ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass MSRVTTDataset(BaseDataset):\n \"\"\"MSR-VTT dataset for text-video clip retrieval.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n features_path,\n bert_model=\"bert-base-uncased\",\n padding_index=0,\n max_seq_length=36,\n max_region_num=36,\n max_action_num=5,\n vision_feature_dim=2048,\n action_feature_dim=2048,\n spatials_dim=5,\n data_prefix=None,\n test_mode=False,\n ):\n self.features_path = features_path\n self.bert_model = bert_model\n self.padding_index = padding_index\n self.max_seq_length = max_seq_length\n self.max_region_num = max_region_num\n self._max_action_num = max_action_num\n self.vision_feature_dim = vision_feature_dim\n self.action_feature_dim = action_feature_dim\n self.spatials_dim = spatials_dim",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:32-67"
+ },
+ "4961": {
+ "file_id": 427,
+ "content": "The code defines the `MSRVTTDataset` class for text-video clip retrieval from MSR-VTT dataset, registering it in the registry. It takes parameters such as file path, pipeline, and maximum sequence length for initializing the dataset, and provides attributes like bert model, padding index, and other dimensions for processing the dataset.",
+ "type": "comment"
+ },
+ "4962": {
+ "file_id": 427,
+ "content": " self._tokenizer = BertTokenizer.from_pretrained(bert_model,\n do_lower_case=True)\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n self.tokenize()\n self.gen_feature()\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n with open(self.file_path) as fin:\n self.image_entries = []\n self.caption_entries = []\n for line in fin.readlines():\n line = line.strip()\n vid_id = line.split(',')[0]\n self.image_entries.append(vid_id)\n self.caption_entries.append({\n \"caption\": line.split(',')[1],\n \"vid_id\": vid_id\n })\n self.env = lmdb.open(self.features_path)\n def tokenize(self):\n for entry in self.caption_entries:\n tokens = []\n tokens.append(\"[CLS]\")\n for token in self._tokenizer.tokenize(entry[\"caption\"]):",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:68-93"
+ },
+ "4963": {
+ "file_id": 427,
+ "content": "The code snippet initializes a BertTokenizer object, loads file containing video information, tokenizes each entry's caption using the initialized tokenizer.",
+ "type": "comment"
+ },
+ "4964": {
+ "file_id": 427,
+ "content": " tokens.append(token)\n tokens.append(\"[SEP]\")\n tokens = self._tokenizer.convert_tokens_to_ids(tokens)\n segment_ids = [0] * len(tokens)\n input_mask = [1] * len(tokens)\n if len(tokens) < self.max_seq_length:\n padding = [self.padding_index\n ] * (self.max_seq_length - len(tokens))\n tokens = tokens + padding\n input_mask += padding\n segment_ids += padding\n entry[\"token\"] = np.array(tokens).astype('int64')\n entry[\"input_mask\"] = np.array(input_mask)\n entry[\"segment_ids\"] = np.array(segment_ids).astype('int64')\n def get_image_feature(self, video_id):\n video_id = str(video_id).encode()\n with self.env.begin(write=False) as txn:\n item = pickle.loads(txn.get(video_id))\n video_id = item[\"video_id\"]\n image_h = int(item[\"image_h\"])\n image_w = int(item[\"image_w\"])\n features = item[\"features\"].reshape(-1, self.vision_feature_dim)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:94-120"
+ },
+ "4965": {
+ "file_id": 427,
+ "content": "This code is part of a class that processes video data. It appends tokens to an entry, converts tokens to ids, creates segment and input masks, and pads the sequence if necessary. The \"get_image_feature\" function retrieves image features from a database for a given video ID.",
+ "type": "comment"
+ },
+ "4966": {
+ "file_id": 427,
+ "content": " boxes = item[\"boxes\"].reshape(-1, 4)\n num_boxes = features.shape[0]\n g_feat = np.sum(features, axis=0) / num_boxes\n num_boxes = num_boxes + 1\n features = np.concatenate(\n [np.expand_dims(g_feat, axis=0), features], axis=0)\n action_features = item[\"action_features\"].reshape(\n -1, self.action_feature_dim)\n image_location = np.zeros((boxes.shape[0], self.spatials_dim),\n dtype=np.float32)\n image_location[:, :4] = boxes\n image_location[:,\n 4] = ((image_location[:, 3] - image_location[:, 1]) *\n (image_location[:, 2] - image_location[:, 0]) /\n (float(image_w) * float(image_h)))\n image_location[:, 0] = image_location[:, 0] / float(image_w)\n image_location[:, 1] = image_location[:, 1] / float(image_h)\n image_location[:, 2] = image_location[:, 2] / float(image_w)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:121-142"
+ },
+ "4967": {
+ "file_id": 427,
+ "content": "This code is resizing and calculating the image location for each box in a dataset. It also concatenates the average feature to the start of the features array, and handles reshaping action_features. The code uses numpy functions extensively for array operations.",
+ "type": "comment"
+ },
+ "4968": {
+ "file_id": 427,
+ "content": " image_location[:, 3] = image_location[:, 3] / float(image_h)\n g_location = np.array([0, 0, 1, 1, 1])\n image_location = np.concatenate(\n [np.expand_dims(g_location, axis=0), image_location], axis=0)\n return features, num_boxes, image_location, action_features\n def gen_feature(self):\n num_inst = len(self.image_entries) #1000\n self.features_all = np.zeros(\n (num_inst, self.max_region_num, self.vision_feature_dim))\n self.action_features_all = np.zeros(\n (num_inst, self._max_action_num, self.action_feature_dim))\n self.spatials_all = np.zeros(\n (num_inst, self.max_region_num, self.spatials_dim))\n self.image_mask_all = np.zeros((num_inst, self.max_region_num))\n self.action_mask_all = np.zeros((num_inst, self._max_action_num))\n for i, image_id in enumerate(self.image_entries):\n features, num_boxes, boxes, action_features = self.get_image_feature(\n image_id)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:143-163"
+ },
+ "4969": {
+ "file_id": 427,
+ "content": "The code defines a function that returns features, number of boxes, image location, and action features after processing an input image. It also initializes arrays for all instances of features, action features, spatial locations, and masks. The code then iterates over each image ID and calls another function to get the respective features, num_boxes, boxes, and action_features.",
+ "type": "comment"
+ },
+ "4970": {
+ "file_id": 427,
+ "content": " mix_num_boxes = min(int(num_boxes), self.max_region_num)\n mix_boxes_pad = np.zeros((self.max_region_num, self.spatials_dim))\n mix_features_pad = np.zeros(\n (self.max_region_num, self.vision_feature_dim))\n image_mask = [1] * (int(mix_num_boxes))\n while len(image_mask) < self.max_region_num:\n image_mask.append(0)\n action_mask = [1] * (self._max_action_num)\n while len(action_mask) < self._max_action_num:\n action_mask.append(0)\n mix_boxes_pad[:mix_num_boxes] = boxes[:mix_num_boxes]\n mix_features_pad[:mix_num_boxes] = features[:mix_num_boxes]\n self.features_all[i] = mix_features_pad\n x = action_features.shape[0]\n self.action_features_all[i][:x] = action_features[:]\n self.image_mask_all[i] = np.array(image_mask)\n self.action_mask_all[i] = np.array(action_mask)\n self.spatials_all[i] = mix_boxes_pad\n self.features_all = self.features_all.astype(\"float32\")",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:165-187"
+ },
+ "4971": {
+ "file_id": 427,
+ "content": "The code handles the padding of features, boxes and masks for a dataset. It ensures that all sequences have the same length by padding them with zeros if necessary. The mixed features (maximum region number), boxes, and action features are assigned to respective lists. These lists will be used later in the program. The code also converts the features list to float32 data type.",
+ "type": "comment"
+ },
+ "4972": {
+ "file_id": 427,
+ "content": " self.action_features_all = self.action_features_all.astype(\"float32\")\n self.image_mask_all = self.image_mask_all.astype(\"int64\")\n self.action_mask_all = self.action_mask_all.astype(\"int64\")\n self.spatials_all = self.spatials_all.astype(\"float32\")\n def prepare_train(self, idx):\n pass\n def prepare_test(self, idx):\n entry = self.caption_entries[idx]\n caption = entry[\"token\"]\n input_mask = entry[\"input_mask\"]\n segment_ids = entry[\"segment_ids\"]\n target_all = np.zeros(1000)\n for i, image_id in enumerate(self.image_entries):\n if image_id == entry[\"vid_id\"]:\n target_all[i] = 1\n return (\n caption,\n self.action_features_all,\n self.features_all,\n self.spatials_all,\n segment_ids,\n input_mask,\n self.image_mask_all,\n self.action_mask_all,\n target_all,\n )\n def __len__(self):\n return len(self.caption_entries)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/msrvtt.py:188-220"
+ },
+ "4973": {
+ "file_id": 427,
+ "content": "This code initializes data types and provides methods for preparing training and testing data. The `prepare_train` method is left empty, while `prepare_test` takes an index, retrieves the corresponding entry, creates a target array, and returns various data arrays to be used in testing. The length of the dataset is determined by the number of caption entries.",
+ "type": "comment"
+ },
+ "4974": {
+ "file_id": 428,
+ "content": "/paddlevideo/loader/dataset/oxford.py",
+ "type": "filepath"
+ },
+ "4975": {
+ "file_id": 428,
+ "content": "This Python class defines the \"MonoDataset\" for PaddleVideo, initializes with file path, data prefix, and pipeline support. The code contains `load_file`, `prepare_train`, and `prepare_test` methods for dataset preparation and information retrieval.",
+ "type": "summary"
+ },
+ "4976": {
+ "file_id": 428,
+ "content": "# Copyright Niantic 2019. Patent Pending. All rights reserved.\n#\n# This software is licensed under the terms of the Monodepth2 licence\n# which allows for non-commercial use only, the full terms of which are made\n# available in the LICENSE file.\nfrom __future__ import absolute_import, division, print_function\nimport copy\nfrom os import path as osp\nfrom PIL import Image\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\ndef pil_loader(path):\n # open path as file to avoid ResourceWarning\n # (https://github.com/python-pillow/Pillow/issues/835)\n with open(path, 'rb') as f:\n with Image.open(f) as img:\n return img.convert('RGB')\n@DATASETS.register()\nclass MonoDataset(BaseDataset):\n def __init__(self,\n file_path,\n data_prefix,\n pipeline,\n num_retries=0,\n suffix='.png',\n **kwargs):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, data_prefix, **kwargs)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/oxford.py:1-37"
+ },
+ "4977": {
+ "file_id": 428,
+ "content": "This code is a Python class defining the \"MonoDataset\" dataset for PaddleVideo. It requires file path, data prefix, and pipeline for initialization, supports retries when accessing files, and utilizes the pil_loader function for loading RGB images from file paths with specified suffixes. The code also registers MonoDataset with DATASETS registry.",
+ "type": "comment"
+ },
+ "4978": {
+ "file_id": 428,
+ "content": " def load_file(self):\n info = []\n with open(self.file_path, 'r') as f:\n for line in f:\n filename = line.strip() + self.suffix\n folder = osp.dirname(filename)\n frame_index = line.strip().split('/')[1]\n info.append(\n dict(data_path=self.data_prefix,\n filename=filename,\n folder=folder,\n frame_index=int(frame_index)))\n return info\n def prepare_train(self, idx):\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n results['imgs']['idx'] = idx\n return results['imgs'], results['day_or_night']\n def prepare_test(self, idx):\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n return results['imgs'], results['day_or_night']",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/oxford.py:39-62"
+ },
+ "4979": {
+ "file_id": 428,
+ "content": "The code defines three methods: `load_file`, `prepare_train`, and `prepare_test`. The `load_file` method reads a file containing information about image files, stripping off newline characters, and appending the necessary file suffix. It then appends a dictionary to the `info` list with data path, filename, folder location, and frame index. The `prepare_train` and `prepare_test` methods copy an entry from `self.info` and apply a pipeline before returning relevant information (e.g., images, labels).",
+ "type": "comment"
+ },
+ "4980": {
+ "file_id": 429,
+ "content": "/paddlevideo/loader/dataset/skeleton.py",
+ "type": "filepath"
+ },
+ "4981": {
+ "file_id": 429,
+ "content": "The code defines a SkeletonDataset class for action recognition, loading skeleton features and applying normalization operations. It imports libraries, registers the dataset, includes a logger, and has a class for loading skeleton data with optional label path and test mode parameter. The class loads and returns data for training or testing, preparing features based on training/testing needs and considering labels if available.",
+ "type": "summary"
+ },
+ "4982": {
+ "file_id": 429,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport pickle\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass SkeletonDataset(BaseDataset):\n \"\"\"\n Skeleton dataset for action recognition.\n The dataset loads skeleton feature, and apply norm operatations.\n Args:\n file_path (str): Path to the index file.",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/skeleton.py:1-34"
+ },
+ "4983": {
+ "file_id": 429,
+ "content": "This code defines a SkeletonDataset class for action recognition. It loads skeleton features and applies normalization operations. It also imports necessary libraries, registers the dataset with DATASETS, and includes a logger for logging purposes.",
+ "type": "comment"
+ },
+ "4984": {
+ "file_id": 429,
+ "content": " pipeline(obj): Define the pipeline of data preprocessing.\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n \"\"\"\n def __init__(self, file_path, pipeline, label_path=None, test_mode=False):\n self.label_path = label_path\n super().__init__(file_path, pipeline, test_mode=test_mode)\n def load_file(self):\n \"\"\"Load feature file to get skeleton information.\"\"\"\n logger.info(\"Loading data, it will take some moment...\")\n self.data = np.load(self.file_path)\n if self.label_path:\n if self.label_path.endswith('npy'):\n self.label = np.load(self.label_path)\n elif self.label_path.endswith('pkl'):\n with open(self.label_path, 'rb') as f:\n sample_name, self.label = pickle.load(f)\n else:\n logger.info(\n \"Label path not provided when test_mode={}, here just output predictions.\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/skeleton.py:35-55"
+ },
+ "4985": {
+ "file_id": 429,
+ "content": "This code defines a class for loading skeleton data. It takes file path, pipeline, and label path (optional) as input parameters. It also has an optional test mode parameter. The `__init__` method initializes the class with provided parameters. The `load_file` method loads feature files to get skeleton information and handles different file types for labels. If a label path is given and it ends with 'npy' or 'pkl', it will load the label; otherwise, it just outputs predictions.",
+ "type": "comment"
+ },
+ "4986": {
+ "file_id": 429,
+ "content": " .format(self.test_mode))\n logger.info(\"Data Loaded!\")\n return self.data # used for __len__\n def prepare_train(self, idx):\n \"\"\"Prepare the feature for training/valid given index. \"\"\"\n results = dict()\n results['data'] = copy.deepcopy(self.data[idx])\n results['label'] = copy.deepcopy(self.label[idx])\n results = self.pipeline(results)\n return results['data'], results['label']\n def prepare_test(self, idx):\n \"\"\"Prepare the feature for test given index. \"\"\"\n results = dict()\n results['data'] = copy.deepcopy(self.data[idx])\n if self.label_path:\n results['label'] = copy.deepcopy(self.label[idx])\n results = self.pipeline(results)\n return results['data'], results['label']\n else:\n results = self.pipeline(results)\n return [results['data']]",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/skeleton.py:56-78"
+ },
+ "4987": {
+ "file_id": 429,
+ "content": "The code defines a class for loading, preparing, and returning data for training or testing. The `__getitem__` method loads the data and returns it when accessed by index. The `prepare_train` method prepares the feature for training/validation given an index. The `prepare_test` method prepares the feature for testing given an index, considering label if available.",
+ "type": "comment"
+ },
+ "4988": {
+ "file_id": 430,
+ "content": "/paddlevideo/loader/dataset/slowfast_video.py",
+ "type": "filepath"
+ },
+ "4989": {
+ "file_id": 430,
+ "content": "PaddleVideo's SFVideoDataset is a video dataset for action recognition, which extends BaseDataset with index file information and optional parameters. It prepares data for training by setting random seeds, loading index files, and appending entries before handling corrupted videos through retry mechanisms and calculating dataset size.",
+ "type": "summary"
+ },
+ "4990": {
+ "file_id": 430,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass SFVideoDataset(BaseDataset):\n \"\"\"Video dataset for action recognition\n The dataset loads raw videos and apply specified transforms on them.\n The index file is a file with multiple lines, and each line indicates",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/slowfast_video.py:1-31"
+ },
+ "4991": {
+ "file_id": 430,
+ "content": "This code snippet is from the PaddleVideo module and defines a class called SFVideoDataset. It is a video dataset for action recognition, loading raw videos and applying specified transforms on them. The index file contains multiple lines with information about each video. The class extends BaseDataset and registers it in the DATASETS registry. The code also includes license and copyright information.",
+ "type": "comment"
+ },
+ "4992": {
+ "file_id": 430,
+ "content": " a sample video with the filepath and label, which are split with a whitesapce.\n Example of a inde file:\n .. code-block:: txt\n path/000.mp4 1\n path/001.mp4 1\n path/002.mp4 2\n path/003.mp4 2\n Args:\n file_path(str): Path to the index file.\n pipeline(XXX): A sequence of data transforms.\n num_ensemble_views(int): temporal segment when multi-crop test\n num_spatial_crops(int): spatial crop number when multi-crop test\n **kwargs: Keyword arguments for ```BaseDataset```.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n num_ensemble_views=1,\n num_spatial_crops=1,\n num_retries=5,\n num_samples_precise_bn=None,\n **kwargs,\n ):\n self.num_ensemble_views = num_ensemble_views\n self.num_spatial_crops = num_spatial_crops\n self.num_retries = num_retries\n self.num_samples_precise_bn = num_samples_precise_bn\n super().__init__(file_path, pipeline, **kwargs)",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/slowfast_video.py:32-64"
+ },
+ "4993": {
+ "file_id": 430,
+ "content": "The code defines a class that represents an index file containing paths to video files and their corresponding labels. It takes arguments such as the path to the index file, data transforms pipeline, and optional parameters for ensemble views and spatial crops. It also includes keyword arguments for the BaseDataset class. The super() function is used to call the parent class's constructor.",
+ "type": "comment"
+ },
+ "4994": {
+ "file_id": 430,
+ "content": " #set random seed\n random.seed(0)\n np.random.seed(0)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split()\n filename, labels = line_split\n if self.data_prefix is not None:\n filename = osp.join(self.data_prefix, filename)\n for tidx in range(self.num_ensemble_views):\n for sidx in range(self.num_spatial_crops):\n info.append(\n dict(\n filename=filename,\n labels=int(labels),\n temporal_sample_index=tidx,\n spatial_sample_index=sidx,\n temporal_num_clips=self.num_ensemble_views,\n spatial_num_clips=self.num_spatial_crops,",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/slowfast_video.py:65-87"
+ },
+ "4995": {
+ "file_id": 430,
+ "content": "Sets random seed for reproducibility, loads index file to get video information, and appends dictionary entries containing filename, labels, temporal, and spatial sample indices.",
+ "type": "comment"
+ },
+ "4996": {
+ "file_id": 430,
+ "content": " ))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n short_cycle = False\n if isinstance(idx, tuple):\n idx, short_cycle_idx = idx\n short_cycle = True\n for ir in range(self.num_retries):\n try:\n #Multi-grid short cycle\n if short_cycle:\n results = copy.deepcopy(self.info[idx])\n results['short_cycle_idx'] = short_cycle_idx\n else:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/slowfast_video.py:88-112"
+ },
+ "4997": {
+ "file_id": 430,
+ "content": "The code is responsible for preparing data for training in the context of a video dataset. It handles potential exceptions caused by reading corrupted video files and allows retries to avoid failures. The function takes an index as input, checks if it's a tuple or not, iterates over a specified number of retries, performs data processing using a pipeline, and handles any exceptions that occur during the process. If there are no exceptions, the results are returned; otherwise, the code logs an error message and tries again.",
+ "type": "comment"
+ },
+ "4998": {
+ "file_id": 430,
+ "content": " idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'][0], results['imgs'][1], np.array(\n [results['labels']])\n def prepare_test(self, idx):\n \"\"\"TEST. Prepare the data for test given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'][0], results['imgs'][1], np.array(\n [results['labels']]), np.array([idx])\n def __len__(self):",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/slowfast_video.py:113-137"
+ },
+ "4999": {
+ "file_id": 430,
+ "content": "The code is implementing a retry mechanism for loading video files. If a corrupted file is encountered, it will attempt to load another random file up to the specified number of retries. If still unsuccessful, it will return an error. The function also includes a logging system to report exceptions and progress in retry attempts.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/5.json b/docs/data/5.json
new file mode 100644
index 000000000..af2c9f392
--- /dev/null
+++ b/docs/data/5.json
@@ -0,0 +1,551 @@
+{
+ "500": {
+ "file_id": 54,
+ "content": " if ('head.' + k) not in model.state_dict().keys():\n print(f'pretrained -----{k} -------is not in model')\n write_dict(state_dicts, 'model_for_infer.txt', **cfg)\n model.set_state_dict(state_dicts)\n inter_file = open(\n os.path.join(\n cfg.get(\"output_dir\", f\"./output/{cfg['model_name']}\"),\n 'inter_file.txt'), 'w')\n seen_seq = False\n with paddle.no_grad():\n # Get the current iteration scribbles\n for scribbles, first_scribble in get_scribbles():\n t_total = timeit.default_timer()\n f, h, w = images.shape[:3]\n if 'prev_label_storage' not in locals().keys():\n prev_label_storage = paddle.zeros([f, h, w])\n if len(annotated_frames(scribbles)) == 0:\n final_masks = prev_label_storage\n # ToDo To AP-kai: save_path传过来了\n submit_masks(cfg[\"save_path\"], final_masks.numpy(), images)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:88-109"
+ },
+ "501": {
+ "file_id": 54,
+ "content": "This code segment checks if certain keys are present in the model's state dictionary. If not, it prints a message and writes the state dictionaries to a file named 'model_for_infer.txt'. It then sets the model's state dict with the state dictionaries, opens an inter_file.txt for writing, and initializes a variable 'seen_seq' as False. Inside a no_grad context, it retrieves scribbles and iterates over them, calculating total time, image shape, and checks if there are any annotated frames. If not, it assigns the previous label storage as final masks and submits those masks to the specified save path with corresponding images.",
+ "type": "comment"
+ },
+ "502": {
+ "file_id": 54,
+ "content": " continue\n # if no scribbles return, keep masks in previous round\n start_annotated_frame = annotated_frames(scribbles)[0]\n pred_masks = []\n pred_masks_reverse = []\n if first_scribble: # If in the first round, initialize memories\n n_interaction = 1\n eval_global_map_tmp_dic = {}\n local_map_dics = ({}, {})\n total_frame_num = f\n else:\n n_interaction += 1\n inter_file.write(sequence + ' ' + 'interaction' +\n str(n_interaction) + ' ' + 'frame' +\n str(start_annotated_frame) + '\\n')\n if first_scribble: # if in the first round, extract pixel embbedings.\n if not seen_seq:\n seen_seq = True\n inter_turn = 1\n embedding_memory = []\n places = paddle.set_device('cpu')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:110-134"
+ },
+ "503": {
+ "file_id": 54,
+ "content": "The code handles the first round of scribbles and initializes memory for future interactions. It writes information to an inter_file, extracts pixel embeddings if it's the first round, and sets up variables for tracking interactions and embedding memories.",
+ "type": "comment"
+ },
+ "504": {
+ "file_id": 54,
+ "content": " for imgs in images:\n if cfg['PIPELINE'].get('test'):\n imgs = paddle.to_tensor([\n build_pipeline(cfg['PIPELINE'].test)({\n 'img1':\n imgs\n })['img1']\n ])\n else:\n imgs = paddle.to_tensor([imgs])\n if parallel:\n for c in model.children():\n frame_embedding = c.head.extract_feature(\n imgs)\n else:\n frame_embedding = model.head.extract_feature(\n imgs)\n embedding_memory.append(frame_embedding)\n del frame_embedding\n embedding_memory = paddle.concat(embedding_memory, 0)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:136-157"
+ },
+ "505": {
+ "file_id": 54,
+ "content": "This code is iterating through each image in a batch and applying a pipeline transformation if testing mode is enabled. It then creates frame embeddings either by looping over model children or directly from the model head. The frame embeddings are appended to a list, concatenated, and stored as embedding_memory.",
+ "type": "comment"
+ },
+ "506": {
+ "file_id": 54,
+ "content": " _, _, emb_h, emb_w = embedding_memory.shape\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n else:\n inter_turn += 1\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n else:\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n ########\n scribble_masks = scribbles2mask(scribbles, (emb_h, emb_w))\n scribble_label = scribble_masks[start_annotated_frame]\n scribble_sample = {'scribble_label': scribble_label}\n scribble_sample = ToTensor_manet()(scribble_sample)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:158-176"
+ },
+ "507": {
+ "file_id": 54,
+ "content": "The code initializes the reference frame embedding and handles cases where the annotation is present or not. It extracts the reference frame embedding from the embedding memory, reshapes it, and then creates a scribble sample with the scribble label for further processing.",
+ "type": "comment"
+ },
+ "508": {
+ "file_id": 54,
+ "content": " # print(ref_frame_embedding, ref_frame_embedding.shape)\n scribble_label = scribble_sample['scribble_label']\n scribble_label = scribble_label.unsqueeze(0)\n model_name = cfg['model_name']\n output_dir = cfg.get(\"output_dir\", f\"./output/{model_name}\")\n inter_file_path = os.path.join(\n output_dir, sequence, 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn))\n if is_save_image:\n ref_scribble_to_show = scribble_label.squeeze().numpy()\n im_ = Image.fromarray(\n ref_scribble_to_show.astype('uint8')).convert('P', )\n im_.putpalette(_palette)\n ref_img_name = str(start_annotated_frame)\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im_.save(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:177-195"
+ },
+ "509": {
+ "file_id": 54,
+ "content": "This code snippet is responsible for saving an interactive scribble image. It first retrieves the scribble label, then constructs the file path to save the image based on configuration settings and iteration parameters. If the directory doesn't exist, it creates one. Finally, it saves the scribble image using a specific palette.",
+ "type": "comment"
+ },
+ "510": {
+ "file_id": 54,
+ "content": " os.path.join(inter_file_path,\n 'inter_' + ref_img_name + '.png'))\n if first_scribble:\n prev_label = None\n prev_label_storage = paddle.zeros([f, h, w])\n else:\n prev_label = prev_label_storage[start_annotated_frame]\n prev_label = prev_label.unsqueeze(0).unsqueeze(0)\n # check if no scribbles.\n if not first_scribble and paddle.unique(\n scribble_label).shape[0] == 1:\n print(\n 'not first_scribble and paddle.unique(scribble_label).shape[0] == 1'\n )\n print(paddle.unique(scribble_label))\n final_masks = prev_label_storage\n submit_masks(cfg[\"save_path\"], final_masks.numpy(), images)\n continue\n ###inteaction segmentation head\n if parallel:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:196-216"
+ },
+ "511": {
+ "file_id": 54,
+ "content": "This code segment is part of a video modeling framework. It deals with handling scribbles and generating masks based on them. If there are no scribbles after the first one, it prints a message and continues execution by submitting the previous label storage as final masks. This code also checks for parallel processing and seems to be part of an interaction segmentation head.",
+ "type": "comment"
+ },
+ "512": {
+ "file_id": 54,
+ "content": " for c in model.children():\n tmp_dic, local_map_dics = c.head.int_seghead(\n ref_frame_embedding=ref_frame_embedding,\n ref_scribble_label=scribble_label,\n prev_round_label=prev_label,\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n frame_num=[start_annotated_frame],\n first_inter=first_scribble)\n else:\n tmp_dic, local_map_dics = model.head.int_seghead(\n ref_frame_embedding=ref_frame_embedding,\n ref_scribble_label=scribble_label,\n prev_round_label=prev_label,\n global_map_tmp_dic=eval_global_map_tmp_dic,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:217-234"
+ },
+ "513": {
+ "file_id": 54,
+ "content": "This code is part of the Manet_Stage1 segmentation model in PaddleVideo. It iterates through the children of the model and calls the 'int_seghead' function to generate temporary dictionaries and local map dictionaries for each child. The 'int_seghead' function takes various parameters such as reference frame embedding, previous round label, global map temporary dictionary, etc., and returns a tuple containing the temporary dictionary and local map dictionaries. If there are no children in the model, it directly calls the 'int_seghead' function on the model's head for the same set of parameters.",
+ "type": "comment"
+ },
+ "514": {
+ "file_id": 54,
+ "content": " local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n frame_num=[start_annotated_frame],\n first_inter=first_scribble)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks.append(float_(pred_label))\n # np.unique(pred_label)\n # array([0], dtype=int64)\n prev_label_storage[start_annotated_frame] = float_(\n pred_label[0])\n if is_save_image: # save image\n pred_label_to_save = pred_label.squeeze(0).numpy()",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:235-254"
+ },
+ "515": {
+ "file_id": 54,
+ "content": "Creates a temporary dictionary with local maps and other parameters. Obtains the predicted label for the sequence, interpolates it to original size, gets the argument of maximum value along axis 1, adds it to prediction masks list, stores the first predicted label for current frame in prev_label_storage if saving images, converts pred_label to numpy array and displays unique elements.",
+ "type": "comment"
+ },
+ "516": {
+ "file_id": 54,
+ "content": " im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)\n imgname = str(start_annotated_frame)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im.save(os.path.join(inter_file_path, imgname + '.png'))\n #######################################\n if first_scribble:\n scribble_label = rough_ROI(scribble_label)\n ##############################\n ref_prev_label = pred_label.unsqueeze(0)\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = ref_frame_embedding\n for ii in range(start_annotated_frame + 1, total_frame_num):\n current_embedding = embedding_memory[ii]\n current_embedding = current_embedding.unsqueeze(0)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:255-274"
+ },
+ "517": {
+ "file_id": 54,
+ "content": "The code segment is generating annotated images from predicted labels and creating scribble-based reference labels. It saves the images in a specified folder path, and initializes variables for iterating through the frames of the video.",
+ "type": "comment"
+ },
+ "518": {
+ "file_id": 54,
+ "content": " prev_label = prev_label\n if parallel:\n for c in model.children():\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = c.head.prop_seghead(\n ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:275-292"
+ },
+ "519": {
+ "file_id": 54,
+ "content": "The code iterates over the model's children and calls `prop_seghead` on each child, passing relevant embeddings and labels to calculate local maps and global maps for segmentation. It also takes into account nearest neighbors, interaction numbers, and annotated frame start.",
+ "type": "comment"
+ },
+ "520": {
+ "file_id": 54,
+ "content": " frame_num=[ii],\n dynamic_seghead=c.head.dynamic_seghead)\n else:\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = model.head.prop_seghead(\n ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:293-310"
+ },
+ "521": {
+ "file_id": 54,
+ "content": "Code segment is part of a larger function in PaddleVideo library. It checks if frame number is the start_annotated_frame, if so, it extracts the current embedding, else it calls head.prop_seghead to get temporary dictionary, global map temporary dictionary and local maps based on reference frame embedding, previous embedding, current embedding, scribble label and previous label using Paddle (a deep learning framework). It also considers K nearest neighbors and interaction number while performing its operation.",
+ "type": "comment"
+ },
+ "522": {
+ "file_id": 54,
+ "content": " frame_num=[ii],\n dynamic_seghead=model.head.dynamic_seghead)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks.append(float_(pred_label))\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = current_embedding\n prev_label_storage[ii] = float_(pred_label[0])\n if is_save_image:\n pred_label_to_save = pred_label.squeeze(0).numpy()\n im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:311-327"
+ },
+ "523": {
+ "file_id": 54,
+ "content": "This code segment is responsible for predicting the labels, creating masks and storing them in a list, and possibly saving an image. The predicted label is interpolated to match the frame size, converted to mask and added to the list of masks. This process continues for each frame. If saving images, the predicted labels are converted to an image format and saved as a grayscale PALETTE image.",
+ "type": "comment"
+ },
+ "524": {
+ "file_id": 54,
+ "content": " imgname = str(ii)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im.save(os.path.join(inter_file_path,\n imgname + '.png'))\n #######################################\n prev_label = ref_prev_label\n prev_embedding = ref_frame_embedding\n #######\n # Propagation <-\n for ii in range(start_annotated_frame):\n current_frame_num = start_annotated_frame - 1 - ii\n current_embedding = embedding_memory[current_frame_num]\n current_embedding = current_embedding.unsqueeze(0)\n prev_label = prev_label\n if parallel:\n for c in model.children():\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = c.head.prop_seghead(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:328-347"
+ },
+ "525": {
+ "file_id": 54,
+ "content": "Code snippet saves frames to disk, initializes variables for propagation loop, and begins the propagation process by iterating through frames from start_annotated_frame down to 0. The model's children are then processed in parallel for segmentation head propagation.",
+ "type": "comment"
+ },
+ "526": {
+ "file_id": 54,
+ "content": " ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,\n frame_num=[current_frame_num],\n dynamic_seghead=c.head.dynamic_seghead)\n else:\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = model.head.prop_seghead(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:348-365"
+ },
+ "527": {
+ "file_id": 54,
+ "content": "This code appears to be part of a deep learning model for video segmentation. It is calling the \"prop_seghead\" function from the \"model.head\" object with specific parameters including reference frame embedding, previous and current embeddings, scribble label, and previous label. If certain conditions are met, additional parameters such as normalize nearest neighbor distances, use local map, sequence names, ground truth IDs, number of nearest neighbors, start annotated frame, and dynamic seghead are passed. The function returns a temporary dictionary, evaluation global map temporary dictionary, and local map dictionaries.",
+ "type": "comment"
+ },
+ "528": {
+ "file_id": 54,
+ "content": " ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,\n frame_num=[current_frame_num],\n dynamic_seghead=model.head.dynamic_seghead)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:366-383"
+ },
+ "529": {
+ "file_id": 54,
+ "content": "This code is calculating the predictions for a specific sequence by using various embeddings, labels, and configurations. It involves interacting with multiple dictionaries, tensor operations, and a dynamic seghead model. The predicted label is then interpolated to match the resolution of the original frame.",
+ "type": "comment"
+ },
+ "530": {
+ "file_id": 54,
+ "content": " size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks_reverse.append(float_(pred_label))\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = current_embedding\n ####\n prev_label_storage[current_frame_num] = float_(\n pred_label[0])\n ###\n if is_save_image:\n pred_label_to_save = pred_label.squeeze(0).numpy()\n im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)\n imgname = str(current_frame_num)\n while len(imgname) < 5:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:384-402"
+ },
+ "531": {
+ "file_id": 54,
+ "content": "This code snippet is part of an image segmentation model. It extracts predictions from the model, converts them to masks, and stores previous label information for each frame. Additionally, it saves visualizations of these predictions as palette-colored images.",
+ "type": "comment"
+ },
+ "532": {
+ "file_id": 54,
+ "content": " imgname = '0' + imgname\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im.save(os.path.join(inter_file_path,\n imgname + '.png'))\n pred_masks_reverse.reverse()\n pred_masks_reverse.extend(pred_masks)\n final_masks = paddle.concat(pred_masks_reverse, 0)\n submit_masks(cfg[\"save_path\"], final_masks.numpy(), images)\n t_end = timeit.default_timer()\n print('Total time for single interaction: ' +\n str(t_end - t_total))\n inter_file.close()\n return None",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py:403-417"
+ },
+ "533": {
+ "file_id": 54,
+ "content": "This code saves images and their corresponding masks, creates final masks, and writes the total time for a single interaction. It handles non-existent folders by creating them before saving images.",
+ "type": "comment"
+ },
+ "534": {
+ "file_id": 55,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py",
+ "type": "filepath"
+ },
+ "535": {
+ "file_id": 55,
+ "content": "This function computes L2 distances, applies nearest neighbor attention and feature extraction, considers padding, uses local search windows and average pooling. It introduces a custom layer, calculates nearest neighbor features with embeddings, updates global map dictionaries, and processes inputs to return output dictionaries after calculations on local distance maps for each frame. The code segment updates global and local map dictionaries, calculates frame embeddings and masks, obtains segmentation predictions, and processes data for improved video processing accuracy.",
+ "type": "summary"
+ },
+ "536": {
+ "file_id": 55,
+ "content": "import numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom EIVideo.paddlevideo.utils.manet_utils import int_, float_, long_, load\nfrom EIVideo.paddlevideo.utils.manet_utils import kaiming_normal_\n#############################################################GLOBAL_DIST_MAP\nMODEL_UNFOLD = True\nWRONG_LABEL_PADDING_DISTANCE = 1e20\ndef _pairwise_distances(x, y, ys=None):\n \"\"\"Computes pairwise squared l2 distances between tensors x and y.\n Args:\n x: Tensor of shape [n, feature_dim].\n y: Tensor of shape [m, feature_dim].\n Returns:\n Float32 distances tensor of shape [n, m].\n \"\"\"\n xs = paddle.sum(x * x, 1)\n xs = xs.unsqueeze(1)\n if ys is None:\n ys = paddle.sum(y * y, 1)\n ys = ys.unsqueeze(0)\n else:\n ys = ys\n d = xs + ys - 2. * paddle.matmul(x, paddle.t(y))\n return d, ys\n##################\ndef _flattened_pairwise_distances(reference_embeddings, query_embeddings, ys):\n \"\"\"Calculates flattened tensor of pairwise distances between ref and query.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:1-37"
+ },
+ "537": {
+ "file_id": 55,
+ "content": "This code defines a function that calculates pairwise squared L2 distances between two tensors. It takes in two tensors, x and y, and optionally a third tensor ys. The function first computes the sum of squares for each row in tensor x and stores them in xs. If ys is None, it then computes the sum of squares for each row in tensor y and stores them in ys. Otherwise, it uses the provided ys. Finally, the function calculates the pairwise distances using the formula xs + ys - 2 * paddle.matmul(x, paddle.t(y)).",
+ "type": "comment"
+ },
+ "538": {
+ "file_id": 55,
+ "content": " Args:\n reference_embeddings: Tensor of shape [..., embedding_dim],\n the embedding vectors for the reference frame\n query_embeddings: Tensor of shape [n_query_images, height, width,\n embedding_dim], the embedding vectors for the query frames.\n Returns:\n A distance tensor of shape [reference_embeddings.size / embedding_dim,\n query_embeddings.size / embedding_dim]\n \"\"\"\n embedding_dim = query_embeddings.shape[-1]\n reference_embeddings = reference_embeddings.reshape([-1, embedding_dim])\n first_dim = -1\n query_embeddings = query_embeddings.reshape([first_dim, embedding_dim])\n dists, ys = _pairwise_distances(query_embeddings, reference_embeddings, ys)\n return dists, ys\ndef _nn_features_per_object_for_chunk(reference_embeddings, query_embeddings,\n wrong_label_mask, k_nearest_neighbors,\n ys):\n \"\"\"Extracts features for each object using nearest neighbor attention.\n Args:\n reference_embeddings: Tensor of shape [n_chunk, embedding_dim],",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:38-60"
+ },
+ "539": {
+ "file_id": 55,
+ "content": "This function takes reference and query embeddings as input, calculates pairwise distances between them using the _pairwise_distances function, and returns the distances in dists and ys. The _nn_features_per_object_for_chunk function extracts features for each object using nearest neighbor attention, taking reference embeddings, query embeddings, wrong_label_mask, k_nearest_neighbors, and ys as input.",
+ "type": "comment"
+ },
+ "540": {
+ "file_id": 55,
+ "content": " the embedding vectors for the reference frame.\n query_embeddings: Tensor of shape [m_chunk, embedding_dim], the embedding\n vectors for the query frames.\n wrong_label_mask:\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m_chunk, n_objects, feature_dim].\n \"\"\"\n # reference_embeddings_key = reference_embeddings\n # query_embeddings_key = query_embeddings\n dists, ys = _flattened_pairwise_distances(reference_embeddings,\n query_embeddings, ys)\n dists = (paddle.unsqueeze(dists, 1) +\n paddle.unsqueeze(float_(wrong_label_mask), 0) *\n WRONG_LABEL_PADDING_DISTANCE)\n if k_nearest_neighbors == 1:\n features = paddle.min(dists, 2, keepdim=True)\n else:\n dists, _ = paddle.topk(-dists, k=k_nearest_neighbors, axis=2)\n dists = -dists\n valid_mask = (dists < WRONG_LABEL_PADDING_DISTANCE)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:61-83"
+ },
+ "541": {
+ "file_id": 55,
+ "content": "This function calculates the pairwise distances between reference and query embeddings, selects the nearest neighbors based on those distances, and returns the nearest neighbor features. It handles cases with different numbers of reference and query embeddings by padding with a specified distance value for missing embeddings.",
+ "type": "comment"
+ },
+ "542": {
+ "file_id": 55,
+ "content": " masked_dists = dists * valid_mask.float()\n pad_dist = paddle.max(masked_dists, axis=2, keepdim=True)[0].tile(\n (1, 1, masked_dists.shape[-1]))\n dists = paddle.where(valid_mask, dists, pad_dist)\n # take mean of distances\n features = paddle.mean(dists, axis=2, keepdim=True)\n return features, ys\n###\ndef _selected_pixel(ref_labels_flat, ref_emb_flat):\n index_list = paddle.arange(len(ref_labels_flat))\n index_list = index_list\n index_ = paddle.masked_select(index_list, ref_labels_flat != -1)\n index_ = long_(index_)\n ref_labels_flat = paddle.index_select(ref_labels_flat, index_, 0)\n ref_emb_flat = paddle.index_select(ref_emb_flat, index_, 0)\n return ref_labels_flat, ref_emb_flat\n###\ndef _nearest_neighbor_features_per_object_in_chunks(reference_embeddings_flat,\n query_embeddings_flat,\n reference_labels_flat,\n ref_obj_ids,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:84-113"
+ },
+ "543": {
+ "file_id": 55,
+ "content": "The code calculates the mean of distances between valid points and assigns the result to \"features\". The function _selected_pixel() selects pixels from flattened arrays where reference labels are not -1. The function _nearest_neighbor_features_per_object_in_chunks() operates on flattened embeddings, labels, and object ids to compute nearest neighbor features per object in chunks.",
+ "type": "comment"
+ },
+ "544": {
+ "file_id": 55,
+ "content": " k_nearest_neighbors,\n n_chunks, **cfg):\n \"\"\"Calculates the nearest neighbor features per object in chunks to save mem.\n Uses chunking to bound the memory use.\n Args:\n reference_embeddings_flat: Tensor of shape [n, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings_flat: Tensor of shape [m, embedding_dim], the embedding\n vectors for the query frames.\n reference_labels_flat: Tensor of shape [n], the class labels of the\n reference frame.\n ref_obj_ids: int tensor of unique object ids in the reference labels.\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m, n_objects, feature_dim].\n \"\"\"\n # reference_embeddings_flat = reference_embeddings_flat.cpu()",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:114-134"
+ },
+ "545": {
+ "file_id": 55,
+ "content": "This function calculates the nearest neighbor features per object in chunks to save memory, using chunking for bounding memory usage. It takes embedding vectors for reference and query frames, their class labels, unique object IDs, number of nearest neighbors, and number of chunks as input. The function returns a tensor of nearest neighbor features for the query frames.",
+ "type": "comment"
+ },
+ "546": {
+ "file_id": 55,
+ "content": " # query_embeddings_flat = query_embeddings_flat.cpu()\n # reference_labels_flat = reference_labels_flat.cpu()\n # ref_obj_ids = ref_obj_ids.cpu()\n chunk_size = int_(\n np.ceil((float_(query_embeddings_flat.shape[0]) / n_chunks).numpy()))\n if cfg.get('test_mode'):\n reference_labels_flat, reference_embeddings_flat = _selected_pixel(\n reference_labels_flat, reference_embeddings_flat)\n wrong_label_mask = (reference_labels_flat != paddle.unsqueeze(\n ref_obj_ids, 1))\n all_features = []\n for n in range(n_chunks):\n if n == 0:\n ys = None\n if n_chunks == 1:\n query_embeddings_flat_chunk = query_embeddings_flat\n else:\n chunk_start = n * chunk_size\n chunk_end = (n + 1) * chunk_size\n query_embeddings_flat_chunk = query_embeddings_flat[\n chunk_start:chunk_end]\n features, ys = _nn_features_per_object_for_chunk(\n reference_embeddings_flat, query_embeddings_flat_chunk,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:135-158"
+ },
+ "547": {
+ "file_id": 55,
+ "content": "This code splits the query embeddings into multiple chunks, depending on the number of chunks specified. It then applies a function to each chunk and appends the results to the all_features list. If in test mode, it selects pixels from the reference and query embeddings. It also creates a wrong label mask for the reference labels and query embeddings.",
+ "type": "comment"
+ },
+ "548": {
+ "file_id": 55,
+ "content": " wrong_label_mask, k_nearest_neighbors, ys)\n all_features.append(features)\n if n_chunks == 1:\n nn_features = all_features[0]\n else:\n nn_features = paddle.concat(all_features, axis=0)\n return nn_features\ndef nearest_neighbor_features_per_object(reference_embeddings,\n query_embeddings,\n reference_labels,\n k_nearest_neighbors,\n gt_ids=None,\n n_chunks=100,\n **cfg):\n \"\"\"Calculates the distance to the nearest neighbor per object.\n For every pixel of query_embeddings calculate the distance to the\n nearest neighbor in the (possibly subsampled) reference_embeddings per object.\n Args:\n reference_embeddings: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings: Tensor of shape [n_query_images, height, width,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:159-181"
+ },
+ "549": {
+ "file_id": 55,
+ "content": "This code calculates the nearest neighbor features per object using reference embeddings, query embeddings, and reference labels. It takes into account k-nearest neighbors and can handle a specified number of chunks for subsampling. The function returns the nearest neighbor features in the form of a tensor.",
+ "type": "comment"
+ },
+ "550": {
+ "file_id": 55,
+ "content": " embedding_dim], the embedding vectors for the query frames.\n reference_labels: Tensor of shape [height, width, 1], the class labels of\n the reference frame.\n max_neighbors_per_object: Integer, the maximum number of candidates\n for the nearest neighbor query per object after subsampling,\n or 0 for no subsampling.\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n gt_ids: Int tensor of shape [n_objs] of the sorted unique ground truth\n ids in the first frame. If None, it will be derived from\n reference_labels.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [n_query_images, height, width, n_objects, feature_dim].\n gt_ids: An int32 tensor of the unique sorted object ids present\n in the reference labels.\n \"\"\"\n # reference_embeddings = reference_embeddings.detach().cpu()\n # query_embeddings = query_embeddings.detach().cpu()",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:182-201"
+ },
+ "551": {
+ "file_id": 55,
+ "content": "This code calculates nearest neighbors for query frames based on the given embedding vectors. It takes input parameters like reference frame class labels, maximum number of candidates, and number of nearest neighbors to use. The function returns nearest neighbor features, unique sorted object ids present in the reference labels, and potentially gt_ids if provided.",
+ "type": "comment"
+ },
+ "552": {
+ "file_id": 55,
+ "content": " # reference_labels = reference_labels.detach().cpu()\n assert (reference_embeddings.shape[:2] == reference_labels.shape[:2])\n h, w, _ = query_embeddings.shape\n reference_labels_flat = reference_labels.reshape([-1])\n if gt_ids is None:\n ref_obj_ids = paddle.unique(reference_labels_flat)[-1]\n ref_obj_ids = np.arange(0, ref_obj_ids + 1)\n gt_ids = paddle.to_tensor(ref_obj_ids)\n gt_ids = int_(gt_ids)\n else:\n gt_ids = int_(paddle.arange(0, gt_ids + 1))\n embedding_dim = query_embeddings.shape[-1]\n query_embeddings_flat = query_embeddings.reshape([-1, embedding_dim])\n reference_embeddings_flat = reference_embeddings.reshape(\n [-1, embedding_dim])\n nn_features = _nearest_neighbor_features_per_object_in_chunks(\n reference_embeddings_flat, query_embeddings_flat,\n reference_labels_flat, gt_ids, k_nearest_neighbors, n_chunks, **cfg)\n nn_features_dim = nn_features.shape[-1]\n nn_features = nn_features.reshape(\n [1, h, w, gt_ids.shape[0], nn_features_dim])",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:202-224"
+ },
+ "553": {
+ "file_id": 55,
+ "content": "This code is reshaping tensors and calculating nearest neighbor features for each object in chunks. It first reshapes the embeddings, then applies a function to find the closest neighbors and returns a tensor of these features. This process is done in chunks for efficiency and memory management.",
+ "type": "comment"
+ },
+ "554": {
+ "file_id": 55,
+ "content": " return nn_features.cuda(), gt_ids\n########################################################################LOCAL_DIST_MAP\ndef local_pairwise_distances2(x, y, max_distance=9):\n \"\"\"Computes pairwise squared l2 distances using a local search window.\n Naive implementation using map_fn.\n Used as a slow fallback for when correlation_cost is not available.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].\n y: Float32 tensor of shape [height, width, feature_dim].\n max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n Returns:\n Float32 distances tensor of shape\n [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n ori_h, ori_w, _ = x.shape\n x = paddle.transpose(x, [2, 0, 1]).unsqueeze(0)\n x = F.avg_pool2d(x, (2, 2), (2, 2))\n y = paddle.transpose(y, [2, 0, 1]).unsqueeze(0)\n y = F.avg_pool2d(y, (2, 2), (2, 2))\n _, channels, height, width = x.shape\n padding_val = 1e20\n padded_y = F.pad(y,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:225-252"
+ },
+ "555": {
+ "file_id": 55,
+ "content": "This function calculates pairwise squared L2 distances using a local search window, with naive implementation using map_fn. It is used as a fallback when correlation_cost is not available. Inputs are tensors x and y of shape [height, width, feature\\_dim]. It returns a tensor of squared distance values shaped [height, width, (2 * max\\_distance + 1) ** 2], where max\\_distance is an integer representing the maximum distance in pixel coordinates per dimension. The function also applies average pooling with a 2x2 filter and pads the tensors x and y before calculating the distances.",
+ "type": "comment"
+ },
+ "556": {
+ "file_id": 55,
+ "content": " (max_distance, max_distance, max_distance, max_distance),\n mode='constant',\n value=padding_val)\n offset_y = F.unfold(padded_y, kernel_sizes=[height, width]).reshape(\n [1, channels, height, width, -1])\n x = x.reshape([1, channels, height, width, 1])\n minus = x - offset_y\n dists = paddle.sum(paddle.multiply(minus, minus),\n axis=1).reshape([1, height, width,\n -1]).transpose([0, 3, 1, 2])\n dists = (paddle.nn.functional.sigmoid(dists) - 0.5) * 2\n dists = F.interpolate(dists,\n size=[ori_h, ori_w],\n mode='bilinear',\n align_corners=True)\n dists = dists.squeeze(0).transpose([1, 2, 0])\n return dists\ndef local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding,\n query_embedding,\n prev_frame_labels,\n gt_ids,\n max_distance=12):\n \"\"\"Computes nearest neighbor features while only allowing local matches.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:253-278"
+ },
+ "557": {
+ "file_id": 55,
+ "content": "This code calculates the nearest neighbor features for local matches in a video. It takes in parameters like previous frame embedding, query embedding, previous frame labels, and ground truth IDs. The function computes distances between frames using Sigmoid activation and bilinear interpolation. Max distance determines the maximum allowed distance for a match to be considered valid.",
+ "type": "comment"
+ },
+ "558": {
+ "file_id": 55,
+ "content": " Args:\n prev_frame_embedding: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the last frame.\n query_embedding: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the query frames.\n prev_frame_labels: Tensor of shape [height, width, 1], the class labels of\n the previous frame.\n gt_ids: Int Tensor of shape [n_objs] of the sorted unique ground truth\n ids in the first frame.\n max_distance: Integer, the maximum distance allowed for local matching.\n Returns:\n nn_features: A float32 np.array of nearest neighbor features of shape\n [1, height, width, n_objects, 1].\n \"\"\"\n # print(query_embedding.shape, prev_frame_embedding.shape)\n # print(query_embedding.place, prev_frame_embedding.place)\n # query_embedding = query_embedding.cpu()\n # prev_frame_embedding = prev_frame_embedding.cpu()\n # prev_frame_labels = prev_frame_labels.cpu()\n # print(prev_frame_labels.place, prev_frame_embedding.place, query_embedding.place)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:279-298"
+ },
+ "559": {
+ "file_id": 55,
+ "content": "This code calculates the nearest neighbor features by comparing embedding vectors of query frames with the last frame. It takes input tensors for embedding vectors, previous frame labels, and ground truth IDs along with a maximum distance limit. The function returns the nearest neighbor features in a specific shape.",
+ "type": "comment"
+ },
+ "560": {
+ "file_id": 55,
+ "content": " d = local_pairwise_distances2(query_embedding,\n prev_frame_embedding,\n max_distance=max_distance)\n height, width = prev_frame_embedding.shape[:2]\n if MODEL_UNFOLD:\n labels = float_(prev_frame_labels).transpose([2, 0, 1]).unsqueeze(0)\n padded_labels = F.pad(labels, (\n 2 * max_distance,\n 2 * max_distance,\n 2 * max_distance,\n 2 * max_distance,\n ))\n offset_labels = F.unfold(padded_labels,\n kernel_sizes=[height, width],\n strides=[2,\n 2]).reshape([height, width, -1, 1])\n offset_masks = paddle.equal(\n offset_labels,\n float_(gt_ids).unsqueeze(0).unsqueeze(0).unsqueeze(0))\n else:\n masks = paddle.equal(prev_frame_labels,\n gt_ids.unsqueeze(0).unsqueeze(0))\n padded_masks = nn.functional.pad(masks, (",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:300-325"
+ },
+ "561": {
+ "file_id": 55,
+ "content": "Code snippet performs local pairwise distance calculation between query and previous frame embeddings. If MODEL_UNFOLD is enabled, it generates offset labels by unfolding padded labels with kernel sizes matching height and width of the previous frame embedding. It then creates offset masks by checking equality between offset labels and gt_ids. If MODEL_UNFOLD is not enabled, it directly creates masks by comparing previous frame labels and gt_ids. Finally, it pads the masks using nn.functional.pad with specified padding values.",
+ "type": "comment"
+ },
+ "562": {
+ "file_id": 55,
+ "content": " 0,\n 0,\n max_distance,\n max_distance,\n max_distance,\n max_distance,\n ))\n offset_masks = []\n for y_start in range(2 * max_distance + 1):\n y_end = y_start + height\n masks_slice = padded_masks[y_start:y_end]\n for x_start in range(2 * max_distance + 1):\n x_end = x_start + width\n offset_mask = masks_slice[:, x_start:x_end]\n offset_masks.append(offset_mask)\n offset_masks = paddle.stack(offset_masks, axis=2)\n d_tiled = d.unsqueeze(-1).tile((1, 1, 1, gt_ids.shape[0]))\n pad = paddle.ones_like(d_tiled)\n d_masked = paddle.where(offset_masks, d_tiled, pad)\n dists = paddle.min(d_masked, axis=2)\n dists = dists.reshape([1, height, width, gt_ids.shape[0], 1])\n return dists\n##############################################################\n#################\nclass _res_block(nn.Layer):\n def __init__(self, in_dim, out_dim, **cfg):\n super(_res_block, self).__init__()",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:326-358"
+ },
+ "563": {
+ "file_id": 55,
+ "content": "The code is performing feature extraction and masking for a specific model. It first tiles input data, then applies offset masks to selected regions, and finally extracts minimum distances using the tiled and masked data. The result is a new set of distances which are then reshaped for further processing.",
+ "type": "comment"
+ },
+ "564": {
+ "file_id": 55,
+ "content": " self.conv1 = nn.Conv2D(in_dim,\n out_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.relu1 = nn.ReLU()\n self.bn1 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg['train_bn_mom'])\n self.conv2 = nn.Conv2D(out_dim,\n out_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.relu2 = nn.ReLU()\n self.bn2 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg['train_bn_mom'])\n def forward(self, x):\n res = x\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu2(x)\n x += res\n return x\n####################\nclass IntSegHead(nn.Layer):\n def __init__(self, in_dim, emb_dim, **cfg):\n super(IntSegHead, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:359-390"
+ },
+ "565": {
+ "file_id": 55,
+ "content": "This code defines a convolutional neural network (CNN) architecture for image processing tasks. The class `IntVOS` contains two 2D convolutions, batch normalization, and ReLU activations in its forward pass. The `IntSegHead` class initializes another CNN with different parameters, which seems to be a part of the overall model. Both classes extend `nn.Layer`, indicating they are PaddlePaddle's version of PyTorch layers.",
+ "type": "comment"
+ },
+ "566": {
+ "file_id": 55,
+ "content": " emb_dim,\n kernel_size=7,\n stride=1,\n padding=3)\n self.bn1 = paddle.nn.BatchNorm2D(emb_dim, momentum=cfg['train_bn_mom'])\n self.relu1 = nn.ReLU(True)\n self.res1 = _res_block(emb_dim, emb_dim, **cfg)\n self.res2 = _res_block(emb_dim, emb_dim, **cfg)\n self.conv2 = nn.Conv2D(256,\n emb_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.bn2 = paddle.nn.BatchNorm2D(emb_dim, momentum=cfg['train_bn_mom'])\n self.relu2 = nn.ReLU(True)\n self.conv3 = nn.Conv2D(emb_dim, 1, 1, 1)\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.res1(x)\n x = self.res2(x)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu2(x)\n x = self.conv3(x)\n return x",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:391-418"
+ },
+ "567": {
+ "file_id": 55,
+ "content": "This code defines a custom Convolutional Neural Network (CNN) layer for extracting features from input images. It consists of multiple convolutions, batch normalizations, and ReLU activations. The input image is first passed through several convolution layers with different configurations, followed by batch normalization and ReLU activation functions to improve model performance. Finally, the output is returned after passing it through a single convolution layer and another batch normalization and ReLU activation.",
+ "type": "comment"
+ },
+ "568": {
+ "file_id": 55,
+ "content": "class _split_separable_conv2d(nn.Layer):\n def __init__(self, in_dim, out_dim, kernel_size=7, **cfg):\n super(_split_separable_conv2d, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,\n in_dim,\n kernel_size=kernel_size,\n stride=1,\n padding=int((kernel_size - 1) / 2),\n groups=in_dim)\n self.relu1 = nn.ReLU(True)\n self.bn1 = paddle.nn.BatchNorm2D(in_dim, momentum=cfg['train_bn_mom'])\n self.conv2 = nn.Conv2D(in_dim, out_dim, kernel_size=1, stride=1)\n self.relu2 = nn.ReLU(True)\n self.bn2 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg['train_bn_mom'])\n kaiming_normal_(self.conv1.weight, mode='fan_out', nonlinearity='relu')\n kaiming_normal_(self.conv2.weight, mode='fan_out', nonlinearity='relu')\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.conv2(x)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:421-442"
+ },
+ "569": {
+ "file_id": 55,
+ "content": "This code defines a custom layer _split_separable_conv2d, which consists of two convolutional layers followed by ReLU and batch normalization. The first convolution is performed with the same number of input and output channels, while the second has fewer output channels than input dimensions. This architecture helps to reduce parameters and computational cost in a deep learning model for image processing tasks.",
+ "type": "comment"
+ },
+ "570": {
+ "file_id": 55,
+ "content": " x = self.bn2(x)\n x = self.relu2(x)\n return x\nclass DynamicSegHead(nn.Layer):\n def __init__(self, in_dim, embed_dim, **cfg):\n super(DynamicSegHead, self).__init__()\n self.layer1 = _split_separable_conv2d(in_dim, embed_dim, **cfg)\n self.layer2 = _split_separable_conv2d(embed_dim, embed_dim, **cfg)\n self.layer3 = _split_separable_conv2d(embed_dim, embed_dim, **cfg)\n self.layer4 = _split_separable_conv2d(embed_dim, embed_dim, **cfg)\n self.conv = nn.Conv2D(embed_dim, 1, 1, 1)\n kaiming_normal_(self.conv.weight, mode='fan_out', nonlinearity='relu')\n def forward(self, x):\n x = self.layer1(x)\n x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n x = self.conv(x)\n return x\nfrom ..registry import HEADS\n\"\"\"\n覆盖原理\nclass c1:\n def __init__(self):\n self.a = 1\nclass c2(c1):\n def __init__(self):\n super(c2, self).__init__()\n self.a = 2\nc = c2()\nprint(c.a)\n\"\"\"\n@HEADS.register()\nclass IntVOS(nn.Layer):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:443-488"
+ },
+ "571": {
+ "file_id": 55,
+ "content": "The code defines two classes: IntVOS and DynamicSegHead. IntVOS is a subclass of nn.Layer and utilizes the DynamicSegHead class as its segmentation head. DynamicSegHead is also a subclass of nn.Layer and consists of several layers (layer1, layer2, layer3, layer4) that apply separable convolutions to the input. Finally, there's a nn.Conv2D layer with Kaiming initialization for the output. This architecture can be used for segmentation tasks in computer vision applications.",
+ "type": "comment"
+ },
+ "572": {
+ "file_id": 55,
+ "content": " def __init__(self, feature_extracter, **cfg):\n super(IntVOS, self).__init__()\n self.feature_extracter = feature_extracter ##embedding extractor\n self.feature_extracter.cls_conv = nn.Sequential()\n self.feature_extracter.upsample4 = nn.Sequential()\n self.semantic_embedding = None\n self.seperate_conv = nn.Conv2D(cfg['model_aspp_outdim'],\n cfg['model_aspp_outdim'],\n kernel_size=3,\n stride=1,\n padding=1,\n groups=cfg['model_aspp_outdim'])\n self.bn1 = paddle.nn.BatchNorm2D(cfg['model_aspp_outdim'],\n momentum=cfg['train_bn_mom'])\n self.relu1 = nn.ReLU(True)\n self.embedding_conv = nn.Conv2D(cfg['model_aspp_outdim'],\n cfg['model_semantic_embedding_dim'], 1,\n 1)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:489-506"
+ },
+ "573": {
+ "file_id": 55,
+ "content": "This code defines a class called IntVOS. The constructor takes in a feature_extracter and **cfg parameters, initializes the instance variables, and adds layers to the feature_extracter if required. It also initializes the embedding convolution layer for semantic embedding extraction.",
+ "type": "comment"
+ },
+ "574": {
+ "file_id": 55,
+ "content": " self.relu2 = nn.ReLU(True)\n self.bn2 = paddle.nn.BatchNorm2D(cfg['model_semantic_embedding_dim'],\n momentum=cfg['train_bn_mom'])\n self.semantic_embedding = nn.Sequential(*[\n self.seperate_conv, self.bn1, self.relu1, self.embedding_conv,\n self.bn2, self.relu2\n ])\n for m in self.semantic_embedding:\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')\n self.dynamic_seghead = DynamicSegHead(\n in_dim=cfg['model_semantic_embedding_dim'] + 3,\n embed_dim=cfg['model_head_embedding_dim'],\n **cfg) # propagation segm head\n if cfg['model_useintseg']:\n self.inter_seghead = IntSegHead(\n in_dim=cfg['model_semantic_embedding_dim'] + 3,\n emb_dim=cfg['model_head_embedding_dim'],\n **cfg)\n else:\n self.inter_seghead = DynamicSegHead(\n in_dim=cfg['model_semantic_embedding_dim'] + 2,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:507-530"
+ },
+ "575": {
+ "file_id": 55,
+ "content": "The code initializes and configures the layers for semantic segmentation. It creates a ReLU activation function, a batch normalization layer with specified parameters, and a sequential neural network containing the separate convolution, first batch norm, first ReLU, embedding convolution, second batch norm, and second ReLU. The code also initializes the dynamic segmentation head and (optionally) an inter-segmentation head depending on the config's 'model_useintseg' flag.",
+ "type": "comment"
+ },
+ "576": {
+ "file_id": 55,
+ "content": " embed_dim=cfg['model_head_embedding_dim'],\n **cfg) # interaction segm head\n self.pretrained = cfg.get('pretrained', None)\n self.cfg = cfg\n def init_weights(self):\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n self.set_state_dict(load(self.pretrained, self.state_dict()))\n print('loaded pretrained model')\n def loss(self, **kwargs):\n return self.loss_func(**kwargs)\n def forward(self,\n x=None,\n ref_scribble_label=None,\n previous_frame_mask=None,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,\n global_map_tmp_dic=None,\n local_map_dics=None,\n interaction_num=None,\n start_annotated_frame=None,\n frame_num=None):\n x = self.extract_feature(x)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:531-559"
+ },
+ "577": {
+ "file_id": 55,
+ "content": "This code defines a class for a model head that takes input, initializes weights (loading pretrained if available), and calculates the loss during forward pass. It uses various input parameters such as x, ref_scribble_label, previous_frame_mask, etc. The forward function extracts features from the input, and is responsible for losses related to the model head.",
+ "type": "comment"
+ },
+ "578": {
+ "file_id": 55,
+ "content": " # print('extract_feature:', x.mean().item())\n ref_frame_embedding, previous_frame_embedding, current_frame_embedding = paddle.split(\n x, num_or_sections=3, axis=0)\n if global_map_tmp_dic is None:\n dic = self.prop_seghead(\n ref_frame_embedding,\n previous_frame_embedding,\n current_frame_embedding,\n ref_scribble_label,\n previous_frame_mask,\n normalize_nearest_neighbor_distances,\n use_local_map,\n seq_names,\n gt_ids,\n k_nearest_neighbors,\n global_map_tmp_dic,\n local_map_dics,\n interaction_num,\n start_annotated_frame,\n frame_num,\n self.dynamic_seghead,\n )\n return dic\n else:\n dic, global_map_tmp_dic = self.prop_seghead(\n ref_frame_embedding,\n previous_frame_embedding,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:560-588"
+ },
+ "579": {
+ "file_id": 55,
+ "content": "This code is splitting input feature x into three parts (ref, previous, current frame embeddings), then calling the prop_seghead function to compute a dictionary of results. If global_map_tmp_dic is None, it returns only the dictionary; otherwise, it also updates global_map_tmp_dic and returns both.",
+ "type": "comment"
+ },
+ "580": {
+ "file_id": 55,
+ "content": " current_frame_embedding,\n ref_scribble_label,\n previous_frame_mask,\n normalize_nearest_neighbor_distances,\n use_local_map,\n seq_names,\n gt_ids,\n k_nearest_neighbors,\n global_map_tmp_dic,\n local_map_dics,\n interaction_num,\n start_annotated_frame,\n frame_num,\n self.dynamic_seghead,\n )\n return dic, global_map_tmp_dic\n def extract_feature(self, x):\n x = self.feature_extracter(x)\n x = self.semantic_embedding(x)\n return x\n def prop_seghead(\n self,\n ref_frame_embedding=None,\n previous_frame_embedding=None,\n current_frame_embedding=None,\n ref_scribble_label=None,\n previous_frame_mask=None,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:589-622"
+ },
+ "581": {
+ "file_id": 55,
+ "content": "This code defines a class with three methods: \"IntVOS\", \"extract_feature\", and \"prop_seghead\". The \"IntVOS\" function returns two dictionaries after performing some operations. The \"extract_feature\" method extracts features from input image using feature extracter and semantic embedding. The \"prop_seghead\" method takes various inputs, including frame embeddings, scribble label, and mask, and performs propagation segmentation head task with optional normalization and local map usage.",
+ "type": "comment"
+ },
+ "582": {
+ "file_id": 55,
+ "content": " global_map_tmp_dic=None,\n local_map_dics=None,\n interaction_num=None,\n start_annotated_frame=None,\n frame_num=None,\n dynamic_seghead=None,\n ):\n \"\"\"return: feature_embedding,global_match_map,local_match_map,previous_frame_mask\"\"\"\n ###############\n cfg = self.cfg\n global_map_tmp_dic = global_map_tmp_dic\n dic_tmp = {}\n bs, c, h, w = current_frame_embedding.shape\n if cfg.get('test_mode'):\n scale_ref_scribble_label = float_(ref_scribble_label)\n else:\n scale_ref_scribble_label = paddle.nn.functional.interpolate(\n float_(ref_scribble_label), size=(h, w), mode='nearest')\n scale_ref_scribble_label = int_(scale_ref_scribble_label)\n scale_previous_frame_label = paddle.nn.functional.interpolate(\n float_(previous_frame_mask), size=(h, w), mode='nearest')\n scale_previous_frame_label = int_(scale_previous_frame_label)\n for n in range(bs):\n seq_current_frame_embedding = current_frame_embedding[n]",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:623-646"
+ },
+ "583": {
+ "file_id": 55,
+ "content": "This function takes in various parameters and returns feature_embedding, global_match_map, local_match_map, and previous_frame_mask. It initializes global_map_tmp_dic, dic_tmp, bs, c, h, w from current_frame_embedding, checks if it is in test mode, scales ref_scribble_label and previous_frame_mask using interpolation for matching dimensions, and then iterates through a range of bs, performing operations on seq_current_frame_embedding.",
+ "type": "comment"
+ },
+ "584": {
+ "file_id": 55,
+ "content": " seq_ref_frame_embedding = ref_frame_embedding[n]\n seq_prev_frame_embedding = previous_frame_embedding[n]\n seq_ref_frame_embedding = seq_ref_frame_embedding.transpose(\n [1, 2, 0])\n seq_current_frame_embedding = seq_current_frame_embedding.transpose(\n [1, 2, 0])\n seq_ref_scribble_label = scale_ref_scribble_label[n].transpose(\n [1, 2, 0])\n #########Global Map\n nn_features_n, ref_obj_ids = nearest_neighbor_features_per_object(\n reference_embeddings=seq_ref_frame_embedding,\n query_embeddings=seq_current_frame_embedding,\n reference_labels=seq_ref_scribble_label,\n k_nearest_neighbors=k_nearest_neighbors,\n gt_ids=gt_ids[n],\n n_chunks=10)\n if normalize_nearest_neighbor_distances:\n nn_features_n = (paddle.nn.functional.sigmoid(nn_features_n) -\n 0.5) * 2",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:647-665"
+ },
+ "585": {
+ "file_id": 55,
+ "content": "This code calculates nearest neighbor features for each object using reference and current frame embeddings, and scribble labels. It transposes the embeddings and label to match the global map format and uses k-nearest neighbors to find the corresponding features. If normalization is enabled, it applies a sigmoid function to normalize the distances.",
+ "type": "comment"
+ },
+ "586": {
+ "file_id": 55,
+ "content": " # print(nn_features_n)\n ###\n if global_map_tmp_dic is not None: ###when testing, use global map memory\n if seq_names[n] not in global_map_tmp_dic:\n global_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).tile([1000, 1, 1, 1, 1])\n nn_features_n = paddle.where(\n nn_features_n <= global_map_tmp_dic[seq_names[n]][\n frame_num[n]].unsqueeze(0), nn_features_n,\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(\n 0))\n # print('detach 1')\n # print(nn_features_n.shape)\n # nn_features_n = nn_features_n.detach()\n global_map_tmp_dic[seq_names[n]][\n frame_num[n]] = nn_features_n.detach()[0]\n #########################Local dist map\n seq_prev_frame_embedding = seq_prev_frame_embedding.transpose(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:667-687"
+ },
+ "587": {
+ "file_id": 55,
+ "content": "This code section checks if a sequence name exists in the global map dictionary, and if not, creates an entry for it. It then compares the current frame's features to the corresponding value in the global map for that sequence. If the current frame's features are less than or equal to the stored value, they remain unchanged; otherwise, they get updated with the stored value. Finally, it updates the global map entry with the new frame's features.",
+ "type": "comment"
+ },
+ "588": {
+ "file_id": 55,
+ "content": " [1, 2, 0])\n seq_previous_frame_label = scale_previous_frame_label[n].transpose(\n [1, 2, 0])\n if use_local_map:\n prev_frame_nn_features_n = local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding=seq_prev_frame_embedding,\n query_embedding=seq_current_frame_embedding,\n prev_frame_labels=seq_previous_frame_label,\n gt_ids=ref_obj_ids,\n max_distance=cfg['model_max_local_distance'])\n else:\n prev_frame_nn_features_n, _ = nearest_neighbor_features_per_object(\n reference_embeddings=seq_prev_frame_embedding,\n query_embeddings=seq_current_frame_embedding,\n reference_labels=seq_previous_frame_label,\n k_nearest_neighbors=k_nearest_neighbors,\n gt_ids=gt_ids[n],\n n_chunks=20)\n prev_frame_nn_features_n = (",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:688-707"
+ },
+ "589": {
+ "file_id": 55,
+ "content": "This code is finding the nearest neighbor features for the previous frame's embedding, based on whether local mapping is used or not. If local mapping is used, it calls a separate function `local_previous_frame_nearest_neighbor_features_per_object` to get the features and labels. Otherwise, it uses the `nearest_neighbor_features_per_object` function with specified parameters to find the nearest neighbors. The resulting features are stored in `prev_frame_nn_features_n`.",
+ "type": "comment"
+ },
+ "590": {
+ "file_id": 55,
+ "content": " paddle.nn.functional.sigmoid(prev_frame_nn_features_n) -\n 0.5) * 2\n # print(prev_frame_nn_features_n.mean().item(), prev_frame_nn_features_n.shape, interaction_num) # o\n #############\n if local_map_dics is not None: ##When testing, use local map memory\n local_map_tmp_dic, local_map_dist_dic = local_map_dics\n if seq_names[n] not in local_map_dist_dic:\n print(seq_names[n], 'not in local_map_dist_dic')\n local_map_dist_dic[seq_names[n]] = paddle.zeros(1000, 9)\n if seq_names[n] not in local_map_tmp_dic:\n print(seq_names[n], 'not in local_map_tmp_dic')\n local_map_tmp_dic[seq_names[n]] = paddle.zeros_like(\n prev_frame_nn_features_n).unsqueeze(0).tile(\n [1000, 9, 1, 1, 1, 1])\n # print(local_map_dist_dic[seq_names[n]].shape)\n # print('detach 2')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:708-724"
+ },
+ "591": {
+ "file_id": 55,
+ "content": "This code is checking if the local map dictionaries are not None, indicating testing with local map memory. If a specific sequence name isn't in the local map distance dictionary or temporary map dictionary, it prints an error message and creates a new zero tensor to store the data.",
+ "type": "comment"
+ },
+ "592": {
+ "file_id": 55,
+ "content": " # prev_frame_nn_features_n = prev_frame_nn_features_n.detach()\n local_map_dist_dic[seq_names[n]][\n frame_num[n], interaction_num -\n 1] = 1.0 / (abs(frame_num[n] - start_annotated_frame)\n ) # bugs fixed.\n local_map_tmp_dic[seq_names[n]][\n frame_num[n],\n interaction_num - 1] = prev_frame_nn_features_n.squeeze(\n 0).detach() # bugs fixed.\n if interaction_num == 1:\n prev_frame_nn_features_n = local_map_tmp_dic[seq_names[n]][\n frame_num[n]][interaction_num - 1]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n else:\n if local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num - 1] > \\\n local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num - 2]:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:725-741"
+ },
+ "593": {
+ "file_id": 55,
+ "content": "This code segment appears to be part of a larger function that processes video frames and interactions. It stores the distance from the current frame to the first annotated frame in the local_map_dist_dic dictionary, as well as the corresponding previous frame features in the local_map_tmp_dic. The code also updates the value of prev_frame_nn_features_n based on certain conditions involving interaction numbers and distances between frames.",
+ "type": "comment"
+ },
+ "594": {
+ "file_id": 55,
+ "content": " prev_frame_nn_features_n = local_map_tmp_dic[\n seq_names[n]][frame_num[n]][interaction_num - 1]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n else:\n prev_frame_nn_features_n = local_map_tmp_dic[\n seq_names[n]][frame_num[n]][interaction_num - 2]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n local_map_dics = (local_map_tmp_dic, local_map_dist_dic)\n to_cat_previous_frame = (\n float_(seq_previous_frame_label) == float_(ref_obj_ids)\n ) # float comparision?\n to_cat_current_frame_embedding = current_frame_embedding[\n n].unsqueeze(0).tile((ref_obj_ids.shape[0], 1, 1, 1))\n to_cat_nn_feature_n = nn_features_n.squeeze(0).transpose(\n [2, 3, 0, 1])\n to_cat_previous_frame = float_(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:742-763"
+ },
+ "595": {
+ "file_id": 55,
+ "content": "This code appears to be part of a video modeling process. It seems to involve local map dictionaries and interaction numbers, comparing previous frames with current ones for float comparisons, unsqueezing and reshaping features and labels, and potentially using these operations in some video modeling or analysis task.",
+ "type": "comment"
+ },
+ "596": {
+ "file_id": 55,
+ "content": " to_cat_previous_frame.unsqueeze(-1).transpose([2, 3, 0, 1]))\n to_cat_prev_frame_nn_feature_n = prev_frame_nn_features_n.squeeze(\n 0).transpose([2, 3, 0, 1])\n to_cat = paddle.concat(\n (to_cat_current_frame_embedding, to_cat_nn_feature_n,\n to_cat_prev_frame_nn_feature_n, to_cat_previous_frame), 1)\n pred_ = dynamic_seghead(to_cat)\n pred_ = pred_.transpose([1, 0, 2, 3])\n dic_tmp[seq_names[n]] = pred_\n if global_map_tmp_dic is None:\n return dic_tmp\n else:\n if local_map_dics is None:\n return dic_tmp, global_map_tmp_dic\n else:\n return dic_tmp, global_map_tmp_dic, local_map_dics\n def int_seghead(self,\n ref_frame_embedding=None,\n ref_scribble_label=None,\n prev_round_label=None,\n normalize_nearest_neighbor_distances=True,\n global_map_tmp_dic=None,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:764-787"
+ },
+ "597": {
+ "file_id": 55,
+ "content": "This code is defining a function \"int_seghead\" that takes in various inputs and returns output dictionaries. It concatenates embeddings and features, passes them to the dynamic_seghead function, transposes the result, and stores it in a dictionary. If global_map_tmp_dic is not None, the function also returns other dictionaries.",
+ "type": "comment"
+ },
+ "598": {
+ "file_id": 55,
+ "content": " local_map_dics=None,\n interaction_num=None,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,\n frame_num=None,\n first_inter=True):\n dic_tmp = {}\n bs, c, h, w = ref_frame_embedding.shape\n scale_ref_scribble_label = paddle.nn.functional.interpolate(\n float_(ref_scribble_label), size=(h, w), mode='nearest')\n scale_ref_scribble_label = int_(scale_ref_scribble_label)\n if not first_inter:\n scale_prev_round_label = paddle.nn.functional.interpolate(\n float_(prev_round_label), size=(h, w), mode='nearest')\n scale_prev_round_label = int_(scale_prev_round_label)\n n_chunks = 500\n for n in range(bs):\n gt_id = paddle.arange(0, gt_ids[n] + 1)\n gt_id = int_(gt_id)\n seq_ref_frame_embedding = ref_frame_embedding[n]\n ########################Local dist map",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:788-813"
+ },
+ "599": {
+ "file_id": 55,
+ "content": "This code snippet calculates the local distance map for each frame in the batch and possibly a previous round if it's not the first interaction. The function takes in various parameters such as ref_frame_embedding, prev_round_label, gt_ids, etc., and performs interpolation to resize the reference scribble label and previous round label. It then iterates over each frame in the batch, creating a gt_id array, and calculating the local distance map for the current frame's embedding. This process may involve interpolation and integer conversion of the resized labels.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/50.json b/docs/data/50.json
new file mode 100644
index 000000000..ca44518a0
--- /dev/null
+++ b/docs/data/50.json
@@ -0,0 +1,546 @@
+{
+ "5000": {
+ "file_id": 430,
+ "content": " \"\"\"get the size of the dataset.\"\"\"\n if self.num_samples_precise_bn is None:\n return len(self.info)\n else:\n random.shuffle(self.info)\n return min(self.num_samples_precise_bn, len(self.info))",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/slowfast_video.py:138-143"
+ },
+ "5001": {
+ "file_id": 430,
+ "content": "This code calculates the size of the dataset. If num_samples_precise_bn is None, it returns the length of self.info. Otherwise, shuffles self.info and returns the minimum value between num_samples_precise_bn and the length of self.info.",
+ "type": "comment"
+ },
+ "5002": {
+ "file_id": 431,
+ "content": "/paddlevideo/loader/dataset/ucf101_skeleton.py",
+ "type": "filepath"
+ },
+ "5003": {
+ "file_id": 431,
+ "content": "This code defines a Python class for the UCF101 Skeleton Dataset in PaddleVideo, loading skeleton features and normalizing data for action recognition tasks. The dataset includes train and test methods for preparing frames with `prepare_train` and `prepare_test` functions.",
+ "type": "summary"
+ },
+ "5004": {
+ "file_id": 431,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport pickle\nimport paddle\nfrom paddle.io import Dataset\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass UCF101SkeletonDataset(BaseDataset):\n \"\"\"\n Skeleton dataset for action recognition.\n The dataset loads skeleton feature, and apply norm operatations.",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ucf101_skeleton.py:1-35"
+ },
+ "5005": {
+ "file_id": 431,
+ "content": "This code snippet is a Python class for UCF101 Skeleton Dataset in PaddleVideo. It loads skeleton features and applies normalization operations, registering the dataset for action recognition tasks.",
+ "type": "comment"
+ },
+ "5006": {
+ "file_id": 431,
+ "content": " Args:\n file_path (str): Path to the index file.\n pipeline(obj): Define the pipeline of data preprocessing.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n \"\"\"\n def __init__(self,\n file_path,\n pipeline,\n split,\n repeat_times,\n test_mode=False):\n self.split = split\n self.repeat_times = repeat_times\n super().__init__(file_path, pipeline, test_mode=test_mode)\n self._ori_len = len(self.info)\n self.start_index = 0\n self.modality = \"Pose\"\n def load_file(self):\n \"\"\"Load annotation file to get video information.\"\"\"\n assert self.file_path.endswith('.pkl')\n return self.load_pkl_annotations()\n def load_pkl_annotations(self):\n with open(self.file_path, \"rb\") as f:\n data = pickle.load(f)\n if self.split:\n split, data = data['split'], data['annotations']\n identifier = 'filename' if 'filename' in data[0] else 'frame_dir'",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ucf101_skeleton.py:36-66"
+ },
+ "5007": {
+ "file_id": 431,
+ "content": "This code defines a class that loads annotation data from a file, specifically for the UCF101 dataset's skeleton information. It takes arguments such as the file path, pipeline object, and whether it's building a test dataset. The load_file method checks if the file is a .pkl file and calls load_pkl_annotations to get video information. If the split argument is provided, it only uses the specified part of the data.",
+ "type": "comment"
+ },
+ "5008": {
+ "file_id": 431,
+ "content": " data = [x for x in data if x[identifier] in split[self.split]]\n return data\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training given the index.\"\"\"\n results = copy.deepcopy(self.info[idx % self._ori_len])\n results['modality'] = self.modality\n results['start_index'] = self.start_index\n return self.pipeline(results)\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for testing given the index.\"\"\"\n results = copy.deepcopy(self.info[idx % self._ori_len])\n results['modality'] = self.modality\n results['start_index'] = self.start_index\n return self.pipeline(results)\n def __len__(self):\n \"\"\"get the size of the dataset.\"\"\"\n return len(self.info) * self.repeat_times",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ucf101_skeleton.py:67-89"
+ },
+ "5009": {
+ "file_id": 431,
+ "content": "This code defines a dataset for PaddleVideo, containing train and test methods for preparing frames. The `prepare_train` and `prepare_test` functions create new results by copying the original information, setting modality and start index based on the given index. The `__len__` function returns the size of the dataset by multiplying the number of info items with repeat times.",
+ "type": "comment"
+ },
+ "5010": {
+ "file_id": 432,
+ "content": "/paddlevideo/loader/dataset/ucf24_dataset.py",
+ "type": "filepath"
+ },
+ "5011": {
+ "file_id": 432,
+ "content": "The Python code defines a \"Ucf24Dataset\" class for loading and transforming UCF24 dataset in PaddleVideo, with methods to prepare data for training/validation and testing. It extracts relevant information like image paths, labels, and frame indices, and converts image path names from 'jpg' to 'txt'.",
+ "type": "summary"
+ },
+ "5012": {
+ "file_id": 432,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass UCF24Dataset(BaseDataset):\n \"\"\"Dataset for YOWO\n The dataset loads raw videos and apply specified transforms on them.\n The index file is a file with multiple lines, and each line indicates\n a sample video with the filepath and label, which are split with a whitesapce.",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ucf24_dataset.py:1-30"
+ },
+ "5013": {
+ "file_id": 432,
+ "content": "This code is a Python class for the UCF24 dataset used in PaddleVideo, which loads raw videos and applies specified transformations on them. It is registered within the registry module and utilizes other modules such as BaseDataset and gets logger from utils. The license information and import statements are also included.",
+ "type": "comment"
+ },
+ "5014": {
+ "file_id": 432,
+ "content": " Example of a inde file:\n .. code-block:: txt\n Args:\n file_path(str): Path to the index file.\n pipeline(XXX): A sequence of data transforms.\n **kwargs: Keyword arguments for ```BaseDataset```.\n \"\"\"\n def __init__(self, file_path, pipeline, num_retries=5, **kwargs):\n self.num_retries = num_retries\n super().__init__(file_path, pipeline, **kwargs)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n with open(self.file_path, 'r') as fin:\n lines = fin.readlines()\n for line in lines:\n line = line.strip() # 'data/ucf24/labels/class_name/video_name/key_frame.txt'\n filename = line.replace('txt', 'jpg').replace(\n 'labels', 'rgb-images') # key frame path\n info.append(dict(filename=filename))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n results = copy.deepcopy(self.info[idx])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ucf24_dataset.py:31-59"
+ },
+ "5015": {
+ "file_id": 432,
+ "content": "This code defines a dataset class, \"Ucf24Dataset\", which loads video information from an index file and prepares data for training or validation. It takes a file path, pipeline, and additional keyword arguments. The load_file method reads the index file to extract video information, such as filenames, while the prepare_train method prepares data for training/validation given an index.",
+ "type": "comment"
+ },
+ "5016": {
+ "file_id": 432,
+ "content": " results = self.pipeline(results)\n im_path = results['filename']\n im_path = im_path.replace('jpg', 'txt')\n im_split = im_path.split('/')\n frame_index = im_split[3] + '_' + im_split[4] + '_' + im_split[5]\n return results['imgs'], np.array([results['labels']]), frame_index\n def prepare_test(self, idx):\n \"\"\"TEST. Prepare the data for test given the index.\"\"\"\n # Try to catch Exception caused by reading corrupted video file\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n im_path = results['filename']\n im_path = im_path.replace('jpg', 'txt')\n im_split = im_path.split('/')\n frame_index = im_split[3] + '_' + im_split[4] + '_' + im_split[5]\n return results['imgs'], np.array([results['labels']]), frame_index",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/ucf24_dataset.py:60-76"
+ },
+ "5017": {
+ "file_id": 432,
+ "content": "Code from \"PaddleVideo/paddlevideo/loader/dataset/ucf24_dataset.py\" prepares data for testing by copying the info at index idx, applying a pipeline function to it and extracting relevant information like image paths and labels. The code also converts image path names from 'jpg' to 'txt'. Finally, it returns images, labels, and frame indices.",
+ "type": "comment"
+ },
+ "5018": {
+ "file_id": 433,
+ "content": "/paddlevideo/loader/dataset/video.py",
+ "type": "filepath"
+ },
+ "5019": {
+ "file_id": 433,
+ "content": "VideoDataset is a subclass of BaseDataset that loads and processes raw videos, using an index file containing video information. It handles corrupted files with retries and error logging. The `prepare_train` and `prepare_test` methods return image data and labels for training and testing respectively.",
+ "type": "summary"
+ },
+ "5020": {
+ "file_id": 433,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass VideoDataset(BaseDataset):\n \"\"\"Video dataset for action recognition\n The dataset loads raw videos and apply specified transforms on them.\n The index file is a file with multiple lines, and each line indicates",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/video.py:1-31"
+ },
+ "5021": {
+ "file_id": 433,
+ "content": "This code is for VideoDataset class, a subclass of BaseDataset, that loads raw videos and applies specified transforms. It uses index file with multiple lines where each line indicates information about videos in the dataset. The class is registered within the DATASETS registry and logger is initialized.",
+ "type": "comment"
+ },
+ "5022": {
+ "file_id": 433,
+ "content": " a sample video with the filepath and label, which are split with a whitesapce.\n Example of a inde file:\n .. code-block:: txt\n path/000.mp4 1\n path/001.mp4 1\n path/002.mp4 2\n path/003.mp4 2\n Args:\n file_path(str): Path to the index file.\n pipeline(XXX): A sequence of data transforms.\n **kwargs: Keyword arguments for ```BaseDataset```.\n \"\"\"\n def __init__(self, file_path, pipeline, num_retries=5, suffix='', **kwargs):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, **kwargs)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split()\n filename, labels = line_split\n #TODO(hj): Required suffix format: may mp4/avi/wmv\n filename = filename + self.suffix",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/video.py:32-57"
+ },
+ "5023": {
+ "file_id": 433,
+ "content": "This code initializes a new class for loading index file data, which contains video information. The index file has path and label entries separated by whitespace. The load_file method reads the index file to retrieve filename and labels for each video.",
+ "type": "comment"
+ },
+ "5024": {
+ "file_id": 433,
+ "content": " if self.data_prefix is not None:\n filename = osp.join(self.data_prefix, filename)\n info.append(dict(filename=filename, labels=int(labels)))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/video.py:58-80"
+ },
+ "5025": {
+ "file_id": 433,
+ "content": "This code defines a class with methods to prepare data for training and testing. It handles potential corrupted video files by retrying if an exception occurs, and logs the error message along with the number of retries. The `prepare_train` method returns image data (`imgs`) and corresponding labels from the given index in the dataset. Similarly, the `prepare_test` method returns image data and labels for testing.",
+ "type": "comment"
+ },
+ "5026": {
+ "file_id": 433,
+ "content": " \"\"\"TEST. Prepare the data for test given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])",
+ "type": "code",
+ "location": "/paddlevideo/loader/dataset/video.py:81-95"
+ },
+ "5027": {
+ "file_id": 433,
+ "content": "This code attempts to load a video file and prepare the data for testing. It handles potential exceptions caused by corrupted files by retrying multiple times. If an exception occurs, it logs an error message and retries with another randomly selected file index. The function returns the images and labels from the successfully loaded video file.",
+ "type": "comment"
+ },
+ "5028": {
+ "file_id": 434,
+ "content": "/paddlevideo/loader/pipelines/__init__.py",
+ "type": "filepath"
+ },
+ "5029": {
+ "file_id": 434,
+ "content": "This file imports necessary modules for PaddleVideo data preprocessing and model training/testing, including annotations, video labels, augmentation, decoding, mixing, segmentation, and sampling. It also defines a list of pipeline components for custom video processing pipelines.",
+ "type": "summary"
+ },
+ "5030": {
+ "file_id": 434,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .anet_pipeline import GetMatchMap, GetVideoLabel, LoadFeat\nfrom .augmentations import (CenterCrop, ColorJitter, GroupRandomFlip,\n GroupResize, Image2Array, JitterScale, MultiCrop,\n Normalization, PackOutput, RandomCrop, RandomFlip,\n RandomResizedCrop, Scale, TenCrop, ToArray,\n UniformCrop, RandomGamma, MultiCenterCrop,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/__init__.py:1-20"
+ },
+ "5031": {
+ "file_id": 434,
+ "content": "This file contains the initialization and imports from different pipeline classes in PaddleVideo. It includes functions for loading annotations, getting video labels, and various augmentation techniques. The license and copyright information are also present.",
+ "type": "comment"
+ },
+ "5032": {
+ "file_id": 434,
+ "content": " RandomBrightness, RandomHue, RandomSaturation, YowoAug)\nfrom .augmentations_ava import *\nfrom .compose import Compose\nfrom .decode import FeatureDecoder, FrameDecoder, VideoDecoder, ActionFeatureDecoder\nfrom .decode_image import ImageDecoder\nfrom .decode_sampler import DecodeSampler\nfrom .mix import Cutmix, Mixup, VideoMix\nfrom .multimodal import FeaturePadding, RandomCap, RandomMask, Tokenize\nfrom .sample import Sampler, SamplerPkl\nfrom .sample_ava import *\nfrom .segmentation import MultiNorm, MultiRestrictSize\nfrom .skeleton_pipeline import AutoPadding, Iden, SkeletonNorm\nfrom .skeleton_pipeline import SketeonCropSample, SketeonModalityTransform, RandomRotation\nfrom .skeleton_pipeline import (UniformSampleFrames, PoseDecode, PoseCompact,\n RandomResizedCrop_V2, Flip_V2, CenterCrop_V2,\n GeneratePoseTarget, FormatShape, Collect)\nfrom .decode_sampler_MRI import SFMRI_DecodeSampler\nfrom .segmentation_pipline import SegmentationSampler",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/__init__.py:21-38"
+ },
+ "5033": {
+ "file_id": 434,
+ "content": "The code imports various classes and functions for different image, video, and skeleton-related pipelines. It includes modules for augmentations, decoding, mixing, segmentation, sampling, and more, used in the PaddleVideo library. These pipelines are used to preprocess, decode, and sample data for training and testing models.",
+ "type": "comment"
+ },
+ "5034": {
+ "file_id": 434,
+ "content": "from .sample_ucf24 import SamplerUCF24\n__all__ = [\n 'ImageDecoder', 'RandomMask', 'UniformCrop', 'SkeletonNorm', 'Tokenize',\n 'Sampler', 'FeatureDecoder', 'DecodeSampler', 'TenCrop', 'Compose',\n 'AutoPadding', 'Normalization', 'Mixup', 'Image2Array', 'Scale',\n 'GroupResize', 'VideoDecoder', 'FrameDecoder', 'PackOutput',\n 'ActionFeatureDecoder', 'GetVideoLabel', 'Cutmix', 'CenterCrop',\n 'RandomCrop', 'LoadFeat', 'RandomCap', 'JitterScale', 'Iden', 'VideoMix',\n 'ColorJitter', 'RandomFlip', 'ToArray', 'FeaturePadding', 'GetMatchMap',\n 'GroupRandomFlip', 'MultiCrop', 'SFMRI_DecodeSampler', 'MultiRestrictSize',\n 'MultiNorm', 'RandomResizedCrop', 'SamplerPkl', 'SegmentationSampler',\n 'SketeonCropSample', 'SketeonModalityTransform', 'RandomRotation',\n 'RandomGamma', 'MultiCenterCrop', 'RandomBrightness', 'RandomHue',\n 'RandomSaturation', 'UniformSampleFrames', 'PoseDecode', 'PoseCompact',\n 'Resize', 'RandomResizedCrop_V2', 'Flip_V2', 'GeneratePoseTarget',\n 'FormatShape', 'Collect', 'RandomSaturation', 'SamplerUCF24', 'YowoAug'",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/__init__.py:39-55"
+ },
+ "5035": {
+ "file_id": 434,
+ "content": "This code is importing the \"SamplerUCF24\" class from the \"sample_ucf24\" module and defining a list of available pipeline components for PaddleVideo, including image and feature decoders, transforms, samplers, and more. These components can be used to build custom video processing pipelines for various tasks.",
+ "type": "comment"
+ },
+ "5036": {
+ "file_id": 434,
+ "content": "]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/__init__.py:56-56"
+ },
+ "5037": {
+ "file_id": 434,
+ "content": "The code appears to be incomplete or empty, as there are no visible operations or assignments. It could potentially be a placeholder or an intentionally empty function/class definition.",
+ "type": "comment"
+ },
+ "5038": {
+ "file_id": 435,
+ "content": "/paddlevideo/loader/pipelines/anet_pipeline.py",
+ "type": "filepath"
+ },
+ "5039": {
+ "file_id": 435,
+ "content": "PaddleVideo library enables feature extraction and map creation, while GetVideoLabel class calculates IoU for object detection tasks. The code stores max IoU values and prepares data for evaluation or processing.",
+ "type": "summary"
+ },
+ "5040": {
+ "file_id": 435,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport numpy as np\nfrom ..registry import PIPELINES\n\"\"\"pipeline ops for Activity Net.\n\"\"\"\n@PIPELINES.register()\nclass LoadFeat(object):\n def __init__(self, feat_path):\n self.feat_path = feat_path\n def __call__(self, results):\n video_name = results['video_name']\n file_name = video_name + \".npy\"\n file_path = os.path.join(self.feat_path, file_name)\n #TODO: check path\n video_feat = np.load(file_path)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/anet_pipeline.py:1-32"
+ },
+ "5041": {
+ "file_id": 435,
+ "content": "This code is part of PaddleVideo library, specifically for loading feature data from a given path. It defines a class \"LoadFeat\" and uses the numpy library to load .npy files based on the video name provided in the results dictionary. The file path is constructed using the specified feat_path and video name.",
+ "type": "comment"
+ },
+ "5042": {
+ "file_id": 435,
+ "content": " video_feat = video_feat.T\n video_feat = video_feat.astype(\"float32\")\n results['video_feat'] = video_feat\n return results\n@PIPELINES.register()\nclass GetMatchMap(object):\n def __init__(self, tscale):\n self.tscale = tscale\n self.tgap = 1. / self.tscale\n def __call__(self, results):\n match_map = []\n for idx in range(self.tscale):\n tmp_match_window = []\n xmin = self.tgap * idx\n for jdx in range(1, self.tscale + 1):\n xmax = xmin + self.tgap * jdx\n tmp_match_window.append([xmin, xmax])\n match_map.append(tmp_match_window)\n match_map = np.array(match_map)\n match_map = np.transpose(match_map, [1, 0, 2])\n match_map = np.reshape(match_map, [-1, 2])\n anchor_xmin = [self.tgap * i for i in range(self.tscale)]\n anchor_xmax = [self.tgap * i for i in range(1, self.tscale + 1)]\n results['match_map'] = match_map\n results['anchor_xmin'] = anchor_xmin",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/anet_pipeline.py:33-62"
+ },
+ "5043": {
+ "file_id": 435,
+ "content": "This code defines a pipeline function that generates matching maps for an input video. It creates temporal matching windows of varying sizes and reshapes the result into a specific format. The anchor positions are also extracted for later use.",
+ "type": "comment"
+ },
+ "5044": {
+ "file_id": 435,
+ "content": " results['anchor_xmax'] = anchor_xmax\n return results\n@PIPELINES.register()\nclass GetVideoLabel(object):\n def __init__(self, tscale, dscale, datatype=\"float32\"):\n self.tscale = tscale\n self.dscale = dscale\n self.tgap = 1. / self.tscale\n self.datatype = datatype\n def iou_with_anchors(self, anchors_min, anchors_max, box_min, box_max):\n \"\"\"Compute jaccard score between a box and the anchors.\n \"\"\"\n len_anchors = anchors_max - anchors_min\n int_xmin = np.maximum(anchors_min, box_min)\n int_xmax = np.minimum(anchors_max, box_max)\n inter_len = np.maximum(int_xmax - int_xmin, 0.)\n union_len = len_anchors - inter_len + box_max - box_min\n jaccard = np.divide(inter_len, union_len)\n return jaccard\n def ioa_with_anchors(self, anchors_min, anchors_max, box_min, box_max):\n \"\"\"Compute intersection between score a box and the anchors.\n \"\"\"\n len_anchors = anchors_max - anchors_min\n int_xmin = np.maximum(anchors_min, box_min)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/anet_pipeline.py:63-90"
+ },
+ "5045": {
+ "file_id": 435,
+ "content": "This code defines a class called \"GetVideoLabel\" which calculates the Intersection over Union (IOU) and intersection scores between a box and the anchors. It also initializes variables for time and distance scaling, and box type data types. The \"iou_with_anchors\" method calculates the Jaccard score and the \"ioa_with_anchors\" method computes the intersection. These methods can be used to determine the best match between an anchor box and a target box in object detection tasks.",
+ "type": "comment"
+ },
+ "5046": {
+ "file_id": 435,
+ "content": " int_xmax = np.minimum(anchors_max, box_max)\n inter_len = np.maximum(int_xmax - int_xmin, 0.)\n scores = np.divide(inter_len, len_anchors)\n return scores\n def __call__(self, results):\n video_info = results['video_info']\n match_map = results['match_map']\n anchor_xmin = results['anchor_xmin']\n anchor_xmax = results['anchor_xmax']\n video_second = video_info['duration_second']\n video_labels = video_info['annotations']\n gt_bbox = []\n gt_iou_map = []\n for gt in video_labels:\n tmp_start = max(min(1, gt[\"segment\"][0] / video_second), 0)\n tmp_end = max(min(1, gt[\"segment\"][1] / video_second), 0)\n gt_bbox.append([tmp_start, tmp_end])\n tmp_gt_iou_map = self.iou_with_anchors(match_map[:, 0],\n match_map[:, 1], tmp_start,\n tmp_end)\n tmp_gt_iou_map = np.reshape(tmp_gt_iou_map,\n [self.dscale, self.tscale])",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/anet_pipeline.py:91-115"
+ },
+ "5047": {
+ "file_id": 435,
+ "content": "The function initializes gt_bbox and gt_iou_map variables to store ground truth bounding box coordinates and their IoU with anchor boxes. It then iterates through video labels, calculating the start and end timestamps in video seconds for each ground truth box. The IoU between match map and the current ground truth is computed using the iou_with_anchors function and stored in gt_iou_map, reshaped to match the dimensions of dscale and tscale.",
+ "type": "comment"
+ },
+ "5048": {
+ "file_id": 435,
+ "content": " gt_iou_map.append(tmp_gt_iou_map)\n gt_iou_map = np.array(gt_iou_map)\n gt_iou_map = np.max(gt_iou_map, axis=0)\n gt_bbox = np.array(gt_bbox)\n gt_xmins = gt_bbox[:, 0]\n gt_xmaxs = gt_bbox[:, 1]\n gt_len_small = 3 * self.tgap\n gt_start_bboxs = np.stack(\n (gt_xmins - gt_len_small / 2, gt_xmins + gt_len_small / 2), axis=1)\n gt_end_bboxs = np.stack(\n (gt_xmaxs - gt_len_small / 2, gt_xmaxs + gt_len_small / 2), axis=1)\n match_score_start = []\n for jdx in range(len(anchor_xmin)):\n match_score_start.append(\n np.max(\n self.ioa_with_anchors(anchor_xmin[jdx], anchor_xmax[jdx],\n gt_start_bboxs[:, 0],\n gt_start_bboxs[:, 1])))\n match_score_end = []\n for jdx in range(len(anchor_xmin)):\n match_score_end.append(\n np.max(\n self.ioa_with_anchors(anchor_xmin[jdx], anchor_xmax[jdx],",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/anet_pipeline.py:116-140"
+ },
+ "5049": {
+ "file_id": 435,
+ "content": "This code calculates the intersection over union (IoU) between ground truth bounding boxes and anchor boxes. It stores the maximum IoU values for each ground truth box and anchor pair, then calculates the maximum IoU values for start and end positions of anchor boxes. This information will be used to determine if a prediction matches with a ground truth box and assign appropriate scores.",
+ "type": "comment"
+ },
+ "5050": {
+ "file_id": 435,
+ "content": " gt_end_bboxs[:, 0], gt_end_bboxs[:,\n 1])))\n gt_start = np.array(match_score_start)\n gt_end = np.array(match_score_end)\n results['gt_iou_map'] = gt_iou_map.astype(self.datatype)\n results['gt_start'] = gt_start.astype(self.datatype)\n results['gt_end'] = gt_end.astype(self.datatype)\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/anet_pipeline.py:141-150"
+ },
+ "5051": {
+ "file_id": 435,
+ "content": "This code is storing ground truth (gt) IOU map, start and end indices for the annotations into the 'results' dictionary. The IOU map is converted to specified datatype before storage. These values will be used later for evaluation or further processing.",
+ "type": "comment"
+ },
+ "5052": {
+ "file_id": 436,
+ "content": "/paddlevideo/loader/pipelines/augmentations.py",
+ "type": "filepath"
+ },
+ "5053": {
+ "file_id": 436,
+ "content": "This code enhances PaddleVideo's loader with resize operation and augmentation pipeline, enabling diverse data preprocessing. It calculates crop offsets and performs object detection image augmentation using uniform sampling, resizing, and flipping techniques, resizes images, scales by 255.0, concatenates frames, transposes array dimensions, stores results in 'results', and returns arrays.",
+ "type": "summary"
+ },
+ "5054": {
+ "file_id": 436,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport random\nfrom collections.abc import Sequence\nimport cv2\nimport numpy as np\nimport paddle\nimport paddle.nn.functional as F\nfrom PIL import Image\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass Scale(object):\n \"\"\"\n Scale images.\n Args:\n short_size(float | int): Short size of an image will be scaled to the short_size.\n fixed_ratio(bool): Set whether to zoom according to a fixed ratio. default: True",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1-34"
+ },
+ "5055": {
+ "file_id": 436,
+ "content": "This code is from the PaddleVideo library, specifically the loader module's augmentations pipeline. It defines a Scale class that scales images based on their short side to the given short_size parameter. The fixed_ratio parameter determines whether or not the image should be resized while maintaining its aspect ratio. This class is then registered in PIPELINES for later use.",
+ "type": "comment"
+ },
+ "5056": {
+ "file_id": 436,
+ "content": " do_round(bool): Whether to round up when calculating the zoom ratio. default: False\n backend(str): Choose pillow or cv2 as the graphics processing backend. default: 'pillow'\n \"\"\"\n def __init__(self,\n short_size,\n fixed_ratio=True,\n keep_ratio=None,\n do_round=False,\n backend='pillow'):\n self.short_size = short_size\n assert (fixed_ratio and not keep_ratio) or (not fixed_ratio), \\\n f\"fixed_ratio and keep_ratio cannot be true at the same time\"\n self.fixed_ratio = fixed_ratio\n self.keep_ratio = keep_ratio\n self.do_round = do_round\n assert backend in [\n 'pillow', 'cv2'\n ], f\"Scale's backend must be pillow or cv2, but get {backend}\"\n self.backend = backend\n def __call__(self, results):\n \"\"\"\n Performs resize operations.\n Args:\n imgs (Sequence[PIL.Image]): List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:35-61"
+ },
+ "5057": {
+ "file_id": 436,
+ "content": "The code defines a class for resize operations. It takes parameters for short size, fixed ratio (defaults to True), keep ratio, do_round (default False), and backend (default 'pillow'). The class checks if fixed_ratio and keep_ratio can't be true at the same time. It also ensures the backend is either 'pillow' or 'cv2'. The __call__ method performs resize operations on images, taking a Sequence of PIL.Image as input.",
+ "type": "comment"
+ },
+ "5058": {
+ "file_id": 436,
+ "content": " return:\n resized_imgs: List where each item is a PIL.Image after scaling.\n \"\"\"\n imgs = results['imgs']\n resized_imgs = []\n for i in range(len(imgs)):\n img = imgs[i]\n if isinstance(img, np.ndarray):\n h, w, _ = img.shape\n elif isinstance(img, Image.Image):\n w, h = img.size\n else:\n raise NotImplementedError\n if (w <= h and w == self.short_size) or (h <= w\n and h == self.short_size):\n if self.backend == 'pillow' and not isinstance(\n img, Image.Image):\n img = Image.fromarray(img)\n resized_imgs.append(img)\n continue\n if w <= h:\n ow = self.short_size\n if self.fixed_ratio:\n oh = int(self.short_size * 4.0 / 3.0)\n elif self.keep_ratio is False:\n oh = self.short_size",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:62-88"
+ },
+ "5059": {
+ "file_id": 436,
+ "content": "This code is responsible for resizing images to a specified size in a PaddleVideo pipeline. It iterates through each image, checks the aspect ratio, and resizes them accordingly before appending to the resized_imgs list. If the image is already the correct size, it is directly added to the list without further processing.",
+ "type": "comment"
+ },
+ "5060": {
+ "file_id": 436,
+ "content": " else:\n scale_factor = self.short_size / w\n oh = int(h * float(scale_factor) +\n 0.5) if self.do_round else int(h *\n self.short_size / w)\n ow = int(w * float(scale_factor) +\n 0.5) if self.do_round else self.short_size\n else:\n oh = self.short_size\n if self.fixed_ratio:\n ow = int(self.short_size * 4.0 / 3.0)\n elif self.keep_ratio is False:\n ow = self.short_size\n else:\n scale_factor = self.short_size / h\n oh = int(h * float(scale_factor) +\n 0.5) if self.do_round else self.short_size\n ow = int(w * float(scale_factor) +\n 0.5) if self.do_round else int(w *\n self.short_size / h)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:89-108"
+ },
+ "5061": {
+ "file_id": 436,
+ "content": "This code calculates the output image size for resizing and maintains aspect ratio if specified. It uses scale_factor to calculate the output height (oh) and width (ow), considering do_round, fixed_ratio, keep_ratio flags and short_size.",
+ "type": "comment"
+ },
+ "5062": {
+ "file_id": 436,
+ "content": " if self.backend == 'pillow':\n resized_imgs.append(img.resize((ow, oh), Image.BILINEAR))\n elif self.backend == 'cv2' and (self.keep_ratio is not None):\n resized_imgs.append(\n cv2.resize(img, (ow, oh), interpolation=cv2.INTER_LINEAR))\n else:\n resized_imgs.append(\n Image.fromarray(\n cv2.resize(np.asarray(img), (ow, oh),\n interpolation=cv2.INTER_LINEAR)))\n results['imgs'] = resized_imgs\n return results\n@PIPELINES.register()\nclass RandomCrop(object):\n \"\"\"\n Random crop images.\n Args:\n target_size(int): Random crop a square with the target_size from an image.\n \"\"\"\n def __init__(self, target_size):\n self.target_size = target_size\n def __call__(self, results):\n \"\"\"\n Performs random crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:109-138"
+ },
+ "5063": {
+ "file_id": 436,
+ "content": "This code defines an augmentation pipeline for image processing. It resizes images using different backends based on the backend specified and whether the ratio should be preserved or not. The results are then returned as a dictionary with 'imgs' key containing the resized images. Additionally, there is a RandomCrop class which performs random crop operations on images of the specified target size.",
+ "type": "comment"
+ },
+ "5064": {
+ "file_id": 436,
+ "content": " return:\n crop_imgs: List where each item is a PIL.Image after random crop.\n \"\"\"\n imgs = results['imgs']\n if 'backend' in results and results['backend'] == 'pyav': # [c,t,h,w]\n h, w = imgs.shape[2:]\n else:\n w, h = imgs[0].size\n th, tw = self.target_size, self.target_size\n assert (w >= self.target_size) and (h >= self.target_size), \\\n \"image width({}) and height({}) should be larger than crop size\".format(\n w, h, self.target_size)\n crop_images = []\n if 'backend' in results and results['backend'] == 'pyav':\n x1 = np.random.randint(0, w - tw)\n y1 = np.random.randint(0, h - th)\n crop_images = imgs[:, :, y1:y1 + th, x1:x1 + tw] # [C, T, th, tw]\n else:\n x1 = random.randint(0, w - tw)\n y1 = random.randint(0, h - th)\n for img in imgs:\n if w == tw and h == th:\n crop_images.append(img)\n else:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:139-164"
+ },
+ "5065": {
+ "file_id": 436,
+ "content": "This code is a part of PaddleVideo's augmentations.py, which applies random cropping to images. It checks if the backend used is 'pyav', and if so, extracts the image dimensions. If not, it gets the image size from the first image in the list. Then, it asserts that the image dimensions are larger than the target size. Finally, it generates a random crop position and crops each image in the list using these positions. The cropped images are stored in the 'crop_images' list which is returned at the end.",
+ "type": "comment"
+ },
+ "5066": {
+ "file_id": 436,
+ "content": " crop_images.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n results['imgs'] = crop_images\n return results\n@PIPELINES.register()\nclass RandomResizedCrop(RandomCrop):\n def __init__(self,\n area_range=(0.08, 1.0),\n aspect_ratio_range=(3 / 4, 4 / 3),\n target_size=224,\n backend='cv2'):\n self.area_range = area_range\n self.aspect_ratio_range = aspect_ratio_range\n self.target_size = target_size\n self.backend = backend\n @staticmethod\n def get_crop_bbox(img_shape,\n area_range,\n aspect_ratio_range,\n max_attempts=10):\n assert 0 < area_range[0] <= area_range[1] <= 1\n assert 0 < aspect_ratio_range[0] <= aspect_ratio_range[1]\n img_h, img_w = img_shape\n area = img_h * img_w\n min_ar, max_ar = aspect_ratio_range\n aspect_ratios = np.exp(\n np.random.uniform(np.log(min_ar), np.log(max_ar),",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:165-197"
+ },
+ "5067": {
+ "file_id": 436,
+ "content": "RandomResizedCrop is a pipeline that resizes and crops images randomly with specified area, aspect ratio range, target size, and backend. The method get_crop_bbox takes image shape, area and aspect ratio ranges as input and returns the crop bounding box within the specified range of area and aspect ratio.",
+ "type": "comment"
+ },
+ "5068": {
+ "file_id": 436,
+ "content": " size=max_attempts))\n target_areas = np.random.uniform(*area_range, size=max_attempts) * area\n candidate_crop_w = np.round(np.sqrt(target_areas *\n aspect_ratios)).astype(np.int32)\n candidate_crop_h = np.round(np.sqrt(target_areas /\n aspect_ratios)).astype(np.int32)\n for i in range(max_attempts):\n crop_w = candidate_crop_w[i]\n crop_h = candidate_crop_h[i]\n if crop_h <= img_h and crop_w <= img_w:\n x_offset = random.randint(0, img_w - crop_w)\n y_offset = random.randint(0, img_h - crop_h)\n return x_offset, y_offset, x_offset + crop_w, y_offset + crop_h\n # Fallback\n crop_size = min(img_h, img_w)\n x_offset = (img_w - crop_size) // 2\n y_offset = (img_h - crop_size) // 2\n return x_offset, y_offset, x_offset + crop_size, y_offset + crop_size\n def __call__(self, results):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:198-219"
+ },
+ "5069": {
+ "file_id": 436,
+ "content": "This function generates a random crop size based on the aspect ratios and target areas. It then iterates through candidate crop sizes, selecting one that fits within the image bounds. If no suitable crop is found, it falls back to centering a smaller crop. The function returns the offset coordinates and crop dimensions for the selected crop.",
+ "type": "comment"
+ },
+ "5070": {
+ "file_id": 436,
+ "content": " imgs = results['imgs']\n if self.backend == 'pillow':\n img_w, img_h = imgs[0].size\n elif self.backend == 'cv2':\n img_h, img_w, _ = imgs[0].shape\n elif self.backend == 'pyav':\n img_h, img_w = imgs.shape[2:] # [cthw]\n else:\n raise NotImplementedError\n left, top, right, bottom = self.get_crop_bbox(\n (img_h, img_w), self.area_range, self.aspect_ratio_range)\n if self.backend == 'pillow':\n img_w, img_h = imgs[0].size\n imgs = [img.crop(left, top, right, bottom) for img in imgs]\n elif self.backend == 'cv2':\n img_h, img_w, _ = imgs[0].shape\n imgs = [img[top:bottom, left:right] for img in imgs]\n elif self.backend == 'pyav':\n img_h, img_w = imgs.shape[2:] # [cthw]\n imgs = imgs[:, :, top:bottom, left:right]\n else:\n raise NotImplementedError\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass CenterCrop(object):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:220-249"
+ },
+ "5071": {
+ "file_id": 436,
+ "content": "This code is a part of PaddleVideo library and performs image cropping based on the specified backend. It first retrieves the image dimensions, then applies a crop box to each image according to the defined area range and aspect ratio range. The code handles different backends such as Pillow, OpenCV (cv2), and PyAV. If an unsupported backend is encountered, it raises a NotImplementedError.",
+ "type": "comment"
+ },
+ "5072": {
+ "file_id": 436,
+ "content": " \"\"\"\n Center crop images.\n Args:\n target_size(int): Center crop a square with the target_size from an image.\n do_round(bool): Whether to round up the coordinates of the upper left corner of the cropping area. default: True\n \"\"\"\n def __init__(self, target_size, do_round=True, backend='pillow'):\n self.target_size = target_size\n self.do_round = do_round\n self.backend = backend\n def __call__(self, results):\n \"\"\"\n Performs Center crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n ccrop_imgs: List where each item is a PIL.Image after Center crop.\n \"\"\"\n imgs = results['imgs']\n ccrop_imgs = []\n th, tw = self.target_size, self.target_size\n if isinstance(imgs, paddle.Tensor):\n h, w = imgs.shape[-2:]\n x1 = int(round((w - tw) / 2.0)) if self.do_round else (w - tw) // 2\n y1 = int(round((h - th) / 2.0)) if self.do_round else (h - th) // 2",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:250-276"
+ },
+ "5073": {
+ "file_id": 436,
+ "content": "This code defines a class for center cropping images. The constructor takes the target size, whether to round the coordinates (True by default), and the backend (default is Pillow). The `__call__` method applies the center crop operation on a list of PIL Image objects, returning a new list with the cropped images.",
+ "type": "comment"
+ },
+ "5074": {
+ "file_id": 436,
+ "content": " ccrop_imgs = imgs[:, :, y1:y1 + th, x1:x1 + tw]\n else:\n for img in imgs:\n if self.backend == 'pillow':\n w, h = img.size\n elif self.backend == 'cv2':\n h, w, _ = img.shape\n else:\n raise NotImplementedError\n assert (w >= self.target_size) and (h >= self.target_size), \\\n \"image width({}) and height({}) should be larger than crop size\".format(\n w, h, self.target_size)\n x1 = int(round(\n (w - tw) / 2.0)) if self.do_round else (w - tw) // 2\n y1 = int(round(\n (h - th) / 2.0)) if self.do_round else (h - th) // 2\n if self.backend == 'cv2':\n ccrop_imgs.append(img[y1:y1 + th, x1:x1 + tw])\n elif self.backend == 'pillow':\n ccrop_imgs.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n results['imgs'] = ccrop_imgs",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:277-297"
+ },
+ "5075": {
+ "file_id": 436,
+ "content": "This function performs center crop on images based on the given target size. It first checks if the image dimensions are larger than the crop size, and then calculates the starting coordinates for cropping. If the backend is Pillow, it uses the crop() method to perform the cropping operation; if the backend is OpenCV (cv2), it slices the image array accordingly. The resulting cropped images are stored in 'ccrop_imgs' list and returned in the 'results' dictionary under the key 'imgs'.",
+ "type": "comment"
+ },
+ "5076": {
+ "file_id": 436,
+ "content": " return results\n@PIPELINES.register()\nclass MultiScaleCrop(object):\n \"\"\"\n Random crop images in with multiscale sizes\n Args:\n target_size(int): Random crop a square with the target_size from an image.\n scales(int): List of candidate cropping scales.\n max_distort(int): Maximum allowable deformation combination distance.\n fix_crop(int): Whether to fix the cutting start point.\n allow_duplication(int): Whether to allow duplicate candidate crop starting points.\n more_fix_crop(int): Whether to allow more cutting starting points.\n \"\"\"\n def __init__(\n self,\n target_size, # NOTE: named target size now, but still pass short size in it!\n scales=None,\n max_distort=1,\n fix_crop=True,\n allow_duplication=False,\n more_fix_crop=True,\n backend='pillow'):\n self.target_size = target_size\n self.scales = scales if scales else [1, .875, .75, .66]\n self.max_distort = max_distort",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:298-325"
+ },
+ "5077": {
+ "file_id": 436,
+ "content": "The MultiScaleCrop class is a pipeline module that randomly crops images with multiple scales, targeting a specific size. It allows adjustable parameters like maximum distortion, fix crop start point, and duplicate candidate crop points for flexibility. This module is useful in image processing tasks where random cropping can provide more data augmentation and improve model performance.",
+ "type": "comment"
+ },
+ "5078": {
+ "file_id": 436,
+ "content": " self.fix_crop = fix_crop\n self.allow_duplication = allow_duplication\n self.more_fix_crop = more_fix_crop\n assert backend in [\n 'pillow', 'cv2'\n ], f\"MultiScaleCrop's backend must be pillow or cv2, but get {backend}\"\n self.backend = backend\n def __call__(self, results):\n \"\"\"\n Performs MultiScaleCrop operations.\n Args:\n imgs: List where wach item is a PIL.Image.\n XXX:\n results:\n \"\"\"\n imgs = results['imgs']\n input_size = [self.target_size, self.target_size]\n im_size = imgs[0].size\n # get random crop offset\n def _sample_crop_size(im_size):\n image_w, image_h = im_size[0], im_size[1]\n base_size = min(image_w, image_h)\n crop_sizes = [int(base_size * x) for x in self.scales]\n crop_h = [\n input_size[1] if abs(x - input_size[1]) < 3 else x\n for x in crop_sizes\n ]\n crop_w = [\n input_size[0] if abs(x - input_size[0]) < 3 else x",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:326-360"
+ },
+ "5079": {
+ "file_id": 436,
+ "content": "This code defines a class for multi-scale cropping of images with Pillow or OpenCV backend. The `__init__` method initializes the instance variables and checks if the provided backend is either 'pillow' or 'cv2'. The `__call__` method performs the actual multi-scale cropping operation on a given list of images, applying random crop offsets to each image with the specified target size.",
+ "type": "comment"
+ },
+ "5080": {
+ "file_id": 436,
+ "content": " for x in crop_sizes\n ]\n pairs = []\n for i, h in enumerate(crop_h):\n for j, w in enumerate(crop_w):\n if abs(i - j) <= self.max_distort:\n pairs.append((w, h))\n crop_pair = random.choice(pairs)\n if not self.fix_crop:\n w_offset = random.randint(0, image_w - crop_pair[0])\n h_offset = random.randint(0, image_h - crop_pair[1])\n else:\n w_step = (image_w - crop_pair[0]) / 4\n h_step = (image_h - crop_pair[1]) / 4\n ret = list()\n ret.append((0, 0)) # upper left\n if self.allow_duplication or w_step != 0:\n ret.append((4 * w_step, 0)) # upper right\n if self.allow_duplication or h_step != 0:\n ret.append((0, 4 * h_step)) # lower left\n if self.allow_duplication or (h_step != 0 and w_step != 0):\n ret.append((4 * w_step, 4 * h_step)) # lower right",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:361-384"
+ },
+ "5081": {
+ "file_id": 436,
+ "content": "This code generates a random crop pair from provided sizes, and then applies different cropping locations to the image. If fix_crop is False, it randomly selects an offset for the crop pair within the image boundaries. If fix_crop is True, it calculates four different offsets in a grid pattern using step values based on the image size. The resulting crops are stored in 'ret'.",
+ "type": "comment"
+ },
+ "5082": {
+ "file_id": 436,
+ "content": " if self.allow_duplication or (h_step != 0 or w_step != 0):\n ret.append((2 * w_step, 2 * h_step)) # center\n if self.more_fix_crop:\n ret.append((0, 2 * h_step)) # center left\n ret.append((4 * w_step, 2 * h_step)) # center right\n ret.append((2 * w_step, 4 * h_step)) # lower center\n ret.append((2 * w_step, 0 * h_step)) # upper center\n ret.append((1 * w_step, 1 * h_step)) # upper left quarter\n ret.append((3 * w_step, 1 * h_step)) # upper right quarter\n ret.append((1 * w_step, 3 * h_step)) # lower left quarter\n ret.append((3 * w_step, 3 * h_step)) # lower righ quarter\n w_offset, h_offset = random.choice(ret)\n return crop_pair[0], crop_pair[1], w_offset, h_offset\n crop_w, crop_h, offset_w, offset_h = _sample_crop_size(im_size)\n crop_img_group = [\n img.crop((offset_w, offset_h, offset_w + crop_w, offset_h + crop_h))",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:385-405"
+ },
+ "5083": {
+ "file_id": 436,
+ "content": "This code samples random crop sizes and offsets for image augmentation. It appends different cropping positions based on user allowance or specific flag settings, then randomly selects one of these positions. Finally, it crops the image using the selected position and size.",
+ "type": "comment"
+ },
+ "5084": {
+ "file_id": 436,
+ "content": " for img in imgs\n ]\n if self.backend == 'pillow':\n ret_img_group = [\n img.resize((input_size[0], input_size[1]), Image.BILINEAR)\n for img in crop_img_group\n ]\n else:\n ret_img_group = [\n Image.fromarray(\n cv2.resize(np.asarray(img),\n dsize=(input_size[0], input_size[1]),\n interpolation=cv2.INTER_LINEAR))\n for img in crop_img_group\n ]\n results['imgs'] = ret_img_group\n return results\n@PIPELINES.register()\nclass RandomFlip(object):\n \"\"\"\n Random Flip images.\n Args:\n p(float): Random flip images with the probability p.\n \"\"\"\n def __init__(self, p=0.5):\n self.p = p\n def __call__(self, results):\n \"\"\"\n Performs random flip operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:406-440"
+ },
+ "5085": {
+ "file_id": 436,
+ "content": "This code is a PaddleVideo pipeline for image augmentation, specifically performing random flips with a given probability. It resizes and crops images according to the provided input size. If the backend is set to 'pillow', it uses PIL library's resize function; otherwise, it uses OpenCV's resize function. The results are stored in the 'imgs' key of the 'results' dictionary, which is then returned.",
+ "type": "comment"
+ },
+ "5086": {
+ "file_id": 436,
+ "content": " return:\n flip_imgs: List where each item is a PIL.Image after random flip.\n \"\"\"\n imgs = results['imgs']\n v = random.random()\n if v < self.p:\n if isinstance(imgs, paddle.Tensor):\n results['imgs'] = paddle.flip(imgs, axis=[3])\n elif isinstance(imgs[0], np.ndarray):\n results['imgs'] = [cv2.flip(img, 1, img) for img in imgs\n ] # [[h,w,c], [h,w,c], ..., [h,w,c]]\n else:\n results['imgs'] = [\n img.transpose(Image.FLIP_LEFT_RIGHT) for img in imgs\n ]\n else:\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass RandomBrightness(object):\n \"\"\"\n Random Brightness images.\n Args:\n p(float): Random brightness images with the probability p.\n \"\"\"\n def __init__(self, p=0.1, brightness=1):\n self.p = p\n self.brightness = brightness\n def __call__(self, results):\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:441-473"
+ },
+ "5087": {
+ "file_id": 436,
+ "content": "This code implements a random image flipping and brightness adjustment in PaddleVideo's pipeline. It takes an image as input, randomly decides whether to flip or keep it intact with probability 'p', and adjusts the brightness if applied. The result is then returned.",
+ "type": "comment"
+ },
+ "5088": {
+ "file_id": 436,
+ "content": " Performs random brightness operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n brightness_imgs: List where each item is a PIL.Image after random brightness.\n \"\"\"\n imgs = results['imgs']\n v = random.random()\n if v < self.p:\n transform = ColorJitter(brightness=self.brightness)\n results['imgs'] = [transform(img) for img in imgs]\n else:\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass RandomSaturation(object):\n \"\"\"\n Random Saturation images.\n Args:\n p(float): Random saturation images with the probability p.\n \"\"\"\n def __init__(self, p=0.1, saturation=2):\n self.p = p\n self.saturation = saturation\n def __call__(self, results):\n \"\"\"\n Performs random saturation operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:474-508"
+ },
+ "5089": {
+ "file_id": 436,
+ "content": "The code defines two classes, RandomBrightness and RandomSaturation, which perform random operations on image brightness and saturation respectively. The RandomBrightness class applies ColorJitter with a specified brightness level to each image in the list with a certain probability, while the RandomSaturation class adjusts the saturation of images with another probability. Both classes are registered as Pipelines for data augmentation in PaddleVideo.",
+ "type": "comment"
+ },
+ "5090": {
+ "file_id": 436,
+ "content": " return:\n saturation_imgs: List where each item is a PIL.Image after random saturation.\n \"\"\"\n imgs = results['imgs']\n v = random.random()\n if v < self.p:\n transform = ColorJitter(saturation=self.saturation)\n results['imgs'] = [transform(img) for img in imgs]\n else:\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass RandomHue(object):\n \"\"\"\n Random Hue images.\n Args:\n p(float): Random hue images with the probability p.\n \"\"\"\n def __init__(self, p=0.1, hue=0.5):\n self.p = p\n self.hue = hue\n def __call__(self, results):\n \"\"\"\n Performs random hue operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n hue_imgs: List where each item is a PIL.Image after random hue.\n \"\"\"\n imgs = results['imgs']\n v = random.random()\n if v < self.p:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:509-546"
+ },
+ "5091": {
+ "file_id": 436,
+ "content": "This code snippet contains two classes: RandomSaturation and RandomHue. Both classes are pipeline transforms for image processing in the PaddleVideo framework. The RandomSaturation class applies random saturation adjustments to images with a certain probability, while the RandomHue class randomly alters hue values of images with another probability. These transforms can be used to augment and enhance the dataset for better model training.",
+ "type": "comment"
+ },
+ "5092": {
+ "file_id": 436,
+ "content": " transform = ColorJitter(hue=self.hue)\n results['imgs'] = [transform(img) for img in imgs]\n else:\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass RandomGamma(object):\n \"\"\"\n Random Gamma images.\n Args:\n p(float): Random gamma images with the probability p.\n gamma (float): Non negative real number, same as `\\\\gamma` in the equation.\n gamma larger than 1 make the shadows darker,\n while gamma smaller than 1 make dark regions lighter.\n \"\"\"\n def __init__(self, p=0.1, gamma=0.2):\n self.p = p\n self.value = [1 - gamma, 1 + gamma]\n self.value[0] = max(self.value[0], 0)\n def _adust_gamma(self, img, gamma, gain=1.0):\n flag = False\n if isinstance(img, np.ndarray):\n flag = True\n img = Image.fromarray(img)\n input_mode = img.mode\n img = img.convert(\"RGB\")\n gamma_map = [\n int((255 + 1 - 1e-3) * gain * pow(ele / 255.0, gamma))",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:547-577"
+ },
+ "5093": {
+ "file_id": 436,
+ "content": "The code defines a pipeline module for data augmentation in PaddleVideo. It includes a RandomGamma class that randomly applies gamma correction to images with a specified probability and gamma value range. The ColorJitter transform is used to apply random changes to the hue of images. The results are stored in a dictionary under the 'imgs' key, either after applying transformations or as is if no transformation is needed. The code also handles adjusting gamma for both numpy arrays and PIL Image objects.",
+ "type": "comment"
+ },
+ "5094": {
+ "file_id": 436,
+ "content": " for ele in range(256)\n ] * 3\n img = img.point(\n gamma_map) # use PIL's point-function to accelerate this part\n img = img.convert(input_mode)\n if flag:\n img = np.array(img)\n return img\n def __call__(self, results):\n \"\"\"\n Performs random gamma operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n gamma_imgs: List where each item is a PIL.Image after random gamma.\n \"\"\"\n imgs = results['imgs']\n v = random.random()\n if v < self.p:\n gamma = random.uniform(self.value[0], self.value[1])\n results['imgs'] = [self._adust_gamma(img, gamma) for img in imgs]\n else:\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass Image2Array(object):\n \"\"\"\n transfer PIL.Image to Numpy array and transpose dimensions from 'dhwc' to 'dchw'.\n Args:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:578-611"
+ },
+ "5095": {
+ "file_id": 436,
+ "content": "This code is defining a pipeline for image augmentation, specifically adjusting gamma values randomly. It checks if a random number falls below the threshold and applies a random gamma adjustment to each image in the input list. If not, it leaves the images unchanged. Finally, it registers an Image2Array class that converts PIL.Image to Numpy array with transposed dimensions from 'dhwc' to 'dchw'.",
+ "type": "comment"
+ },
+ "5096": {
+ "file_id": 436,
+ "content": " transpose: whether to transpose or not, default True, False for slowfast.\n \"\"\"\n def __init__(self, transpose=True, data_format='tchw'):\n assert data_format in [\n 'tchw', 'cthw'\n ], f\"Target format must in ['tchw', 'cthw'], but got {data_format}\"\n self.transpose = transpose\n self.data_format = data_format\n def __call__(self, results):\n \"\"\"\n Performs Image to NumpyArray operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n np_imgs: Numpy array.\n \"\"\"\n imgs = results['imgs']\n if 'backend' in results and results[\n 'backend'] == 'pyav': # [T,H,W,C] in [0, 1]\n if self.transpose:\n if self.data_format == 'tchw':\n t_imgs = imgs.transpose((0, 3, 1, 2)) # tchw\n else:\n t_imgs = imgs.transpose((3, 0, 1, 2)) # cthw\n results['imgs'] = t_imgs",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:612-638"
+ },
+ "5097": {
+ "file_id": 436,
+ "content": "This code is part of a class that performs Image to NumpyArray operations. It initializes with the option to transpose or not, and specifies the data format as either 'tchw' or 'cthw'. The class checks if the backend is 'pyav', then transposes the images accordingly. If 'transpose' is True and 'data_format' is 'tchw', it transposes the images from (0, 3, 1, 2) to (0, 3, 1, 2), resulting in 'tchw'. Otherwise, if 'transpose' is True and 'data_format' is 'cthw', it transposes the images from (3, 0, 1, 2) to (3, 0, 1, 2), resulting in 'cthw'. The transposed images are then stored back into 'imgs'.",
+ "type": "comment"
+ },
+ "5098": {
+ "file_id": 436,
+ "content": " else:\n t_imgs = np.stack(imgs).astype('float32')\n if self.transpose:\n if self.data_format == 'tchw':\n t_imgs = t_imgs.transpose(0, 3, 1, 2) # tchw\n else:\n t_imgs = t_imgs.transpose(3, 0, 1, 2) # cthw\n results['imgs'] = t_imgs\n return results\n@PIPELINES.register()\nclass Normalization(object):\n \"\"\"\n Normalization.\n Args:\n mean(Sequence[float]): mean values of different channels.\n std(Sequence[float]): std values of different channels.\n tensor_shape(list): size of mean, default [3,1,1]. For slowfast, [1,1,1,3]\n \"\"\"\n def __init__(self, mean, std, tensor_shape=[3, 1, 1], inplace=False):\n if not isinstance(mean, Sequence):\n raise TypeError(\n f'Mean must be list, tuple or np.ndarray, but got {type(mean)}')\n if not isinstance(std, Sequence):\n raise TypeError(\n f'Std must be list, tuple or np.ndarray, but got {type(std)}')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:639-665"
+ },
+ "5099": {
+ "file_id": 436,
+ "content": "This code is defining a class for normalization in PaddleVideo's loader pipelines. It takes mean and std values as arguments to normalize the image data, and allows for transpose operation depending on data_format. The tensor_shape parameter is optional with default value [3,1,1] for standard usage or [1,1,1,3] for slowfast support. Inplace flag can be set to True to perform in-place operations if desired.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/51.json b/docs/data/51.json
new file mode 100644
index 000000000..5fa2eba68
--- /dev/null
+++ b/docs/data/51.json
@@ -0,0 +1,551 @@
+{
+ "5100": {
+ "file_id": 436,
+ "content": " self.inplace = inplace\n if not inplace:\n self.mean = np.array(mean).reshape(tensor_shape).astype(np.float32)\n self.std = np.array(std).reshape(tensor_shape).astype(np.float32)\n else:\n self.mean = np.array(mean, dtype=np.float32)\n self.std = np.array(std, dtype=np.float32)\n def __call__(self, results):\n \"\"\"\n Performs normalization operations.\n Args:\n imgs: Numpy array.\n return:\n np_imgs: Numpy array after normalization.\n \"\"\"\n if self.inplace:\n n = len(results['imgs'])\n h, w, c = results['imgs'][0].shape\n norm_imgs = np.empty((n, h, w, c), dtype=np.float32)\n for i, img in enumerate(results['imgs']):\n norm_imgs[i] = img\n for img in norm_imgs: # [n,h,w,c]\n mean = np.float64(self.mean.reshape(1, -1)) # [1, 3]\n stdinv = 1 / np.float64(self.std.reshape(1, -1)) # [1, 3]\n cv2.subtract(img, mean, img)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:667-693"
+ },
+ "5101": {
+ "file_id": 436,
+ "content": "This code defines a class for normalizing images. It takes mean and std values as inputs, which are used for image normalization. If inplace is set to False, it converts the input into numpy arrays with appropriate shapes and data types. The __call__ method performs normalization on the given results. If inplace is True, it uses the existing array and avoids making copies. The method calculates mean and std values for normalization and applies them to each image in the results using cv2.subtract.",
+ "type": "comment"
+ },
+ "5102": {
+ "file_id": 436,
+ "content": " cv2.multiply(img, stdinv, img)\n else:\n imgs = results['imgs']\n norm_imgs = imgs / 255.0\n norm_imgs -= self.mean\n norm_imgs /= self.std\n if 'backend' in results and results['backend'] == 'pyav':\n norm_imgs = paddle.to_tensor(norm_imgs, dtype=paddle.float32)\n results['imgs'] = norm_imgs\n return results\n@PIPELINES.register()\nclass JitterScale(object):\n \"\"\"\n Scale image, while the target short size is randomly select between min_size and max_size.\n Args:\n min_size: Lower bound for random sampler.\n max_size: Higher bound for random sampler.\n \"\"\"\n def __init__(self,\n min_size,\n max_size,\n short_cycle_factors=[0.5, 0.7071],\n default_min_size=256):\n self.default_min_size = default_min_size\n self.orig_min_size = self.min_size = min_size\n self.max_size = max_size\n self.short_cycle_factors = short_cycle_factors",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:694-722"
+ },
+ "5103": {
+ "file_id": 436,
+ "content": "This code applies image normalization and potentially scales the images while preserving aspect ratio, with options for random scaling. This is part of a PaddleVideo pipeline, likely for preprocessing input data before feeding it to a model for training or inference. It can be used with different backends such as \"cv2\" or \"pyav\", and returns the processed image results.",
+ "type": "comment"
+ },
+ "5104": {
+ "file_id": 436,
+ "content": " def __call__(self, results):\n \"\"\"\n Performs jitter resize operations.\n Args:\n imgs (Sequence[PIL.Image]): List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n resized_imgs: List where each item is a PIL.Image after scaling.\n \"\"\"\n short_cycle_idx = results.get('short_cycle_idx')\n if short_cycle_idx in [0, 1]:\n self.min_size = int(\n round(self.short_cycle_factors[short_cycle_idx] *\n self.default_min_size))\n else:\n self.min_size = self.orig_min_size\n imgs = results['imgs']\n size = int(round(np.random.uniform(self.min_size, self.max_size)))\n assert (len(imgs) >= 1), \\\n \"len(imgs):{} should be larger than 1\".format(len(imgs))\n if 'backend' in results and results['backend'] == 'pyav':\n height, width = imgs.shape[2:]\n else:\n width, height = imgs[0].size",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:724-749"
+ },
+ "5105": {
+ "file_id": 436,
+ "content": "This code defines a function that performs jitter resize operations. It takes in an image sequence and scales each image based on a random size between min_size and max_size, considering short cycle factors and asserting the minimum length of images. If the backend is pyav, it retrieves height and width separately; otherwise, it gets the size from the first image.",
+ "type": "comment"
+ },
+ "5106": {
+ "file_id": 436,
+ "content": " if (width <= height and width == size) or (height <= width\n and height == size):\n return results\n new_width = size\n new_height = size\n if width < height:\n new_height = int(math.floor((float(height) / width) * size))\n else:\n new_width = int(math.floor((float(width) / height) * size))\n if 'backend' in results and results['backend'] == 'pyav':\n frames_resize = F.interpolate(imgs,\n size=(new_height, new_width),\n mode=\"bilinear\",\n align_corners=False) # [c,t,h,w]\n else:\n frames_resize = []\n for j in range(len(imgs)):\n img = imgs[j]\n scale_img = img.resize((new_width, new_height), Image.BILINEAR)\n frames_resize.append(scale_img)\n results['imgs'] = frames_resize\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:750-774"
+ },
+ "5107": {
+ "file_id": 436,
+ "content": "This code resizes images to a specified size (width or height equals size). It checks if the image is loaded by PyAV and performs the resize operation using F.interpolate for PyAV-loaded images, otherwise it uses PIL's Image.resize function for other images. The resized images are added to 'imgs' in the results dictionary and returned.",
+ "type": "comment"
+ },
+ "5108": {
+ "file_id": 436,
+ "content": "@PIPELINES.register()\nclass MultiCenterCrop(object):\n \"\"\"\n center crop, left center crop right center crop\n Args:\n target_size(int): Random crop a square with the target_size from an image.\n \"\"\"\n def __init__(self, target_size):\n self.target_size = target_size\n def __call__(self, results):\n \"\"\"\n Performs random crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n crop_imgs: List where each item is a PIL.Image after random crop.\n \"\"\"\n imgs = results['imgs']\n if 'backend' in results and results['backend'] == 'pyav': # [c,t,h,w]\n h, w = imgs.shape[2:]\n else:\n w, h = imgs[0].size\n th, tw = self.target_size, self.target_size\n assert (w >= self.target_size) and (h >= self.target_size), \\\n \"image width({}) and height({}) should be larger than crop size\".format(\n w, h, self.target_size)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:777-805"
+ },
+ "5109": {
+ "file_id": 436,
+ "content": "This code defines a MultiCenterCrop class that performs center crop, left center crop, and right center crop operations on images. It takes a target size as input and returns the cropped images. The function checks if the image size is larger than the target size before performing the operation. If the image size is smaller, it throws an assertion error.",
+ "type": "comment"
+ },
+ "5110": {
+ "file_id": 436,
+ "content": " crop_images = []\n #just for tensor\n crop_imgs_center = []\n crop_imgs_left = []\n crop_imgs_right = []\n if 'backend' in results and results['backend'] == 'pyav':\n #center_corp\n x1 = 0\n if w > self.target_size:\n x1 = int((w - self.target_size) / 2.0)\n y1 = 0\n if h > self.target_size:\n y1 = int((h - self.target_size) / 2.0)\n crop_imgs_center = imgs[:, :, y1:y1 + th,\n x1:x1 + tw].numpy() # [C, T, th, tw]\n #left_crop\n x1 = 0\n y1 = 0\n if h > self.target_size:\n y1 = int((h - self.target_size) / 2.0)\n crop_imgs_left = imgs[:, :, y1:y1 + th, x1:x1 + tw].numpy()\n #right_crop\n x1 = 0\n y1 = 0\n if w > self.target_size:\n x1 = w - self.target_size\n if h > self.target_size:\n y1 = int((h - self.target_size) / 2.0)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:807-834"
+ },
+ "5111": {
+ "file_id": 436,
+ "content": "This code is performing image cropping for a specific backend (pyav) and storing the results in three separate lists: crop_imgs_center, crop_imgs_left, and crop_imgs_right. The cropping is done based on the size of the original image compared to the target size, with different crops for center, left, and right areas.",
+ "type": "comment"
+ },
+ "5112": {
+ "file_id": 436,
+ "content": " crop_imgs_right = imgs[:, :, y1:y1 + th, x1:x1 + tw].numpy()\n crop_imgs = np.concatenate(\n (crop_imgs_center, crop_imgs_left, crop_imgs_right), axis=1)\n crop_images = paddle.to_tensor(crop_imgs)\n else:\n x1 = 0\n if w > self.target_size:\n x1 = random.randint(0, w - tw)\n y1 = 0\n if h > self.target_size:\n y1 = random.randint(0, h - th)\n for img in imgs:\n if w == tw and h == th:\n crop_images.append(img)\n else:\n crop_images.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n results['imgs'] = crop_images\n return results\n@PIPELINES.register()\nclass MultiCrop(object):\n \"\"\"\n Random crop image.\n This operation can perform multi-crop during multi-clip test, as in slowfast model.\n Args:\n target_size(int): Random crop a square with the target_size from an image.\n \"\"\"\n def __init__(self,\n target_size,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:835-865"
+ },
+ "5113": {
+ "file_id": 436,
+ "content": "This code defines a MultiCrop pipeline that randomly crops an image into three parts: center, left, and right. The cropped images are concatenated horizontally and converted to Paddle Tensor before returning the results.",
+ "type": "comment"
+ },
+ "5114": {
+ "file_id": 436,
+ "content": " default_crop_size=224,\n short_cycle_factors=[0.5, 0.7071],\n test_mode=False):\n self.orig_target_size = self.target_size = target_size\n self.short_cycle_factors = short_cycle_factors\n self.default_crop_size = default_crop_size\n self.test_mode = test_mode\n def __call__(self, results):\n \"\"\"\n Performs random crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n crop_imgs: List where each item is a PIL.Image after random crop.\n \"\"\"\n imgs = results['imgs']\n spatial_sample_index = results['spatial_sample_index']\n spatial_num_clips = results['spatial_num_clips']\n short_cycle_idx = results.get('short_cycle_idx')\n if short_cycle_idx in [0, 1]:\n self.target_size = int(\n round(self.short_cycle_factors[short_cycle_idx] *\n self.default_crop_size))",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:866-891"
+ },
+ "5115": {
+ "file_id": 436,
+ "content": "The code initializes an augmentation class with parameters for target size, short cycle factors, default crop size, and test mode. It then defines a __call__ method that performs random cropping operations on images based on the provided parameters.",
+ "type": "comment"
+ },
+ "5116": {
+ "file_id": 436,
+ "content": " else:\n self.target_size = self.orig_target_size # use saved value before call\n w, h = imgs[0].size\n if w == self.target_size and h == self.target_size:\n return results\n assert (w >= self.target_size) and (h >= self.target_size), \\\n \"image width({}) and height({}) should be larger than crop size({},{})\".format(w, h, self.target_size, self.target_size)\n frames_crop = []\n if not self.test_mode:\n x_offset = random.randint(0, w - self.target_size)\n y_offset = random.randint(0, h - self.target_size)\n else: # multi-crop\n x_gap = int(\n math.ceil((w - self.target_size) / (spatial_num_clips - 1)))\n y_gap = int(\n math.ceil((h - self.target_size) / (spatial_num_clips - 1)))\n if h > w:\n x_offset = int(math.ceil((w - self.target_size) / 2))\n if spatial_sample_index == 0:\n y_offset = 0\n elif spatial_sample_index == spatial_num_clips - 1:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:892-914"
+ },
+ "5117": {
+ "file_id": 436,
+ "content": "This code checks if the image size matches the target size. If it does, it returns the results. If not, it generates crops for multi-crop testing mode or a single crop for non-testing mode based on random offsets. The code also handles the case where the target size is determined from a saved value before the call.",
+ "type": "comment"
+ },
+ "5118": {
+ "file_id": 436,
+ "content": " y_offset = h - self.target_size\n else:\n y_offset = y_gap * spatial_sample_index\n else:\n y_offset = int(math.ceil((h - self.target_size) / 2))\n if spatial_sample_index == 0:\n x_offset = 0\n elif spatial_sample_index == spatial_num_clips - 1:\n x_offset = w - self.target_size\n else:\n x_offset = x_gap * spatial_sample_index\n for img in imgs:\n nimg = img.crop((x_offset, y_offset, x_offset + self.target_size,\n y_offset + self.target_size))\n frames_crop.append(nimg)\n results['imgs'] = frames_crop\n return results\n@PIPELINES.register()\nclass PackOutput(object):\n \"\"\"\n In slowfast model, we want to get slow pathway from fast pathway based on\n alpha factor.\n Args:\n alpha(int): temporal length of fast/slow\n \"\"\"\n def __init__(self, alpha):\n self.alpha = alpha",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:915-944"
+ },
+ "5119": {
+ "file_id": 436,
+ "content": "This code calculates the crop offsets for a set of images based on their size and target size. If the aspect ratio is preserved, it determines the y_offset, otherwise, it calculates the x and y offsets separately for each spatial sample index. The resulting cropped images are stored in frames\\_crop and added to results['imgs']. PackOutput is a pipeline register that takes an alpha argument and is used in slowfast model to get slow pathway from fast pathway based on alpha factor.",
+ "type": "comment"
+ },
+ "5120": {
+ "file_id": 436,
+ "content": " def __call__(self, results):\n fast_pathway = results['imgs']\n # sample num points between start and end\n slow_idx_start = 0\n slow_idx_end = fast_pathway.shape[0] - 1\n slow_idx_num = fast_pathway.shape[0] // self.alpha\n slow_idxs_select = np.linspace(slow_idx_start, slow_idx_end,\n slow_idx_num).astype(\"int64\")\n slow_pathway = fast_pathway[slow_idxs_select]\n # T H W C -> C T H W.\n slow_pathway = slow_pathway.transpose(3, 0, 1, 2)\n fast_pathway = fast_pathway.transpose(3, 0, 1, 2)\n # slow + fast\n frames_list = [slow_pathway, fast_pathway]\n results['imgs'] = frames_list\n return results\n@PIPELINES.register()\nclass GroupFullResSample(object):\n def __init__(self, crop_size, flip=False):\n self.crop_size = crop_size if not isinstance(crop_size, int) else (\n crop_size, crop_size)\n self.flip = flip\n def __call__(self, results):\n img_group = results['imgs']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:946-975"
+ },
+ "5121": {
+ "file_id": 436,
+ "content": "The code defines a GroupFullResSample pipeline that selects and groups slow and fast pathways from input images. It resizes the pathways to the specified crop_size, performs horizontal flips if flip is True, and stores them in frames_list before updating results['imgs'].",
+ "type": "comment"
+ },
+ "5122": {
+ "file_id": 436,
+ "content": " image_w, image_h = img_group[0].size\n crop_w, crop_h = self.crop_size\n w_step = (image_w - crop_w) // 4\n h_step = (image_h - crop_h) // 4\n offsets = list()\n offsets.append((0 * w_step, 2 * h_step)) # left\n offsets.append((4 * w_step, 2 * h_step)) # right\n offsets.append((2 * w_step, 2 * h_step)) # center\n oversample_group = list()\n for o_w, o_h in offsets:\n normal_group = list()\n flip_group = list()\n for i, img in enumerate(img_group):\n crop = img.crop((o_w, o_h, o_w + crop_w, o_h + crop_h))\n normal_group.append(crop)\n if self.flip:\n flip_crop = crop.copy().transpose(Image.FLIP_LEFT_RIGHT)\n flip_group.append(flip_crop)\n oversample_group.extend(normal_group)\n if self.flip:\n oversample_group.extend(flip_group)\n results['imgs'] = oversample_group\n return results\n@PIPELINES.register()",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:977-1007"
+ },
+ "5123": {
+ "file_id": 436,
+ "content": "This code performs image augmentation by creating a list of different crops and flips from the input image group. It calculates the crop size and step sizes, creates offsets for each crop position, iterates over the input images to create normal and flipped crops, and stores them in separate groups before combining them into the oversample_group. Finally, it adds the oversample_group to the results dictionary and returns the results.",
+ "type": "comment"
+ },
+ "5124": {
+ "file_id": 436,
+ "content": "class TenCrop:\n \"\"\"\n Crop out 5 regions (4 corner points + 1 center point) from the picture,\n and then flip the cropping result to get 10 cropped images, which can make the prediction result more robust.\n Args:\n target_size(int | tuple[int]): (w, h) of target size for crop.\n \"\"\"\n def __init__(self, target_size):\n self.target_size = (target_size, target_size)\n def __call__(self, results):\n imgs = results['imgs']\n img_w, img_h = imgs[0].size\n crop_w, crop_h = self.target_size\n w_step = (img_w - crop_w) // 4\n h_step = (img_h - crop_h) // 4\n offsets = [\n (0, 0),\n (4 * w_step, 0),\n (0, 4 * h_step),\n (4 * w_step, 4 * h_step),\n (2 * w_step, 2 * h_step),\n ]\n img_crops = list()\n for x_offset, y_offset in offsets:\n crop = [\n img.crop(\n (x_offset, y_offset, x_offset + crop_w, y_offset + crop_h))\n for img in imgs\n ]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1008-1037"
+ },
+ "5125": {
+ "file_id": 436,
+ "content": "This code defines a class \"TenCrop\" which crops a given image into 10 cropped images, taking the top-left corner, bottom-left corner, top-right corner, bottom-right corner, and center of the image. It achieves this by using the target size for crop and calculating the width and height steps based on the original image's dimensions. The class also includes a __call__ method which takes a results dictionary as input and returns an array of cropped images.",
+ "type": "comment"
+ },
+ "5126": {
+ "file_id": 436,
+ "content": " crop_fliped = [\n timg.transpose(Image.FLIP_LEFT_RIGHT) for timg in crop\n ]\n img_crops.extend(crop)\n img_crops.extend(crop_fliped)\n results['imgs'] = img_crops\n return results\n@PIPELINES.register()\nclass UniformCrop:\n \"\"\"\n Perform uniform spatial sampling on the images,\n select the two ends of the long side and the middle position (left middle right or top middle bottom) 3 regions.\n Args:\n target_size(int | tuple[int]): (w, h) of target size for crop.\n \"\"\"\n def __init__(self, target_size, backend='cv2'):\n if isinstance(target_size, tuple):\n self.target_size = target_size\n elif isinstance(target_size, int):\n self.target_size = (target_size, target_size)\n else:\n raise TypeError(\n f'target_size must be int or tuple[int], but got {type(target_size)}'\n )\n self.backend = backend\n def __call__(self, results):\n imgs = results['imgs']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1038-1069"
+ },
+ "5127": {
+ "file_id": 436,
+ "content": "This code is for the \"UniformCrop\" pipeline, which performs uniform spatial sampling on images by selecting three regions: two ends of the long side and the middle position (left/right or top/bottom). The target size can be provided as an integer for square crop or a tuple for specific width and height. It uses either OpenCV or PIL for image manipulation based on the 'backend' argument, which defaults to OpenCV.",
+ "type": "comment"
+ },
+ "5128": {
+ "file_id": 436,
+ "content": " if 'backend' in results and results['backend'] == 'pyav': # [c,t,h,w]\n img_h, img_w = imgs.shape[2:]\n elif self.backend == 'pillow':\n img_w, img_h = imgs[0].size\n else:\n img_h, img_w = imgs[0].shape[:2]\n crop_w, crop_h = self.target_size\n if crop_h == img_h:\n w_step = (img_w - crop_w) // 2\n offsets = [\n (0, 0),\n (w_step * 2, 0),\n (w_step, 0),\n ]\n elif crop_w == img_w:\n h_step = (img_h - crop_h) // 2\n offsets = [\n (0, 0),\n (0, h_step * 2),\n (0, h_step),\n ]\n else:\n raise ValueError(\n f\"img_w({img_w}) == crop_w({crop_w}) or img_h({img_h}) == crop_h({crop_h})\"\n )\n img_crops = []\n if 'backend' in results and results['backend'] == 'pyav': # [c,t,h,w]\n for x_offset, y_offset in offsets:\n crop = imgs[:, :, y_offset:y_offset + crop_h,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1070-1099"
+ },
+ "5129": {
+ "file_id": 436,
+ "content": "This code is determining the image offsets for cropping based on the target size and the original image dimensions. It supports two backends: 'pyav' and 'pillow'. If the backend is 'pyav', it extracts the height and width of the image. If the backend is 'pillow', it retrieves the width and height from the first image. The code then calculates the step size for cropping based on whether the target size matches the image dimensions or not, and finally constructs a list of offsets to crop the images.",
+ "type": "comment"
+ },
+ "5130": {
+ "file_id": 436,
+ "content": " x_offset:x_offset + crop_w]\n img_crops.append(crop)\n img_crops = paddle.concat(img_crops, axis=1)\n else:\n if self.backend == 'pillow':\n for x_offset, y_offset in offsets:\n crop = [\n img.crop((x_offset, y_offset, x_offset + crop_w,\n y_offset + crop_h)) for img in imgs\n ]\n img_crops.extend(crop)\n else:\n for x_offset, y_offset in offsets:\n crop = [\n img[y_offset:y_offset + crop_h,\n x_offset:x_offset + crop_w] for img in imgs\n ]\n img_crops.extend(crop)\n results['imgs'] = img_crops\n return results\n@PIPELINES.register()\nclass GroupResize(object):\n def __init__(self, height, width, scale, K, mode='train'):\n self.height = height\n self.width = width\n self.scale = scale",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1100-1127"
+ },
+ "5131": {
+ "file_id": 436,
+ "content": "The code is defining a pipeline for image augmentation, including cropping and resizing operations. If the backend is 'pillow', it performs cropping using pixel coordinates; otherwise, it uses slice notation to crop images. The results are stored in 'img_crops' and returned as 'results['imgs']'.",
+ "type": "comment"
+ },
+ "5132": {
+ "file_id": 436,
+ "content": " self.resize = {}\n self.K = np.array(K, dtype=np.float32)\n self.mode = mode\n for i in range(self.scale):\n s = 2**i\n self.resize[i] = paddle.vision.transforms.Resize(\n (self.height // s, self.width // s), interpolation='lanczos')\n def __call__(self, results):\n if self.mode == 'infer':\n imgs = results['imgs']\n for k in list(imgs): # (\"color\", 0, -1)\n if \"color\" in k or \"color_n\" in k:\n n, im, _ = k\n for i in range(self.scale):\n imgs[(n, im, i)] = self.resize[i](imgs[(n, im, i - 1)])\n else:\n imgs = results['imgs']\n for scale in range(self.scale):\n K = self.K.copy()\n K[0, :] *= self.width // (2**scale)\n K[1, :] *= self.height // (2**scale)\n inv_K = np.linalg.pinv(K)\n imgs[(\"K\", scale)] = K\n imgs[(\"inv_K\", scale)] = inv_K\n for k in list(imgs):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1128-1156"
+ },
+ "5133": {
+ "file_id": 436,
+ "content": "This code initializes a resize transformation for image augmentation in PaddleVideo. The transformations are applied based on the scale and mode ('infer' or 'train') specified. For infer mode, it processes color images by applying resizing to each scale level. In train mode, it calculates the K matrix and its inverse for each scale level and stores them in the results dictionary.",
+ "type": "comment"
+ },
+ "5134": {
+ "file_id": 436,
+ "content": " if \"color\" in k or \"color_n\" in k:\n n, im, i = k\n for i in range(self.scale):\n imgs[(n, im, i)] = self.resize[i](imgs[(n, im, i - 1)])\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass ColorJitter(object):\n \"\"\"Randomly change the brightness, contrast, saturation and hue of an image.\n \"\"\"\n def __init__(self,\n brightness=0,\n contrast=0,\n saturation=0,\n hue=0,\n mode='train',\n p=0.5,\n keys=None):\n self.mode = mode\n self.colorjitter = paddle.vision.transforms.ColorJitter(\n brightness, contrast, saturation, hue)\n self.p = p\n def __call__(self, results):\n \"\"\"\n Args:\n results (PIL Image): Input image.\n Returns:\n PIL Image: Color jittered image.\n \"\"\"\n do_color_aug = random.random() > self.p\n imgs = results['imgs']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1157-1193"
+ },
+ "5135": {
+ "file_id": 436,
+ "content": "This code applies color jitter augmentation to the images by randomly adjusting brightness, contrast, saturation, and hue. The ColorJitter class initializes a colorjitter transform with specified parameters for train mode or test mode. The __call__ method is called on each image in the results dictionary and checks if the color augmentation should be applied. If true, the color jittered image is returned.",
+ "type": "comment"
+ },
+ "5136": {
+ "file_id": 436,
+ "content": " for k in list(imgs):\n f = imgs[k]\n if \"color\" in k or \"color_n\" in k:\n n, im, i = k\n imgs[(n, im, i)] = f\n if do_color_aug:\n imgs[(n + \"_aug\", im, i)] = self.colorjitter(f)\n else:\n imgs[(n + \"_aug\", im, i)] = f\n if self.mode == \"train\":\n for i in results['frame_idxs']:\n del imgs[(\"color\", i, -1)]\n del imgs[(\"color_aug\", i, -1)]\n del imgs[(\"color_n\", i, -1)]\n del imgs[(\"color_n_aug\", i, -1)]\n else:\n for i in results['frame_idxs']:\n del imgs[(\"color\", i, -1)]\n del imgs[(\"color_aug\", i, -1)]\n results['img'] = imgs\n return results\n@PIPELINES.register()\nclass GroupRandomFlip(object):\n def __init__(self, p=0.5):\n self.p = p\n def __call__(self, results):\n imgs = results['imgs']\n do_flip = random.random() > self.p\n if do_flip:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1194-1227"
+ },
+ "5137": {
+ "file_id": 436,
+ "content": "This code is part of a data augmentation pipeline, specifically handling color and flip transformations for images. It iterates over the 'imgs' dictionary to find and organize color images, applying color jitter if required. Then it removes specific color images based on the mode (\"train\" or \"test\"). Finally, it returns the updated results dictionary with the modified image groupings. The GroupRandomFlip class performs random flipping of images with a specified probability.",
+ "type": "comment"
+ },
+ "5138": {
+ "file_id": 436,
+ "content": " for k in list(imgs):\n if \"color\" in k or \"color_n\" in k:\n n, im, i = k\n imgs[(n, im,\n i)] = imgs[(n, im,\n i)].transpose(Image.FLIP_LEFT_RIGHT)\n if \"depth_gt\" in imgs:\n imgs['depth_gt'] = np.array(np.fliplr(imgs['depth_gt']))\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass ToArray(object):\n def __init__(self):\n pass\n def __call__(self, results):\n imgs = results['imgs']\n for k in list(imgs):\n if \"color\" in k or \"color_n\" in k or \"color_aug\" in k or \"color_n_aug\" in k:\n n, im, i = k\n imgs[(n, im,\n i)] = np.array(imgs[(n, im, i)]).astype('float32') / 255.0\n imgs[(n, im, i)] = imgs[(n, im, i)].transpose((2, 0, 1))\n if \"depth_gt\" in imgs:\n imgs['depth_gt'] = np.array(imgs['depth_gt']).astype('float32')\n results['imgs'] = imgs",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1228-1257"
+ },
+ "5139": {
+ "file_id": 436,
+ "content": "This code is part of a machine learning pipeline that processes image data. It first flips left-right some images marked with \"color\" or \"color_n\". Then, it converts certain color and depth images to floats and normalizes them to [0,1] for training. Finally, it returns the updated image dictionary as part of the results.",
+ "type": "comment"
+ },
+ "5140": {
+ "file_id": 436,
+ "content": " return results\n@PIPELINES.register()\nclass YowoAug(object):\n def __init__(self, target_size=224, jitter=0.2, hue=0.1, saturation=1.5, exposure=1.5, valid_mode=False):\n self.shape = (target_size, target_size)\n self.jitter = jitter\n self.hue = hue\n self.saturation = saturation\n self.exposure = exposure\n self.valid_mode = valid_mode\n def _rand_scale(self, s):\n scale = random.uniform(1, s)\n if (random.randint(1, 10000) % 2):\n return scale\n return 1. / scale\n def _distort_image(self, im, hue, sat, val):\n im = im.convert('HSV')\n cs = list(im.split())\n cs[1] = cs[1].point(lambda i: i * sat)\n cs[2] = cs[2].point(lambda i: i * val)\n def _change_hue(x):\n x += hue * 255\n if x > 255:\n x -= 255\n if x < 0:\n x += 255\n return x\n cs[0] = cs[0].point(_change_hue)\n im = Image.merge(im.mode, tuple(cs))\n im = im.convert('RGB')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1258-1294"
+ },
+ "5141": {
+ "file_id": 436,
+ "content": "This code defines a class called YowoAug for image augmentation. It takes in parameters such as target size, jitter, hue, saturation, exposure, and valid mode. The class has methods to randomly scale the image, distort the image by changing hue, saturation, and exposure levels, and returns the augmented results.",
+ "type": "comment"
+ },
+ "5142": {
+ "file_id": 436,
+ "content": " # constrain_image(im)\n return im\n def _random_distort_image(self, im, dhue, dsat, dexp):\n res = self._distort_image(im, dhue, dsat, dexp)\n return res\n def _read_truths_args(self, lab_path, min_box_scale):\n truths = np.loadtxt(lab_path)\n truths = np.reshape(truths, (truths.size // 5, 5))\n new_truths = []\n for i in range(truths.shape[0]):\n cx = (truths[i][1] + truths[i][3]) / (2 * 320)\n cy = (truths[i][2] + truths[i][4]) / (2 * 240)\n imgw = (truths[i][3] - truths[i][1]) / 320\n imgh = (truths[i][4] - truths[i][2]) / 240\n truths[i][0] = truths[i][0] - 1\n truths[i][1] = cx\n truths[i][2] = cy\n truths[i][3] = imgw\n truths[i][4] = imgh\n if truths[i][3] < min_box_scale:\n continue\n new_truths.append([truths[i][0], truths[i][1], truths[i][2], truths[i][3], truths[i][4]])\n return np.array(new_truths)\n def _fill_truth_detection(self, labpath, flip, dx, dy, sx, sy):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1295-1322"
+ },
+ "5143": {
+ "file_id": 436,
+ "content": "The code snippet defines several functions related to image augmentation and truth detection in an object detection task. The \"constrain_image\" function ensures the image is within a specific range of values. \"random_distort_image\" applies distortion to the input image randomly. \"read_truths_args\" reads the ground truth boxes from a file, scales and transforms them accordingly, and checks if the box scale is smaller than the minimum required scale before adding it to new_truths. Lastly, \"_fill_truth_detection\" fills in the ground truth detection with additional parameters like flip, dx, dy, sx, and sy.",
+ "type": "comment"
+ },
+ "5144": {
+ "file_id": 436,
+ "content": " max_boxes = 50\n label = np.zeros((max_boxes, 5))\n bs = np.loadtxt(labpath)\n bs = np.reshape(bs, (-1, 5))\n for i in range(bs.shape[0]):\n cx = (bs[i][1] + bs[i][3]) / (2 * 320)\n cy = (bs[i][2] + bs[i][4]) / (2 * 240)\n imgw = (bs[i][3] - bs[i][1]) / 320\n imgh = (bs[i][4] - bs[i][2]) / 240\n bs[i][0] = bs[i][0] - 1\n bs[i][1] = cx\n bs[i][2] = cy\n bs[i][3] = imgw\n bs[i][4] = imgh\n cc = 0\n for i in range(bs.shape[0]):\n x1 = bs[i][1] - bs[i][3] / 2\n y1 = bs[i][2] - bs[i][4] / 2\n x2 = bs[i][1] + bs[i][3] / 2\n y2 = bs[i][2] + bs[i][4] / 2\n x1 = min(0.999, max(0, x1 * sx - dx))\n y1 = min(0.999, max(0, y1 * sy - dy))\n x2 = min(0.999, max(0, x2 * sx - dx))\n y2 = min(0.999, max(0, y2 * sy - dy))\n bs[i][1] = (x1 + x2) / 2\n bs[i][2] = (y1 + y2) / 2\n bs[i][3] = (x2 - x1)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1323-1353"
+ },
+ "5145": {
+ "file_id": 436,
+ "content": "The code resizes and normalizes bounding box coordinates from a loaded label file, adjusts them based on image size scaling factors and offsets, and updates the bounding boxes accordingly.",
+ "type": "comment"
+ },
+ "5146": {
+ "file_id": 436,
+ "content": " bs[i][4] = (y2 - y1)\n if flip:\n bs[i][1] = 0.999 - bs[i][1]\n if bs[i][3] < 0.001 or bs[i][4] < 0.001:\n continue\n label[cc] = bs[i]\n cc += 1\n if cc >= 50:\n break\n label = np.reshape(label, (-1))\n return label\n def __call__(self, results):\n clip = results['imgs']\n frame_num = len(clip)\n oh = clip[0].height\n ow = clip[0].width\n labpath = results['filename'].replace('jpg', 'txt').replace('rgb-images', 'labels')\n if not self.valid_mode:\n dw = int(ow * self.jitter)\n dh = int(oh * self.jitter)\n pleft = random.randint(-dw, dw)\n pright = random.randint(-dw, dw)\n ptop = random.randint(-dh, dh)\n pbot = random.randint(-dh, dh)\n swidth = ow - pleft - pright\n sheight = oh - ptop - pbot\n sx = float(swidth) / ow\n sy = float(sheight) / oh\n dx = (float(pleft) / ow) / sx",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1354-1390"
+ },
+ "5147": {
+ "file_id": 436,
+ "content": "This code initializes a list of bounding boxes, applies jitter if not in valid mode (randomly adjusts image size and position), reshapes the list into a single array of bounding boxes, and returns it.",
+ "type": "comment"
+ },
+ "5148": {
+ "file_id": 436,
+ "content": " dy = (float(ptop) / oh) / sy\n flip = random.randint(1, 10000) % 2\n dhue = random.uniform(-self.hue, self.hue)\n dsat = self._rand_scale(self.saturation)\n dexp = self._rand_scale(self.exposure)\n # Augment\n cropped = [img.crop((pleft, ptop, pleft + swidth - 1, ptop + sheight - 1)) for img in clip]\n sized = [img.resize(self.shape) for img in cropped]\n if flip:\n sized = [img.transpose(Image.FLIP_LEFT_RIGHT) for img in sized]\n clip = [self._random_distort_image(img, dhue, dsat, dexp) for img in sized]\n label = self._fill_truth_detection(labpath, flip, dx, dy, 1. / sx, 1. / sy)\n else:\n label = np.zeros([50 * 5])\n tmp = self._read_truths_args(labpath, 8.0 / clip[0].width).astype('float32')\n tmp = np.reshape(tmp, [-1])\n tsz = tmp.size\n if tsz > 50 * 5:\n label = tmp[0:50 * 5]\n elif tsz > 0:\n label[0:tsz] = tmp",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1391-1419"
+ },
+ "5149": {
+ "file_id": 436,
+ "content": "This code performs image augmentations and label manipulations. It applies random crop, resize, flip (horizontally), and distortion to the image(s) with a certain probability. The label is either filled from the truth detection or set to zero vector depending on the size of the extracted truth data.",
+ "type": "comment"
+ },
+ "5150": {
+ "file_id": 436,
+ "content": " clip = [img.resize(self.shape) for img in clip]\n clip = [np.asarray(img).astype('float32') / 255.0 for img in clip]\n clip = np.concatenate(clip, 0).reshape([frame_num, 224, 224, 3])\n clip = np.transpose(clip, [3, 0, 1, 2])\n results['imgs'] = clip\n results['labels'] = label\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations.py:1420-1427"
+ },
+ "5151": {
+ "file_id": 436,
+ "content": "Resizes images to a specific shape, converts them to float32 type and scales by 255.0, concatenates frames into a single array, transposes array dimensions, stores image and label arrays in 'results', returns results",
+ "type": "comment"
+ },
+ "5152": {
+ "file_id": 437,
+ "content": "/paddlevideo/loader/pipelines/augmentations_ava.py",
+ "type": "filepath"
+ },
+ "5153": {
+ "file_id": 437,
+ "content": "This code defines image augmentation and resizing functions for AVA dataset in PaddleVideo library using operations like resizing, lazy initialization, and RandomRescale/Resize transforms. It creates classes for ground truth bounding boxes and proposals, cropping and flipping entity boxes, and includes Flip and Normalize classes for image processing and normalization.",
+ "type": "summary"
+ },
+ "5154": {
+ "file_id": 437,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport numpy as np\nimport math\nfrom PIL import Image\nfrom ..registry import PIPELINES\nfrom collections.abc import Sequence\nimport cv2\npillow_interp_codes = {\n 'nearest': Image.NEAREST,\n 'bilinear': Image.BILINEAR,\n 'bicubic': Image.BICUBIC,\n 'box': Image.BOX,\n 'lanczos': Image.LANCZOS,\n 'hamming': Image.HAMMING\n}\ncv2_interp_codes = {\n 'nearest': cv2.INTER_NEAREST,\n 'bilinear': cv2.INTER_LINEAR,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:1-34"
+ },
+ "5155": {
+ "file_id": 437,
+ "content": "This code is part of the PaddleVideo library and contains a module for image augmentations in the AVA dataset. It imports necessary libraries, defines conversion dictionaries for interpolation methods between PIL and OpenCV, and sets up registry entries for the PIPELINES module.",
+ "type": "comment"
+ },
+ "5156": {
+ "file_id": 437,
+ "content": " 'bicubic': cv2.INTER_CUBIC,\n 'area': cv2.INTER_AREA,\n 'lanczos': cv2.INTER_LANCZOS4\n}\ndef _init_lazy_if_proper(results, lazy):\n \"\"\"Initialize lazy operation properly.\n Make sure that a lazy operation is properly initialized,\n and avoid a non-lazy operation accidentally getting mixed in.\n Required keys in results are \"imgs\" if \"img_shape\" not in results,\n otherwise, Required keys in results are \"img_shape\", add or modified keys\n are \"img_shape\", \"lazy\".\n Add or modified keys in \"lazy\" are \"original_shape\", \"crop_bbox\", \"flip\",\n \"flip_direction\", \"interpolation\".\n Args:\n results (dict): A dict stores data pipeline result.\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n if 'img_shape' not in results:\n results['img_shape'] = results['imgs'][0].shape[:2]\n if lazy:\n if 'lazy' not in results:\n img_h, img_w = results['img_shape']\n lazyop = dict()\n lazyop['original_shape'] = results['img_shape']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:35-64"
+ },
+ "5157": {
+ "file_id": 437,
+ "content": "This function initializes the lazy operation properly, ensuring a non-lazy operation is not accidentally mixed in. If 'img_shape' is not in results, it adds 'img_shape'. If 'lazy' is set to True and 'lazy' does not exist in results, it creates a new dictionary for lazy operation containing 'original_shape', 'img_shape', 'crop_bbox', 'flip', 'flip_direction', and 'interpolation'.",
+ "type": "comment"
+ },
+ "5158": {
+ "file_id": 437,
+ "content": " lazyop['crop_bbox'] = np.array([0, 0, img_w, img_h],\n dtype=np.float32)\n lazyop['flip'] = False\n lazyop['flip_direction'] = None\n lazyop['interpolation'] = None\n results['lazy'] = lazyop\n else:\n assert 'lazy' not in results, 'Use Fuse after lazy operations'\ndef _scale_size(size, scale):\n \"\"\"Rescale a size by a ratio.\n Args:\n size (tuple[int]): (w, h).\n scale (float): Scaling factor.\n Returns:\n tuple[int]: scaled size.\n \"\"\"\n w, h = size\n return int(w * float(scale) + 0.5), int(h * float(scale) + 0.5)\ndef rescale_size(old_size, scale, return_scale=False):\n \"\"\"Calculate the new size to be rescaled to.\n Args:\n old_size (tuple[int]): The old size (w, h) of image.\n scale (float | tuple[int]): The scaling factor or maximum size.\n If it is a float number, then the image will be rescaled by this\n factor, else if it is a tuple of 2 integers, then the image will",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:65-96"
+ },
+ "5159": {
+ "file_id": 437,
+ "content": "This code defines functions for image augmentations in the AVA dataset pipeline. The \"_scale_size\" function scales a size by a ratio, while \"rescale_size\" calculates the new size to be rescaled based on an input scale factor or maximum size. The code also includes the initialization of crop parameters and flipping options for lazy operations in the results dictionary.",
+ "type": "comment"
+ },
+ "5160": {
+ "file_id": 437,
+ "content": " be rescaled as large as possible within the scale.\n return_scale (bool): Whether to return the scaling factor besides the\n rescaled image size.\n Returns:\n tuple[int]: The new rescaled image size.\n \"\"\"\n w, h = old_size\n if isinstance(scale, (float, int)):\n if scale <= 0:\n raise ValueError(f'Invalid scale {scale}, must be positive.')\n scale_factor = scale\n elif isinstance(scale, tuple):\n max_long_edge = max(scale)\n max_short_edge = min(scale)\n scale_factor = min(max_long_edge / max(h, w),\n max_short_edge / min(h, w))\n else:\n raise TypeError(\n f'Scale must be a number or tuple of int, but got {type(scale)}')\n new_size = _scale_size((w, h), scale_factor)\n if return_scale:\n return new_size, scale_factor\n else:\n return new_size\ndef imresize(img,\n size,\n return_scale=False,\n interpolation='bilinear',\n out=None,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:97-130"
+ },
+ "5161": {
+ "file_id": 437,
+ "content": "Function \"imresize\" resizes an image based on the provided scale factor. If the scale is a number, it's used directly as the scaling factor. If it's a tuple of ints, it sets max and min edge sizes for resizing. Returns new resized size or both size and scaling factor if requested.",
+ "type": "comment"
+ },
+ "5162": {
+ "file_id": 437,
+ "content": " backend=None):\n \"\"\"Resize image to a given size. \"\"\"\n h, w = img.shape[:2]\n if backend is None:\n backend = 'cv2'\n if backend not in ['cv2', 'pillow']:\n raise ValueError(f'backend: {backend} is not supported for resize.'\n f\"Supported backends are 'cv2', 'pillow'\")\n if backend == 'pillow':\n assert img.dtype == np.uint8, 'Pillow backend only support uint8 type'\n pil_image = Image.fromarray(img)\n pil_image = pil_image.resize(size, pillow_interp_codes[interpolation])\n resized_img = np.array(pil_image)\n else:\n resized_img = cv2.resize(\n img, size, dst=out, interpolation=cv2_interp_codes[interpolation])\n if not return_scale:\n return resized_img\n else:\n w_scale = size[0] / w\n h_scale = size[1] / h\n return resized_img, w_scale, h_scale\n@PIPELINES.register()\nclass EntityBoxRescale:\n \"\"\"Rescale the entity box and proposals according to the image shape.\n Required keys are \"proposals\", \"gt_bboxes\", added or modified keys are",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:131-160"
+ },
+ "5163": {
+ "file_id": 437,
+ "content": "This code defines a function for resizing an image to a given size. It supports two backends: 'cv2' and 'pillow'. If the backend is not specified, it defaults to 'cv2'. The function first gets the original image's height and width, checks if the backend is valid, handles unsupported backends, asserts the image type for 'pillow' backend, resizes the image using either OpenCV or Pillow library based on the backend, and finally returns the resized image along with scale factors if return_scale is True. The EntityBoxRescale class registers a pipeline to rescale entity boxes and proposals according to the image shape.",
+ "type": "comment"
+ },
+ "5164": {
+ "file_id": 437,
+ "content": " \"gt_bboxes\". If original \"proposals\" is not None, \"proposals\" and\n will be added or modified.\n Args:\n scale_factor (np.ndarray): The scale factor used entity_box rescaling.\n \"\"\"\n def __init__(self, scale_factor):\n self.scale_factor = scale_factor\n def __call__(self, results):\n scale_factor = np.concatenate([self.scale_factor, self.scale_factor])\n if 'gt_bboxes' in results:\n gt_bboxes = results['gt_bboxes']\n results['gt_bboxes'] = gt_bboxes * scale_factor\n if 'proposals' in results:\n proposals = results['proposals']\n if proposals is not None:\n assert proposals.shape[1] == 4, (\n 'proposals shape should be in '\n f'(n, 4), but got {proposals.shape}')\n results['proposals'] = proposals * scale_factor\n return results\n def __repr__(self):\n return f'{self.__class__.__name__}(scale_factor={self.scale_factor})'\n@PIPELINES.register()\nclass EntityBoxCrop:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:161-193"
+ },
+ "5165": {
+ "file_id": 437,
+ "content": "The code defines a class called EntityBoxCrop that scales the ground truth bounding boxes (gt_bboxes) and proposals, if present, by a given scale factor. It ensures that the number of columns in the proposals is 4. This class can be registered as a pipeline for video augmentation.",
+ "type": "comment"
+ },
+ "5166": {
+ "file_id": 437,
+ "content": " \"\"\"Crop the entity boxes and proposals according to the cropped images.\n Required keys are \"proposals\", \"gt_bboxes\", added or modified keys are\n \"gt_bboxes\". If original \"proposals\" is not None, \"proposals\" will be\n modified.\n Args:\n crop_bbox(np.ndarray | None): The bbox used to crop the original image.\n \"\"\"\n def __init__(self, crop_bbox):\n self.crop_bbox = crop_bbox\n def __call__(self, results):\n proposals = results['proposals']\n gt_bboxes = results['gt_bboxes']\n if self.crop_bbox is None:\n return results\n x1, y1, x2, y2 = self.crop_bbox\n img_w, img_h = x2 - x1, y2 - y1\n assert gt_bboxes.shape[-1] == 4\n gt_bboxes_ = gt_bboxes.copy()\n gt_bboxes_[..., 0::2] = np.clip(gt_bboxes[..., 0::2] - x1, 0, img_w - 1)\n gt_bboxes_[..., 1::2] = np.clip(gt_bboxes[..., 1::2] - y1, 0, img_h - 1)\n results['gt_bboxes'] = gt_bboxes_\n if proposals is not None:\n assert proposals.shape[-1] == 4",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:194-224"
+ },
+ "5167": {
+ "file_id": 437,
+ "content": "This code initializes an object that crops the entity boxes and proposals according to the cropped images. The required keys are \"proposals\" and \"gt_bboxes\", while \"gt_bboxes\" is added or modified. If original \"proposals\" is not None, \"proposals\" will be modified. The crop_bbox argument specifies the bbox used to crop the original image.",
+ "type": "comment"
+ },
+ "5168": {
+ "file_id": 437,
+ "content": " proposals_ = proposals.copy()\n proposals_[..., 0::2] = np.clip(proposals[..., 0::2] - x1, 0,\n img_w - 1)\n proposals_[..., 1::2] = np.clip(proposals[..., 1::2] - y1, 0,\n img_h - 1)\n results['proposals'] = proposals_\n return results\n def __repr__(self):\n return f'{self.__class__.__name__}(crop_bbox={self.crop_bbox})'\n@PIPELINES.register()\nclass EntityBoxFlip:\n \"\"\"Flip the entity boxes and proposals with a probability.\n Reverse the order of elements in the given bounding boxes and proposals\n with a specific direction. The shape of them are preserved, but the\n elements are reordered. Only the horizontal flip is supported (seems\n vertical flipping makes no sense). Required keys are \"proposals\",\n \"gt_bboxes\", added or modified keys are \"gt_bboxes\". If \"proposals\"\n is not None, it will also be modified.\n Args:\n img_shape (tuple[int]): The img shape.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:225-249"
+ },
+ "5169": {
+ "file_id": 437,
+ "content": "This code defines two classes, \"EntityBoxFlip\" and a nameless class. The nameless class performs cropping operations on proposals based on given coordinates (x1, y1). It also updates the results['proposals'] with the modified proposals. The EntityBoxFlip class flips the entity boxes and proposals horizontally with a certain probability. It adds or modifies keys \"gt_bboxes\" in the results dictionary. If \"proposals\" is not None, it will also modify them. The img_shape tuple represents the image shape.",
+ "type": "comment"
+ },
+ "5170": {
+ "file_id": 437,
+ "content": " \"\"\"\n def __init__(self, img_shape):\n self.img_shape = img_shape\n def __call__(self, results):\n proposals = results['proposals']\n gt_bboxes = results['gt_bboxes']\n img_h, img_w = self.img_shape\n assert gt_bboxes.shape[-1] == 4\n gt_bboxes_ = gt_bboxes.copy()\n gt_bboxes_[..., 0::4] = img_w - gt_bboxes[..., 2::4] - 1\n gt_bboxes_[..., 2::4] = img_w - gt_bboxes[..., 0::4] - 1\n if proposals is not None:\n assert proposals.shape[-1] == 4\n proposals_ = proposals.copy()\n proposals_[..., 0::4] = img_w - proposals[..., 2::4] - 1\n proposals_[..., 2::4] = img_w - proposals[..., 0::4] - 1\n else:\n proposals_ = None\n results['proposals'] = proposals_\n results['gt_bboxes'] = gt_bboxes_\n return results\n def __repr__(self):\n repr_str = f'{self.__class__.__name__}(img_shape={self.img_shape})'\n return repr_str\n@PIPELINES.register()\nclass Resize:\n \"\"\"Resize images to a specific size.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:250-284"
+ },
+ "5171": {
+ "file_id": 437,
+ "content": "This code defines a pipeline for resizing images to a specific size. It first initializes the pipeline with an image shape and then, in the __call__ method, it adjusts the ground truth bounding boxes and proposal bounding boxes by subtracting their width values from the total image width minus 1. If there are no proposals, it sets proposals_ to None. The __repr__ method provides a string representation of the pipeline.",
+ "type": "comment"
+ },
+ "5172": {
+ "file_id": 437,
+ "content": " Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified\n keys are \"imgs\", \"img_shape\", \"keep_ratio\", \"scale_factor\", \"lazy\",\n \"resize_size\". Required keys in \"lazy\" is None, added or modified key is\n \"interpolation\".\n Args:\n scale (float | Tuple[int]): If keep_ratio is True, it serves as scaling\n factor or maximum size:\n If it is a float number, the image will be rescaled by this\n factor, else if it is a tuple of 2 integers, the image will\n be rescaled as large as possible within the scale.\n Otherwise, it serves as (w, h) of output size.\n keep_ratio (bool): If set to True, Images will be resized without\n changing the aspect ratio. Otherwise, it will resize images to a\n given size. Default: True.\n interpolation (str): Algorithm used for interpolation:\n \"nearest\" | \"bilinear\". Default: \"bilinear\".\n lazy (bool): Determine whether to apply lazy operation. Default: False.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:286-303"
+ },
+ "5173": {
+ "file_id": 437,
+ "content": "This code defines a function for image augmentation in PaddleVideo. The function takes arguments like scale, keep_ratio, interpolation, and lazy. If keep_ratio is True, it scales the image by the given factor or resizes to the maximum size specified by the tuple. It uses bilinear interpolation by default. Lazy operation can be determined if lazy argument is set to True.",
+ "type": "comment"
+ },
+ "5174": {
+ "file_id": 437,
+ "content": " \"\"\"\n def __init__(self,\n scale,\n keep_ratio=True,\n interpolation='bilinear',\n lazy=False):\n if isinstance(scale, str):\n scale = eval(scale)\n if isinstance(scale, float):\n if scale <= 0:\n raise ValueError(f'Invalid scale {scale}, must be positive.')\n elif isinstance(scale, tuple):\n max_long_edge = max(scale)\n max_short_edge = min(scale)\n if max_short_edge == -1:\n # assign np.inf to long edge for rescaling short edge later.\n scale = (np.inf, max_long_edge)\n else:\n raise TypeError(\n f'Scale must be float or tuple of int, but got {type(scale)}')\n self.scale = scale\n self.keep_ratio = keep_ratio\n self.interpolation = interpolation\n self.lazy = lazy\n def __call__(self, results):\n \"\"\"Performs the Resize augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:304-334"
+ },
+ "5175": {
+ "file_id": 437,
+ "content": "Initializes a resize augmentation object with scale, keeps ratio (True or False), interpolation method ('bilinear' default), and lazy flag (False).",
+ "type": "comment"
+ },
+ "5176": {
+ "file_id": 437,
+ "content": " to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n if 'scale_factor' not in results:\n results['scale_factor'] = np.array([1, 1], dtype=np.float32)\n img_h, img_w = results['img_shape']\n if self.keep_ratio:\n new_w, new_h = rescale_size((img_w, img_h), self.scale)\n else:\n new_w, new_h = self.scale\n self.scale_factor = np.array([new_w / img_w, new_h / img_h],\n dtype=np.float32)\n results['img_shape'] = (new_h, new_w)\n results['keep_ratio'] = self.keep_ratio\n results['scale_factor'] = results['scale_factor'] * self.scale_factor\n if not self.lazy:\n if 'imgs' in results:\n results['imgs'] = [\n imresize(\n img, (new_w, new_h), interpolation=self.interpolation)\n for img in results['imgs']\n ]\n if 'keypoint' in results:\n results['keypoint'] = results['keypoint'] * self.scale_factor",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:335-363"
+ },
+ "5177": {
+ "file_id": 437,
+ "content": "This code resizes images and keypoints based on the 'scale_factor' or scale provided. If 'scale_factor' is not already in the results, it initializes it with a default value of [1, 1]. The code then calculates new image width (new_w) and height (new_h) based on the scale or keep_ratio setting. It updates 'img_shape', 'keep_ratio', and 'scale_factor' in the results dictionary, and if not lazy, it resizes images and keypoints accordingly using the imresize function.",
+ "type": "comment"
+ },
+ "5178": {
+ "file_id": 437,
+ "content": " else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Put Flip at last for now')\n lazyop['interpolation'] = self.interpolation\n #if 'gt_bboxes' in results:\n assert not self.lazy\n entity_box_rescale = EntityBoxRescale(self.scale_factor)\n results = entity_box_rescale(results)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'scale={self.scale}, keep_ratio={self.keep_ratio}, '\n f'interpolation={self.interpolation}, '\n f'lazy={self.lazy})')\n return repr_str\n@PIPELINES.register()\nclass RandomRescale:\n \"\"\"Randomly resize images so that the short_edge is resized to a specific\n size in a given range. The scale ratio is unchanged after resizing.\n \"\"\"\n def __init__(self, scale_range, interpolation='bilinear'):\n scale_range = eval(scale_range)\n self.scale_range = scale_range",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:364-393"
+ },
+ "5179": {
+ "file_id": 437,
+ "content": "This code defines a class RandomRescale that performs random rescaling on images, maintaining the aspect ratio. It takes in a range for the short edge size, and an optional interpolation method. The class also has a lazy attribute to control whether the transformation is applied lazily or not. The class also includes an EntityBoxRescale function to rescale bounding boxes. The code ends with registering RandomRescale as a pipeline module.",
+ "type": "comment"
+ },
+ "5180": {
+ "file_id": 437,
+ "content": " assert len(scale_range) == 2\n assert scale_range[0] < scale_range[1]\n assert np.all([x > 0 for x in scale_range])\n self.keep_ratio = True\n self.interpolation = interpolation\n def __call__(self, results):\n \"\"\"Performs the Resize augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n short_edge = np.random.randint(self.scale_range[0],\n self.scale_range[1] + 1)\n resize = Resize((-1, short_edge),\n keep_ratio=True,\n interpolation=self.interpolation,\n lazy=False)\n results = resize(results)\n results['short_edge'] = short_edge\n return results\n def __repr__(self):\n scale_range = self.scale_range\n repr_str = (f'{self.__class__.__name__}('\n f'scale_range=({scale_range[0]}, {scale_range[1]}), '",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:395-423"
+ },
+ "5181": {
+ "file_id": 437,
+ "content": "This code defines a Resize augmentation transform with random scaling range, keeps aspect ratio and applies specified interpolation. It also includes a __repr__ method to provide class name, scale range and short edge value for debugging purposes.",
+ "type": "comment"
+ },
+ "5182": {
+ "file_id": 437,
+ "content": " f'interpolation={self.interpolation})')\n return repr_str\n@PIPELINES.register()\nclass Rescale:\n \"\"\"resize images so that the short_edge is resized to a specific\n size in a given range. The scale ratio is unchanged after resizing.\n Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified\n keys are \"imgs\", \"img_shape\", \"keep_ratio\", \"scale_factor\", \"resize_size\",\n \"short_edge\".\n Args:\n scale_range (tuple[int]): The range of short edge length. A closed\n interval.\n interpolation (str): Algorithm used for interpolation:\n \"nearest\" | \"bilinear\". Default: \"bilinear\".\n \"\"\"\n def __init__(self, scale_range, interpolation='bilinear'):\n scale_range = eval(scale_range)\n self.scale_range = scale_range\n self.keep_ratio = True\n self.interpolation = interpolation\n def __call__(self, results):\n \"\"\"Performs the Resize augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:424-455"
+ },
+ "5183": {
+ "file_id": 437,
+ "content": "This code defines a Rescale augmentation class for image processing in the PaddleVideo framework. It resizes images so that the short edge length is within a specified range, while maintaining the aspect ratio. The interpolation method can be set to 'nearest' or 'bilinear'. This augmentation modifies the 'imgs', 'img_shape', 'keep_ratio', 'scale_factor', 'resize_size', and 'short_edge' keys in the results dictionary.",
+ "type": "comment"
+ },
+ "5184": {
+ "file_id": 437,
+ "content": " to the next transform in pipeline.\n \"\"\"\n resize = Resize(\n self.scale_range,\n keep_ratio=True,\n interpolation=self.interpolation,\n lazy=False)\n results = resize(results)\n return results\n def __repr__(self):\n scale_range = self.scale_range\n repr_str = (f'{self.__class__.__name__}('\n f'scale_range=({scale_range[0]}, {scale_range[1]}), '\n f'interpolation={self.interpolation})')\n return repr_str\n@PIPELINES.register()\nclass RandomCrop_v2:\n \"\"\"Vanilla square random crop that specifics the output size.\n Required keys in results are \"imgs\" and \"img_shape\", added or\n modified keys are \"imgs\", \"lazy\"; Required keys in \"lazy\" are \"flip\",\n \"crop_bbox\", added or modified key is \"crop_bbox\".\n Args:\n size (int): The output size of the images.\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n def __init__(self, size, lazy=False):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:456-487"
+ },
+ "5185": {
+ "file_id": 437,
+ "content": "This code defines a Resize transform and a RandomCrop_v2 class for image processing pipelines in PaddleVideo. The Resize transform scales images within a specified range and with optional interpolation, while the RandomCrop_v2 performs square random cropping on images to a specific output size.",
+ "type": "comment"
+ },
+ "5186": {
+ "file_id": 437,
+ "content": " if not isinstance(size, int):\n raise TypeError(f'Size must be an int, but got {type(size)}')\n self.size = size\n self.lazy = lazy\n def __call__(self, results):\n \"\"\"Performs the RandomCrop augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n img_h, img_w = results['img_shape']\n assert self.size <= img_h and self.size <= img_w\n y_offset = 0\n x_offset = 0\n if img_h > self.size:\n y_offset = int(np.random.randint(0, img_h - self.size))\n if img_w > self.size:\n x_offset = int(np.random.randint(0, img_w - self.size))\n if 'crop_quadruple' not in results:\n results['crop_quadruple'] = np.array(\n [0, 0, 1, 1], # x, y, w, h\n dtype=np.float32)\n x_ratio, y_ratio = x_offset / img_w, y_offset / img_h\n w_ratio, h_ratio = self.size / img_w, self.size / img_h",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:488-517"
+ },
+ "5187": {
+ "file_id": 437,
+ "content": "This code defines a class with an __init__ method that checks if the input 'size' is an integer, and a __call__ method to perform random cropping. The __call__ method takes a dictionary of results and performs random cropping based on the size attribute of the class instance. It asserts that the size is less than or equal to the image height and width, then randomly selects y and x offsets for cropping. If 'crop_quadruple' is not in the results dictionary, it adds a new entry with initial values. Finally, it calculates ratios for cropping based on the input size and image dimensions.",
+ "type": "comment"
+ },
+ "5188": {
+ "file_id": 437,
+ "content": " old_crop_quadruple = results['crop_quadruple']\n old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]\n old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]\n new_crop_quadruple = [\n old_x_ratio + x_ratio * old_w_ratio,\n old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,\n h_ratio * old_x_ratio\n ]\n results['crop_quadruple'] = np.array(\n new_crop_quadruple, dtype=np.float32)\n new_h, new_w = self.size, self.size\n results['crop_bbox'] = np.array(\n [x_offset, y_offset, x_offset + new_w, y_offset + new_h])\n results['img_shape'] = (new_h, new_w)\n if not self.lazy:\n results['imgs'] = [\n img[y_offset:y_offset + new_h, x_offset:x_offset + new_w]\n for img in results['imgs']\n ]\n else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Put Flip at last for now')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:519-544"
+ },
+ "5189": {
+ "file_id": 437,
+ "content": "This code segment is adjusting the crop quadruple, calculating a new crop bounding box, and updating the image shape based on provided offsets. It also handles lazy loading if enabled.",
+ "type": "comment"
+ },
+ "5190": {
+ "file_id": 437,
+ "content": " # record crop_bbox in lazyop dict to ensure only crop once in Fuse\n lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']\n left = x_offset * (lazy_right - lazy_left) / img_w\n right = (x_offset + new_w) * (lazy_right - lazy_left) / img_w\n top = y_offset * (lazy_bottom - lazy_top) / img_h\n bottom = (y_offset + new_h) * (lazy_bottom - lazy_top) / img_h\n lazyop['crop_bbox'] = np.array(\n [(lazy_left + left), (lazy_top + top), (lazy_left + right),\n (lazy_top + bottom)],\n dtype=np.float32)\n # Process entity boxes\n if 'gt_bboxes' in results:\n assert not self.lazy\n entity_box_crop = EntityBoxCrop(results['crop_bbox'])\n results = entity_box_crop(results)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}(size={self.size}, '\n f'lazy={self.lazy})')\n return repr_str\ndef imflip_(img, direction='horizontal'):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:546-571"
+ },
+ "5191": {
+ "file_id": 437,
+ "content": "This code section is responsible for applying augmentations to video frames, specifically crop and flip operations. It takes input parameters such as image size, whether it should be applied lazily or not, and the direction of flipping (if applicable). The code adjusts the crop region based on the specified offset values and stores them in the 'crop_bbox' field of the lazy operation dictionary. Additionally, if there are entity boxes present, they will also be processed according to the applied crop and flip operations.",
+ "type": "comment"
+ },
+ "5192": {
+ "file_id": 437,
+ "content": " \"\"\"Inplace flip an image horizontally or vertically.\n Args:\n img (ndarray): Image to be flipped.\n direction (str): The flip direction, either \"horizontal\" or\n \"vertical\" or \"diagonal\".\n Returns:\n ndarray: The flipped image (inplace).\n \"\"\"\n assert direction in ['horizontal', 'vertical', 'diagonal']\n if direction == 'horizontal':\n return cv2.flip(img, 1, img)\n elif direction == 'vertical':\n return cv2.flip(img, 0, img)\n else:\n return cv2.flip(img, -1, img)\ndef iminvert(img):\n \"\"\"Invert (negate) an image.\n Args:\n img (ndarray): Image to be inverted.\n Returns:\n ndarray: The inverted image.\n \"\"\"\n return np.full_like(img, 255) - img\n@PIPELINES.register()\nclass Flip:\n \"\"\"Flip the input images with a probability.\n Reverse the order of elements in the given imgs with a specific direction.\n The shape of the imgs is preserved, but the elements are reordered.\n Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:572-609"
+ },
+ "5193": {
+ "file_id": 437,
+ "content": "The provided code contains three functions: `inplace_flip`, `iminvert`, and a pipeline class called `Flip`. \n\n`inplace_flip` takes an image (`ndarray`) and the direction for flipping (horizontal, vertical or diagonal), asserts that the direction is valid, and returns the flipped image in-place. If the direction is horizontal, it uses `cv2.flip()` with parameter 1 to flip horizontally; if the direction is vertical, it uses `cv2.flip()` with parameter 0 to flip vertically; if the direction is diagonal, it uses `cv2.flip()` with parameter -1 for diagonal flipping.\n\n`iminvert` takes an image (`ndarray`) and returns its negative (inverted) version by subtracting the original image from a numpy array of full value 255 (the maximum possible value for an 8-bit image). This effectively reverses all pixel intensities in the image.\n\nThe `Flip` class is a pipeline module that flips the input images with a certain probability. It requires keys \"imgs\", \"img_shape\", and \"modality\" (although it does not modify them) and adds no new keys.",
+ "type": "comment"
+ },
+ "5194": {
+ "file_id": 437,
+ "content": " keys are \"imgs\", \"lazy\" and \"flip_direction\". Required keys in \"lazy\" is\n None, added or modified key are \"flip\" and \"flip_direction\". The Flip\n augmentation should be placed after any cropping / reshaping augmentations,\n to make sure crop_quadruple is calculated properly.\n Args:\n flip_ratio (float): Probability of implementing flip. Default: 0.5.\n direction (str): Flip imgs horizontally or vertically. Options are\n \"horizontal\" | \"vertical\". Default: \"horizontal\".\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n _directions = ['horizontal', 'vertical']\n def __init__(self, flip_ratio=0.5, direction='horizontal', lazy=False):\n if direction not in self._directions:\n raise ValueError(f'Direction {direction} is not supported. '\n f'Currently support ones are {self._directions}')\n self.flip_ratio = flip_ratio\n self.direction = direction\n self.lazy = lazy\n def __call__(self, results):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:610-631"
+ },
+ "5195": {
+ "file_id": 437,
+ "content": "This code defines a Flip augmentation class for image processing in PaddleVideo. It takes flip ratio, direction (horizontal or vertical), and lazy operation as parameters. The flip_ratio determines the probability of applying the flip transformation, while direction specifies whether to flip horizontally or vertically. If the 'lazy' parameter is True, the transformation will be applied lazily. This augmentation should be placed after cropping/reshaping transformations for proper crop_quadruple calculation.",
+ "type": "comment"
+ },
+ "5196": {
+ "file_id": 437,
+ "content": " \"\"\"Performs the Flip augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n flip = np.random.rand() < self.flip_ratio\n results['flip'] = flip\n results['flip_direction'] = self.direction\n if not self.lazy:\n if flip:\n for i, img in enumerate(results['imgs']):\n imflip_(img, self.direction)\n lt = len(results['imgs'])\n else:\n results['imgs'] = list(results['imgs'])\n else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Use one Flip please')\n lazyop['flip'] = flip\n lazyop['flip_direction'] = self.direction\n if 'gt_bboxes' in results and flip:\n assert not self.lazy and self.direction == 'horizontal'\n entity_box_flip = EntityBoxFlip(results['img_shape'])",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:632-660"
+ },
+ "5197": {
+ "file_id": 437,
+ "content": "The code snippet performs a Flip augmentation on images, randomly flipping them horizontally based on a given flip ratio. It also sets the 'flip' and 'flip_direction' keys in the results dictionary. If the 'lazy' option is not used (self.lazy), it iterates through the images, applying the flip transformation if necessary. It also handles 'gt_bboxes' if they exist in the results dictionary, ensuring horizontal flips are applied correctly without any issues.",
+ "type": "comment"
+ },
+ "5198": {
+ "file_id": 437,
+ "content": " results = entity_box_flip(results)\n return results\n def __repr__(self):\n repr_str = (\n f'{self.__class__.__name__}('\n f'flip_ratio={self.flip_ratio}, direction={self.direction}, '\n f'lazy={self.lazy})')\n return repr_str\ndef imnormalize_(img, mean, std, to_rgb=True):\n \"\"\"Inplace normalize an image with mean and std.\n Args:\n img (ndarray): Image to be normalized.\n mean (ndarray): The mean to be used for normalize.\n std (ndarray): The std to be used for normalize.\n to_rgb (bool): Whether to convert to rgb.\n Returns:\n ndarray: The normalized image.\n \"\"\"\n # cv2 inplace normalization does not accept uint8\n assert img.dtype != np.uint8\n mean = np.float64(mean.reshape(1, -1))\n stdinv = 1 / np.float64(std.reshape(1, -1))\n if to_rgb:\n cv2.cvtColor(img, cv2.COLOR_BGR2RGB, img) # inplace\n cv2.subtract(img, mean, img) # inplace\n cv2.multiply(img, stdinv, img) # inplace\n return img",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:661-693"
+ },
+ "5199": {
+ "file_id": 437,
+ "content": "This code contains a class for image augmentation, including flip ratio and direction, with a method to normalize an image using mean and std values. It also includes an inplace normalization function that converts BGR to RGB if necessary. The `__repr__` method returns a string representation of the class attributes.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/52.json b/docs/data/52.json
new file mode 100644
index 000000000..95c1f11cb
--- /dev/null
+++ b/docs/data/52.json
@@ -0,0 +1,545 @@
+{
+ "5200": {
+ "file_id": 437,
+ "content": "@PIPELINES.register()\nclass Normalize:\n \"\"\"Normalize images with the given mean and std value.\n Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified\n keys are \"imgs\" and \"img_norm_cfg\". If modality is 'Flow', additional\n keys \"scale_factor\" is required\n Args:\n mean (Sequence[float]): Mean values of different channels.\n std (Sequence[float]): Std values of different channels.\n to_bgr (bool): Whether to convert channels from RGB to BGR.\n Default: False.\n adjust_magnitude (bool): Indicate whether to adjust the flow magnitude\n on 'scale_factor' when modality is 'Flow'. Default: False.\n \"\"\"\n def __init__(self, mean, std, to_bgr=False, adjust_magnitude=False):\n if not isinstance(mean, Sequence):\n raise TypeError(\n f'Mean must be list, tuple or np.ndarray, but got {type(mean)}')\n if not isinstance(std, Sequence):\n raise TypeError(\n f'Std must be list, tuple or np.ndarray, but got {type(std)}')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:696-720"
+ },
+ "5201": {
+ "file_id": 437,
+ "content": "This code defines a class called \"Normalize\" that normalizes images based on given mean and std values. It can also convert channels from RGB to BGR if necessary. Additionally, it adjusts flow magnitude when modality is 'Flow' with an optional adjust_magnitude parameter. The class requires keys \"imgs\", \"img_shape\", \"modality\" with additional keys \"imgs\" and \"img_norm_cfg\" being added or modified.",
+ "type": "comment"
+ },
+ "5202": {
+ "file_id": 437,
+ "content": " self.mean = np.array(mean, dtype=np.float32)\n self.std = np.array(std, dtype=np.float32)\n self.to_bgr = to_bgr\n self.adjust_magnitude = adjust_magnitude\n def __call__(self, results):\n n = len(results['imgs'])\n h, w, c = results['imgs'][0].shape\n imgs = np.empty((n, h, w, c), dtype=np.float32)\n for i, img in enumerate(results['imgs']):\n imgs[i] = img\n for img in imgs:\n imnormalize_(img, self.mean, self.std, self.to_bgr)\n results['imgs'] = imgs\n results['img_norm_cfg'] = dict(\n mean=self.mean, std=self.std, to_bgr=self.to_bgr)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'mean={self.mean}, '\n f'std={self.std}, '\n f'to_bgr={self.to_bgr}, '\n f'adjust_magnitude={self.adjust_magnitude})')\n return repr_str",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/augmentations_ava.py:722-749"
+ },
+ "5203": {
+ "file_id": 437,
+ "content": "This code defines an augmentation pipeline for image normalization in AVA. It initializes mean, std, and to_bgr values, and then applies the normalization transformation to each input image. The normalized images are stored in 'imgs' and the configuration is saved in 'img_norm_cfg'. The __repr__ method provides a string representation of the object's state.",
+ "type": "comment"
+ },
+ "5204": {
+ "file_id": 438,
+ "content": "/paddlevideo/loader/pipelines/compose.py",
+ "type": "filepath"
+ },
+ "5205": {
+ "file_id": 438,
+ "content": "The Compose class combines registry-based pipeline components like decode functions, sample functions, and transforms to apply transformations flexibly on dictionary or list inputs. It includes a workaround for old format configuration files.",
+ "type": "summary"
+ },
+ "5206": {
+ "file_id": 438,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections.abc import Sequence\nfrom ..registry import PIPELINES\nimport traceback\nfrom ...utils import build\nfrom ...utils import get_logger\n@PIPELINES.register()\nclass Compose(object):\n \"\"\"\n Composes several pipelines(include decode func, sample func, and transforms) together.\n Note: To deal with ```list``` type cfg temporaray, like:\n transform:\n - Crop: # A list\n attribute: 10",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/loader/pipelines/compose.py:1-31"
+ },
+ "5207": {
+ "file_id": 438,
+ "content": "This code defines the Compose class, which composes multiple pipelines such as decode functions, sample functions, and transforms. It uses the PIPELINES registry for registration and builds pipelines based on input configurations. The code also handles temporary list-type configuration for flexibility.",
+ "type": "comment"
+ },
+ "5208": {
+ "file_id": 438,
+ "content": " - Resize: # A list\n attribute: 20\n every key of list will pass as the key name to build a module.\n XXX: will be improved in the future.\n Args:\n pipelines (list): List of transforms to compose.\n Returns:\n A compose object which is callable, __call__ for this Compose\n object will call each given :attr:`transforms` sequencely.\n \"\"\"\n def __init__(self, pipelines):\n #assert isinstance(pipelines, Sequence)\n self.pipelines = []\n for p in pipelines.values():\n if isinstance(p, dict):\n p = build(p, PIPELINES)\n self.pipelines.append(p)\n elif isinstance(p, list):\n for t in p:\n #XXX: to deal with old format cfg, ugly code here!\n temp_dict = dict(name=list(t.keys())[0])\n for all_sub_t in t.values():\n if all_sub_t is not None:\n temp_dict.update(all_sub_t) \n t = build(temp_dict, PIPELINES)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/loader/pipelines/compose.py:32-59"
+ },
+ "5209": {
+ "file_id": 438,
+ "content": "The code defines a Compose class that takes a list of transforms and composes them sequentially. It checks if the input is a dictionary or a list, builds the transform modules using build function from PIPELINES, and appends them to the pipelines list. The code also includes an ugly workaround for dealing with old format configuration files.",
+ "type": "comment"
+ },
+ "5210": {
+ "file_id": 438,
+ "content": " self.pipelines.append(t)\n elif callable(p):\n self.pipelines.append(p)\n else:\n raise TypeError(f'pipelines must be callable or a dict,'\n f'but got {type(p)}')\n def __call__(self, data):\n for p in self.pipelines:\n try:\n data = p(data)\n except Exception as e:\n stack_info = traceback.format_exc()\n logger = get_logger(\"paddlevideo\")\n logger.info(\"fail to perform transform [{}] with error: \"\n \"{} and stack:\\n{}\".format(p, e, str(stack_info)))\n raise e\n return data",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/loader/pipelines/compose.py:60-76"
+ },
+ "5211": {
+ "file_id": 438,
+ "content": "The code defines a class with a `__call__` method and an append function for adding pipelines. The `__call__` method applies transformations to input data by iterating over the pipelines. If any pipeline fails, it logs the error and raises an exception. Pipelines can be either callable or dictionaries, but if not, a TypeError is raised.",
+ "type": "comment"
+ },
+ "5212": {
+ "file_id": 439,
+ "content": "/paddlevideo/loader/pipelines/decode.py",
+ "type": "filepath"
+ },
+ "5213": {
+ "file_id": 439,
+ "content": "This code utilizes PaddleVideo's TimeSformer model for video processing, including a VideoDecoder class to decode mp4 files and handle varying durations. The \"ActionFeatureDecoder\" class handles feature decoding, while the function prepares data for model input and normalizes inputs for PaddlePaddle's video pipeline.",
+ "type": "summary"
+ },
+ "5214": {
+ "file_id": 439,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\ntry:\n import av\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [av] package and it's dependencies is required for TimeSformer and other models.\"\n )\nimport cv2\nimport pickle\nimport decord as de\nimport math\nimport random\nfrom ..registry import PIPELINES\ndef get_start_end_idx(video_size, clip_size, clip_idx, num_clips):\n delta = max(video_size - clip_size, 0)\n if clip_idx == -1: # here",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:1-32"
+ },
+ "5215": {
+ "file_id": 439,
+ "content": "This code snippet is part of the PaddleVideo library and it appears to import various packages, define a function \"get_start_end_idx\", and register something into the PIPELINES registry. It seems to handle video clip processing for TimeSformer and other models. The function calculates start and end indices for video clips based on video size, clip size, clip index, and the total number of clips.",
+ "type": "comment"
+ },
+ "5216": {
+ "file_id": 439,
+ "content": " # Random temporal sampling.\n start_idx = random.uniform(0, delta)\n else: # ignore\n # Uniformly sample the clip with the given index.\n start_idx = delta * clip_idx / num_clips\n end_idx = start_idx + clip_size - 1\n return start_idx, end_idx\n@PIPELINES.register()\nclass VideoDecoder(object):\n \"\"\"\n Decode mp4 file to frames.\n Args:\n filepath: the file path of mp4 file\n \"\"\"\n def __init__(self,\n backend='cv2',\n mode='train',\n sampling_rate=32,\n num_seg=8,\n num_clips=1,\n target_fps=30):\n self.backend = backend\n # params below only for TimeSformer\n self.mode = mode\n self.sampling_rate = sampling_rate\n self.num_seg = num_seg\n self.num_clips = num_clips\n self.target_fps = target_fps\n def __call__(self, results):\n \"\"\"\n Perform mp4 decode operations.\n return:\n List where each item is a numpy array after decoder.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:33-69"
+ },
+ "5217": {
+ "file_id": 439,
+ "content": "This code defines a VideoDecoder class for decoding mp4 files to frames. It takes the file path as input and has additional parameters for time-series applications like TimeSformer. The __call__ method performs the decoding operation, returning a list of numpy arrays representing the decoded frames.",
+ "type": "comment"
+ },
+ "5218": {
+ "file_id": 439,
+ "content": " \"\"\"\n file_path = results['filename']\n results['format'] = 'video'\n results['backend'] = self.backend\n if self.backend == 'cv2':\n cap = cv2.VideoCapture(file_path)\n videolen = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n sampledFrames = []\n for i in range(videolen):\n ret, frame = cap.read()\n # maybe first frame is empty\n if ret == False:\n continue\n img = frame[:, :, ::-1]\n sampledFrames.append(img)\n results['frames'] = sampledFrames\n results['frames_len'] = len(sampledFrames)\n elif self.backend == 'decord':\n container = de.VideoReader(file_path)\n frames_len = len(container)\n results['frames'] = container\n results['frames_len'] = frames_len\n elif self.backend == 'pyav': # for TimeSformer\n if self.mode in [\"train\", \"valid\"]:\n clip_idx = -1\n elif self.mode in [\"test\"]:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:70-98"
+ },
+ "5219": {
+ "file_id": 439,
+ "content": "This code is part of a video decoding pipeline. It checks the backend and decodes videos using either cv2, decord or pyav depending on the backend specified. It reads frames from the video and stores them in 'results' dictionary for further processing.",
+ "type": "comment"
+ },
+ "5220": {
+ "file_id": 439,
+ "content": " clip_idx = 0\n else:\n raise NotImplementedError\n container = av.open(file_path)\n num_clips = 1 # always be 1\n # decode process\n fps = float(container.streams.video[0].average_rate)\n frames_length = container.streams.video[0].frames\n duration = container.streams.video[0].duration\n if duration is None:\n # If failed to fetch the decoding information, decode the entire video.\n decode_all_video = True\n video_start_pts, video_end_pts = 0, math.inf\n else:\n decode_all_video = False\n start_idx, end_idx = get_start_end_idx(\n frames_length,\n self.sampling_rate * self.num_seg / self.target_fps * fps,\n clip_idx, num_clips)\n timebase = duration / frames_length\n video_start_pts = int(start_idx * timebase)\n video_end_pts = int(end_idx * timebase)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:99-125"
+ },
+ "5221": {
+ "file_id": 439,
+ "content": "This code checks if the duration of a video file is None. If it is, it sets decode_all_video to True and calculates video_start_pts and video_end_pts as 0 and infinity respectively. If the duration is not None, it calculates start and end indices for decoding specific clips from the video file.",
+ "type": "comment"
+ },
+ "5222": {
+ "file_id": 439,
+ "content": " frames = None\n # If video stream was found, fetch video frames from the video.\n if container.streams.video:\n margin = 1024\n seek_offset = max(video_start_pts - margin, 0)\n container.seek(seek_offset,\n any_frame=False,\n backward=True,\n stream=container.streams.video[0])\n tmp_frames = {}\n buffer_count = 0\n max_pts = 0\n for frame in container.decode(**{\"video\": 0}):\n max_pts = max(max_pts, frame.pts)\n if frame.pts < video_start_pts:\n continue\n if frame.pts <= video_end_pts:\n tmp_frames[frame.pts] = frame\n else:\n buffer_count += 1\n tmp_frames[frame.pts] = frame\n if buffer_count >= 0:\n break",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:127-150"
+ },
+ "5223": {
+ "file_id": 439,
+ "content": "This code snippet is part of a video decoding pipeline in PaddleVideo. It seeks to a specific start time of the video stream, then decodes and filters frames based on their start and end points. Frames before the start point are skipped, while frames after the end point are buffered. Finally, it stores the relevant frames in the \"tmp\\_frames\" dictionary.",
+ "type": "comment"
+ },
+ "5224": {
+ "file_id": 439,
+ "content": " video_frames = [tmp_frames[pts] for pts in sorted(tmp_frames)]\n container.close()\n frames = [frame.to_rgb().to_ndarray() for frame in video_frames]\n clip_sz = self.sampling_rate * self.num_seg / self.target_fps * fps\n start_idx, end_idx = get_start_end_idx(\n len(frames), # frame_len\n clip_sz,\n clip_idx if decode_all_video else\n 0, # If decode all video, -1 in train and valid, 0 in test;\n # else, always 0 in train, valid and test, as we has selected clip size frames when decode.\n 1)\n results['frames'] = frames\n results['frames_len'] = len(frames)\n results['start_idx'] = start_idx\n results['end_idx'] = end_idx\n else:\n raise NotImplementedError\n return results\n@PIPELINES.register()\nclass FrameDecoder(object):\n \"\"\"just parse results\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:151-177"
+ },
+ "5225": {
+ "file_id": 439,
+ "content": "This code extracts video frames, sorts them by timestamp, and then converts the frames to RGB format. It calculates the start and end indices for a given clip size based on the number of frames and the selected clip size. The results are stored in a dictionary along with additional information such as frame length and indices. If no code is provided for the \"else\" condition, a NotImplementedError will be raised. This class is registered as a pipeline using @PIPELINES.register().",
+ "type": "comment"
+ },
+ "5226": {
+ "file_id": 439,
+ "content": " def __init__(self):\n pass\n def __call__(self, results):\n results['format'] = 'frame'\n return results\n@PIPELINES.register()\nclass MRIDecoder(object):\n \"\"\"just parse results\n \"\"\"\n def __init__(self):\n pass\n def __call__(self, results):\n results['format'] = 'MRI'\n return results\n@PIPELINES.register()\nclass FeatureDecoder(object):\n \"\"\"\n Perform feature decode operations.e.g.youtube8m\n \"\"\"\n def __init__(self, num_classes, max_len=512, has_label=True):\n self.max_len = max_len\n self.num_classes = num_classes\n self.has_label = has_label\n def __call__(self, results):\n \"\"\"\n Perform feature decode operations.\n return:\n List where each item is a numpy array after decoder.\n \"\"\"\n #1. load pkl\n #2. parse to rgb/audio/\n #3. padding\n filepath = results['filename']\n data = pickle.load(open(filepath, 'rb'), encoding='bytes')\n record = data\n nframes = record['nframes'] if 'nframes' in record else record[",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:178-222"
+ },
+ "5227": {
+ "file_id": 439,
+ "content": "The code defines three pipeline classes for decoding different types of data. The MRIDecoder class sets the format to 'MRI'. The FeatureDecoder class initializes with parameters num_classes, max_len and has_label, then performs feature decode operations on loaded pkl files, parsing them into RGB/audio format, padding as necessary, and returning a list of numpy arrays.",
+ "type": "comment"
+ },
+ "5228": {
+ "file_id": 439,
+ "content": " b'nframes']\n rgb = record['feature'].astype(\n float) if 'feature' in record else record[b'feature'].astype(float)\n audio = record['audio'].astype(\n float) if 'audio' in record else record[b'audio'].astype(float)\n if self.has_label:\n label = record['label'] if 'label' in record else record[b'label']\n one_hot_label = self.make_one_hot(label, self.num_classes)\n rgb = rgb[0:nframes, :]\n audio = audio[0:nframes, :]\n rgb = self.dequantize(rgb,\n max_quantized_value=2.,\n min_quantized_value=-2.)\n audio = self.dequantize(audio,\n max_quantized_value=2,\n min_quantized_value=-2)\n if self.has_label:\n results['labels'] = one_hot_label.astype(\"float32\")\n feat_pad_list = []\n feat_len_list = []\n mask_list = []\n vitem = [rgb, audio]\n for vi in range(2): #rgb and audio",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:223-249"
+ },
+ "5229": {
+ "file_id": 439,
+ "content": "This code is preparing data for a model. It loads the 'feature' and 'audio' from the record if available, converts them to float type, and cuts them up to the specified number of frames (nframes). If labels are present in the record, it makes one-hot encoding out of them. The data is then dequantized using a method, and results are stored into the 'labels' variable. Finally, three lists (feat_pad_list, feat_len_list, mask_list) are initialized for further data processing. The code handles two types of data: 'feature' and 'audio', iterating over them in a range loop.",
+ "type": "comment"
+ },
+ "5230": {
+ "file_id": 439,
+ "content": " if vi == 0:\n prefix = \"rgb_\"\n else:\n prefix = \"audio_\"\n feat = vitem[vi]\n results[prefix + 'len'] = feat.shape[0]\n #feat pad step 1. padding\n feat_add = np.zeros((self.max_len - feat.shape[0], feat.shape[1]),\n dtype=np.float32)\n feat_pad = np.concatenate((feat, feat_add), axis=0)\n results[prefix + 'data'] = feat_pad.astype(\"float32\")\n #feat pad step 2. mask\n feat_mask_origin = np.ones(feat.shape, dtype=np.float32)\n feat_mask_add = feat_add\n feat_mask = np.concatenate((feat_mask_origin, feat_mask_add),\n axis=0)\n results[prefix + 'mask'] = feat_mask.astype(\"float32\")\n return results\n def dequantize(self,\n feat_vector,\n max_quantized_value=2.,\n min_quantized_value=-2.):\n \"\"\"\n Dequantize the feature from the byte format to the float format",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:250-275"
+ },
+ "5231": {
+ "file_id": 439,
+ "content": "This function pads and dequantizes video features for model input. It first checks the type of feature (video or audio) and prepends 'rgb_' or 'audio_' to the result keys accordingly. Then it pads the feature with zeros to match the max length, creates a mask for the padded feature, and dequantizes the feature from byte format to float format.",
+ "type": "comment"
+ },
+ "5232": {
+ "file_id": 439,
+ "content": " \"\"\"\n assert max_quantized_value > min_quantized_value\n quantized_range = max_quantized_value - min_quantized_value\n scalar = quantized_range / 255.0\n bias = (quantized_range / 512.0) + min_quantized_value\n return feat_vector * scalar + bias\n def make_one_hot(self, label, dim=3862):\n one_hot_label = np.zeros(dim)\n one_hot_label = one_hot_label.astype(float)\n for ind in label:\n one_hot_label[int(ind)] = 1\n return one_hot_label\n@PIPELINES.register()\nclass ActionFeatureDecoder(object):\n \"\"\"\n Perform feature decode operations on footballaction\n \"\"\"\n def __init__(self, num_classes, max_len=512, has_label=True):\n self.max_len = max_len\n self.num_classes = num_classes\n self.has_label = has_label\n def __call__(self, results):\n \"\"\"\n Perform feature decode operations.\n return:\n List where each item is a numpy array after decoder.\n \"\"\"\n #1. load pkl\n #2. parse to rgb/audio/",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:276-310"
+ },
+ "5233": {
+ "file_id": 439,
+ "content": "The code defines a class called \"ActionFeatureDecoder\" for feature decoding operations in football actions. It initializes with parameters for the maximum length, number of classes, and whether or not it should handle labels. The __call__ method performs the decoding operation on input results and returns a list of numpy arrays after decoding.",
+ "type": "comment"
+ },
+ "5234": {
+ "file_id": 439,
+ "content": " #3. padding\n filepath = results['filename']\n data = pickle.load(open(filepath, 'rb'), encoding='bytes')\n pkl_data = data\n rgb = pkl_data['image_feature'].astype(float)\n audio = pkl_data['audio_feature'].astype(float)\n label_id_info = pkl_data['label_info']\n label_cls = [label_id_info['label']]\n label_one = int(label_cls[0])\n if len(label_cls) > 1:\n label_index = random.randint(0, 1)\n label_one = int(label_cls[label_index])\n iou_norm = float(label_id_info['norm_iou'])\n results['labels'] = np.array([label_one])\n results['iou_norm'] = float(iou_norm)\n vitem = [rgb, audio]\n for vi in range(2): #rgb and audio\n if vi == 0:\n prefix = \"rgb_\"\n else:\n prefix = \"audio_\"\n feat = vitem[vi]\n results[prefix + 'len'] = feat.shape[0]\n #feat pad step 1. padding\n feat_add = np.zeros((self.max_len - feat.shape[0], feat.shape[1]),",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:311-338"
+ },
+ "5235": {
+ "file_id": 439,
+ "content": "The code is reading a pickle file, extracting the rgb image and audio features, label information, and performing some data manipulations. It sets the label to either 0 or 1 randomly if there's more than one in the data, normalizes iou values, and adds padding to the data for further processing. This is used in a video processing pipeline for PaddlePaddle.",
+ "type": "comment"
+ },
+ "5236": {
+ "file_id": 439,
+ "content": " dtype=np.float32)\n feat_pad = np.concatenate((feat, feat_add), axis=0)\n results[prefix + 'data'] = feat_pad.astype(\"float32\")\n #feat pad step 2. mask\n feat_mask_origin = np.ones(feat.shape, dtype=np.float32)\n feat_mask = np.concatenate((feat_mask_origin, feat_add), axis=0)\n results[prefix + 'mask'] = feat_mask.astype(\"float32\")\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode.py:339-347"
+ },
+ "5237": {
+ "file_id": 439,
+ "content": "This code pads the feature data and its corresponding mask for a PaddleVideo pipeline, concatenating them and casting the results to float32 type before storing in the 'results' dictionary.",
+ "type": "comment"
+ },
+ "5238": {
+ "file_id": 440,
+ "content": "/paddlevideo/loader/pipelines/decode_image.py",
+ "type": "filepath"
+ },
+ "5239": {
+ "file_id": 440,
+ "content": "The PaddleVideo class in PaddlePipelines uses PIL and skimage for image decoding operations. It accepts parameters such as scales, side map, and backend. This class can be used with datasets like KITTI and KITTI ODOM, supporting the retrieval of image paths and resizing of depth images. The code organizes results into a dictionary structure, processes image data based on 'train' or 'val', retrieves color images, adjusts intrinsics for depth estimation, stores results in the 'imgs' dictionary, and adds processed 'imgs' to 'results'.",
+ "type": "summary"
+ },
+ "5240": {
+ "file_id": 440,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport numpy as np\nimport PIL.Image as pil\ntry:\n import skimage.transform\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [scikit-image] package and it's dependencies is required for ADDS.\"\n )\nfrom PIL import Image\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass ImageDecoder(object):\n \"\"\"Decode Image\n \"\"\"\n def __init__(self,\n dataset,\n frame_idxs,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:1-37"
+ },
+ "5241": {
+ "file_id": 440,
+ "content": "This code is a Python class for decoding images, registered with the PADDLEPIPELINES module. It uses the PIL and skimage libraries, and is part of the PaddleVideo package in PaddlePaddle. The class takes in a dataset and frame_idxs as parameters for image decoding operations.",
+ "type": "comment"
+ },
+ "5242": {
+ "file_id": 440,
+ "content": " num_scales,\n side_map,\n full_res_shape,\n img_ext,\n backend='cv2'):\n self.backend = backend\n self.dataset = dataset\n self.frame_idxs = frame_idxs\n self.num_scales = num_scales\n self.side_map = side_map\n self.full_res_shape = full_res_shape\n self.img_ext = img_ext\n def _pil_loader(self, path):\n with open(path, 'rb') as f:\n with Image.open(f) as img:\n return img.convert('RGB')\n def get_color(self, folder, frame_index, side):\n color = self._pil_loader(\n self.get_image_path(self.dataset, folder, frame_index, side))\n return color\n def get_image_path(self, dataset, folder, frame_index, side):\n if dataset == \"kitti\":\n f_str = \"{:010d}{}\".format(frame_index, self.img_ext)\n image_path = os.path.join(self.data_path, folder, f_str)\n elif dataset == \"kitti_odom\":\n f_str = \"{:06d}{}\".format(frame_index, self.img_ext)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:38-66"
+ },
+ "5243": {
+ "file_id": 440,
+ "content": "This code defines a class for image decoding pipelines, accepting parameters such as number of scales, side map, full resolution shape, image extension, and backend. It also includes methods for loading images using the PIL library and retrieving image paths based on the dataset. The class is intended to be used for decoding images from specific datasets like KITTI and KITTI ODOM.",
+ "type": "comment"
+ },
+ "5244": {
+ "file_id": 440,
+ "content": " image_path = os.path.join(self.data_path,\n \"sequences/{:02d}\".format(int(folder)),\n \"image_{}\".format(self.side_map[side]),\n f_str)\n elif dataset == \"kitti_depth\":\n f_str = \"{:010d}{}\".format(frame_index, self.img_ext)\n image_path = os.path.join(\n self.data_path, folder,\n \"image_0{}/data\".format(self.side_map[side]), f_str)\n return image_path\n def get_depth(self, dataset, folder, frame_index, side):\n if dataset == \"kitii_depth\":\n f_str = \"{:010d}.png\".format(frame_index)\n depth_path = os.path.join(\n self.data_path, folder,\n \"proj_depth/groundtruth/image_0{}\".format(self.side_map[side]),\n f_str)\n depth_gt = pil.open(depth_path)\n depth_gt = depth_gt.resize(self.full_res_shape, pil.NEAREST)\n depth_gt = np.array(depth_gt).astype(np.float32) / 256",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:67-89"
+ },
+ "5245": {
+ "file_id": 440,
+ "content": "This code defines a class with two methods: `get_image_path` and `get_depth`. The first method returns the path of an image based on the dataset, folder, frame index, and side. If the dataset is \"kitti_depth\", it constructs the path using frame index and extension. The second method retrieves depth data for a given dataset, folder, frame index, and side. It uses the \"kitii_depth\" dataset, constructs the path to the depth file, opens the image, resizes it, and converts it into a float32 array divided by 256.",
+ "type": "comment"
+ },
+ "5246": {
+ "file_id": 440,
+ "content": " else:\n f_str = \"{:010d}{}\".format(frame_index, self.img_ext)\n depth_path = os.path.join(self.data_path, folder + '_gt', f_str)\n img_file = Image.open(depth_path)\n depth_png = np.array(img_file, dtype=int)\n img_file.close()\n # make sure we have a proper 16bit depth map here.. not 8bit!\n assert np.max(depth_png) > 255, \\\n \"np.max(depth_png)={}, path={}\".format(np.max(depth_png), depth_path)\n depth_gt = depth_png.astype(np.float) / 256.\n depth_gt = depth_gt[160:960 - 160, :]\n depth_gt = skimage.transform.resize(depth_gt,\n self.full_res_shape[::-1],\n order=0,\n preserve_range=True,\n mode='constant')\n return depth_gt\n def __call__(self, results):\n \"\"\"\n Perform mp4 decode operations.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:91-116"
+ },
+ "5247": {
+ "file_id": 440,
+ "content": "This function reads the depth image from file and resizes it to the desired shape, ensuring that it is a 16-bit depth map. It also checks if the maximum value exceeds 255, asserting that this is not an 8-bit image. The final output is a resized depth_gt with dimensions self.full_res_shape[::-1]. The function returns this resized depth_gt after performing any necessary operations for mp4 decode operations.",
+ "type": "comment"
+ },
+ "5248": {
+ "file_id": 440,
+ "content": " return:\n List where each item is a numpy array after decoder.\n \"\"\"\n if results.get('mode', None) == 'infer':\n imgs = {}\n imgs[(\"color\", 0,\n -1)] = Image.open(results[\"filename\"]).convert(\"RGB\")\n results['imgs'] = imgs\n return results\n self.data_path = results['data_path']\n results['backend'] = self.backend\n imgs = {}\n results['frame_idxs'] = self.frame_idxs\n results['num_scales'] = self.num_scales\n file_name = results['filename']\n folder = results['folder']\n frame_index = results['frame_index']\n line = file_name.split('/')\n istrain = folder.split('_')[1]\n if 'mode' not in results:\n results['mode'] = istrain\n results['day_or_night'] = folder.split('_')[0]\n if istrain == \"train\":\n if folder[0] == 'd':\n folder2 = folder + '_fake_night'\n flag = 0\n else:\n folder2 = folder + '_fake_day'",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:117-149"
+ },
+ "5249": {
+ "file_id": 440,
+ "content": "This code handles the decoding of images and organizes the results into a dictionary structure. It checks if the mode is set to 'infer', where it opens an image in RGB format, stores it in the results dictionary under 'imgs' key, and returns the results. If the mode is not set or is 'train', it sets up necessary variables for organizing data based on day or night folders and whether the folder is real or fake.",
+ "type": "comment"
+ },
+ "5250": {
+ "file_id": 440,
+ "content": " tmp = folder\n folder = folder2\n folder2 = tmp\n flag = 1\n if len(line) == 3:\n side = line[2]\n else:\n side = None\n results['side'] = side\n for i in self.frame_idxs:\n if i == \"s\":\n other_side = {\"r\": \"l\", \"l\": \"r\"}[side]\n imgs[(\"color\", i,\n -1)] = self.get_color(folder, frame_index, other_side)\n imgs[(\"color_n\", i,\n -1)] = self.get_color(folder2, frame_index,\n other_side)\n else:\n imgs[(\"color\", i,\n -1)] = self.get_color(folder, frame_index + i, side)\n imgs[(\"color_n\", i,\n -1)] = self.get_color(folder2, frame_index + i, side)\n istrain = folder.split('_')[1]\n if istrain != 'train':\n if flag:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:150-179"
+ },
+ "5251": {
+ "file_id": 440,
+ "content": "This code is setting up image files for decoding from a given folder and folder2 based on the frame indexes. It also considers whether the images are for the left or right side, identified by 'r' and 'l'. The flag variable is used to check if there's a change in side. If the folder name does not contain 'train', it executes something else (not shown in this code snippet).",
+ "type": "comment"
+ },
+ "5252": {
+ "file_id": 440,
+ "content": " depth_gt = self.get_depth(folder2, frame_index, side)\n else:\n depth_gt = self.get_depth(folder, frame_index, side)\n imgs[\"depth_gt\"] = np.expand_dims(depth_gt, 0)\n elif istrain == 'val':\n if len(line) == 3:\n side = line[2]\n else:\n side = None\n for i in self.frame_idxs:\n if i == \"s\":\n other_side = {\"r\": \"l\", \"l\": \"r\"}[side]\n imgs[(\"color\", i,\n -1)] = self.get_color(folder, frame_index, other_side)\n else:\n imgs[(\"color\", i,\n -1)] = self.get_color(folder, frame_index + i, side)\n # adjusting intrinsics to match each scale in the pyramid\n depth_gt = self.get_depth(self.dataset, folder, frame_index, side)\n imgs[\"depth_gt\"] = np.expand_dims(depth_gt, 0)\n results['imgs'] = imgs\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_image.py:180-206"
+ },
+ "5253": {
+ "file_id": 440,
+ "content": "The code checks the 'train' or 'val' flag and processes image data accordingly. It retrieves color images based on 'frame_idxs', adjusts intrinsics for depth estimation, and stores results in 'imgs' dictionary. The processed 'imgs' is then added to 'results' before returning it.",
+ "type": "comment"
+ },
+ "5254": {
+ "file_id": 441,
+ "content": "/paddlevideo/loader/pipelines/decode_sampler.py",
+ "type": "filepath"
+ },
+ "5255": {
+ "file_id": 441,
+ "content": "The code imports libraries, defines a DecodeSampler class for video decoding, and initializes parameters. It then decodes video frames, clips the index, retrieves frames, converts them to images using PIL library, and stores the images in a list.",
+ "type": "summary"
+ },
+ "5256": {
+ "file_id": 441,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport numpy as np\nfrom PIL import Image\nimport decord as de\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass DecodeSampler(object):\n \"\"\"\n We use 'decord' for decode and sampling, which is faster than opencv.\n This is used in slowfast model.\n Args:\n num_frames(int): the number of frames we want to sample.\n sampling_rate(int): sampling rate for video data.\n target_fps(int): desired fps, default 30",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler.py:1-30"
+ },
+ "5257": {
+ "file_id": 441,
+ "content": "This code imports necessary libraries, defines the DecodeSampler class for faster decoding and sampling of video data using 'decord', and registers it with the PIPELINES registry. It is used in the slowfast model and takes arguments such as num_frames, sampling_rate, and target_fps.",
+ "type": "comment"
+ },
+ "5258": {
+ "file_id": 441,
+ "content": " test_mode(bool): whether test or train/valid. In slowfast, we use multicrop when test.\n \"\"\"\n def __init__(self,\n num_frames,\n sampling_rate,\n default_sampling_rate=2,\n target_fps=30,\n test_mode=False):\n self.num_frames = num_frames\n self.orig_sampling_rate = self.sampling_rate = sampling_rate\n self.default_sampling_rate = default_sampling_rate\n self.target_fps = target_fps\n self.test_mode = test_mode\n def get_start_end_idx(self, video_size, clip_size, clip_idx,\n temporal_num_clips):\n delta = max(video_size - clip_size, 0)\n if not self.test_mode:\n # Random temporal sampling.\n start_idx = random.uniform(0, delta)\n else:\n # Uniformly sample the clip with the given index.\n start_idx = delta * clip_idx / temporal_num_clips\n end_idx = start_idx + clip_size - 1\n return start_idx, end_idx",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler.py:31-55"
+ },
+ "5259": {
+ "file_id": 441,
+ "content": "This code initializes a class with parameters for sampling video frames, and determines the start and end indices for each clip based on test mode (random or uniform).",
+ "type": "comment"
+ },
+ "5260": {
+ "file_id": 441,
+ "content": " def __call__(self, results):\n \"\"\"\n Perform mp4 decode operations.\n return:\n List where each item is a numpy array after decoder.\n \"\"\"\n short_cycle_idx = results.get('short_cycle_idx')\n if short_cycle_idx:\n self.sampling_rate = random.randint(self.default_sampling_rate,\n self.orig_sampling_rate)\n filepath = results['filename']\n temporal_sample_index = results['temporal_sample_index']\n temporal_num_clips = results['temporal_num_clips']\n vr = de.VideoReader(filepath)\n videolen = len(vr)\n fps = vr.get_avg_fps()\n clip_size = self.num_frames * self.sampling_rate * fps / self.target_fps\n start_idx, end_idx = self.get_start_end_idx(videolen, clip_size,\n temporal_sample_index,\n temporal_num_clips)\n index = np.linspace(start_idx, end_idx, self.num_frames).astype(\"int64\")",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler.py:57-81"
+ },
+ "5261": {
+ "file_id": 441,
+ "content": "This function performs mp4 decode operations and returns a list of numpy arrays after decoding. It considers the short_cycle_idx to adjust the sampling rate, takes the filepath and temporal parameters from results, initializes a VideoReader object, calculates clip size, gets start and end indices for video clipping based on these values, and finally creates an index list for the decoded frames.",
+ "type": "comment"
+ },
+ "5262": {
+ "file_id": 441,
+ "content": " index = np.clip(index, 0, videolen)\n frames_select = vr.get_batch(index) #1 for buffer\n # dearray_to_img\n np_frames = frames_select.asnumpy()\n frames_select_list = []\n for i in range(np_frames.shape[0]):\n imgbuf = np_frames[i]\n frames_select_list.append(Image.fromarray(imgbuf, mode='RGB'))\n results['imgs'] = frames_select_list\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler.py:82-93"
+ },
+ "5263": {
+ "file_id": 441,
+ "content": "This code segment is responsible for decoding and preparing image frames from a video. It clips the index value to ensure it falls within the valid range, retrieves the corresponding batch of frames using get_batch function, converts these frames into an array, and then loops through the array to convert each frame into an image using the PIL library's Image.fromarray method. The resulting images are stored in a list which is then assigned to 'results'['imgs'] before the function returns the results.",
+ "type": "comment"
+ },
+ "5264": {
+ "file_id": 442,
+ "content": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py",
+ "type": "filepath"
+ },
+ "5265": {
+ "file_id": 442,
+ "content": "The SFMRI_DecodeSampler class is a tool that decodes and samples MRI frames, creating segments based on sampling indices and handling video length constraints. It calculates offsets for 's' and 'f' frame types, determines average durations per segment, and returns an object containing the frame indices.",
+ "type": "summary"
+ },
+ "5266": {
+ "file_id": 442,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport random\nimport numpy as np\nfrom PIL import Image\ntry:\n import SimpleITK as sitk\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [SimpleITK] package and it's dependencies is required for PP-Care.\"\n )\nimport cv2\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass SFMRI_DecodeSampler(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:1-36"
+ },
+ "5267": {
+ "file_id": 442,
+ "content": "This code snippet is a Python class for the SFMRI_DecodeSampler pipeline, which decodes and samples MRI frames. It uses PIL and SimpleITK packages to read images and relies on OpenCV for image processing. The class is registered in the PIPELINES module.",
+ "type": "comment"
+ },
+ "5268": {
+ "file_id": 442,
+ "content": " num_seg(int): number of segments.\n seg_len(int): number of sampled frames in each segment.\n valid_mode(bool): True or False.\n select_left: Whether to select the frame to the left in the middle when the sampling interval is even in the test mode.\n Returns:\n frames_idx: the index of sampled #frames.\n \"\"\"\n def __init__(self,\n num_seg,\n seg_len,\n valid_mode=False,\n select_left=False,\n dense_sample=False,\n linspace_sample=False):\n self.num_seg = num_seg\n self.seg_len = seg_len\n self.valid_mode = valid_mode\n self.select_left = select_left\n self.dense_sample = dense_sample\n self.linspace_sample = linspace_sample\n def _get(self, frames_idx_s, frames_idx_f, results):\n frame_dir = results['frame_dir']\n imgs_s = []\n imgs_f = []\n MRI = sitk.GetArrayFromImage(sitk.ReadImage(frame_dir))\n for idx in frames_idx_s:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:37-64"
+ },
+ "5269": {
+ "file_id": 442,
+ "content": "This code defines a class with methods for creating segments of frames from an MRI image. The constructor takes arguments for the number of segments, length of each segment, and optional parameters for sampling mode. It returns the indexes of sampled frames in each segment. The class also includes a method for getting images from the MRI and storing them.",
+ "type": "comment"
+ },
+ "5270": {
+ "file_id": 442,
+ "content": " item = MRI[idx]\n item = cv2.resize(item, (224, 224))\n imgs_s.append(item)\n for idx in frames_idx_f:\n item = MRI[idx]\n item = cv2.resize(item, (224, 224))\n imgs_f.append(item)\n results['imgs'] = [imgs_s, imgs_f]\n return results\n def __call__(self, results):\n \"\"\"\n Args:\n frames_len: length of frames.\n return:\n sampling id.\n \"\"\"\n frames_len = int(results['frames_len'])\n average_dur1 = int(frames_len / self.num_seg[0])\n average_dur2 = int(frames_len / self.num_seg[1])\n frames_idx_s = []\n frames_idx_f = []\n if self.linspace_sample:\n if 'start_idx' in results and 'end_idx' in results:\n offsets_s = np.linspace(results['start_idx'],\n results['end_idx'], self.num_seg[0])\n offsets_f = np.linspace(results['start_idx'],\n results['end_idx'], self.num_seg[1])",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:65-94"
+ },
+ "5271": {
+ "file_id": 442,
+ "content": "This code defines a class that takes in MRI data and returns resized images for sampling. It creates two lists, imgs_s and imgs_f, which contain the resized MRI frames. The results dictionary contains these lists under the 'imgs' key. The __call__ method calculates the average duration of each segment based on frames_len, and generates frame indices for each segment using linspace_sample. It does not return any value in this context.",
+ "type": "comment"
+ },
+ "5272": {
+ "file_id": 442,
+ "content": " else:\n offsets_s = np.linspace(0, frames_len - 1, self.num_seg[0])\n offsets_f = np.linspace(0, frames_len - 1, self.num_seg[1])\n offsets_s = np.clip(offsets_s, 0, frames_len - 1).astype(np.int64)\n offsets_f = np.clip(offsets_f, 0, frames_len - 1).astype(np.int64)\n frames_idx_s = list(offsets_s)\n frames_idx_f = list(offsets_f)\n return self._get(frames_idx_s, frames_idx_f, results)\n if not self.select_left:\n if self.dense_sample: # For ppTSM\n if not self.valid_mode: # train\n sample_pos = max(1, 1 + frames_len - 64)\n t_stride1 = 64 // self.num_seg[0]\n t_stride2 = 64 // self.num_seg[1]\n start_idx = 0 if sample_pos == 1 else np.random.randint(\n 0, sample_pos - 1)\n offsets_s = [(idx * t_stride1 + start_idx) % frames_len + 1\n for idx in range(self.num_seg[0])]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:95-115"
+ },
+ "5273": {
+ "file_id": 442,
+ "content": "This code segment handles sampling of frames for video decoding. It sets the offsets for sample positions based on the number of segments specified and ensures they are within the valid frame range. If `select_left` is not set, it further checks if `dense_sample` is enabled in dense sampling mode. For ppTSM, it selects a sample position and calculates the corresponding offsets for each segment using the given formulas.",
+ "type": "comment"
+ },
+ "5274": {
+ "file_id": 442,
+ "content": " offsets_f = [(idx * t_stride2 + start_idx) % frames_len + 1\n for idx in range(self.num_seg[1])]\n frames_idx_s = offsets_s\n frames_idx_f = offsets_f\n else:\n sample_pos = max(1, 1 + frames_len - 64)\n t_stride1 = 64 // self.num_seg[0]\n t_stride2 = 64 // self.num_seg[1]\n start_list = np.linspace(0,\n sample_pos - 1,\n num=10,\n dtype=int)\n offsets_s = []\n offsets_f = []\n for start_idx in start_list.tolist():\n offsets_s += [\n (idx * t_stride1 + start_idx) % frames_len + 1\n for idx in range(self.num_seg[0])\n ]\n for start_idx in start_list.tolist():",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:116-135"
+ },
+ "5275": {
+ "file_id": 442,
+ "content": "This code calculates the sampling indices for both spatial and frequency domains. It creates two lists, frames_idx_s and frames_idx_f, based on the number of segments in each dimension (self.num_seg[0] and self.num_seg[1]). If the video length is less than 64 frames, it sets a smaller sampling range for both domains. The code also includes a backup strategy that uses a list of starting points for sampling if the video length is longer but still shorter than 64 frames.",
+ "type": "comment"
+ },
+ "5276": {
+ "file_id": 442,
+ "content": " offsets_f += [\n (idx * t_stride2 + start_idx) % frames_len + 1\n for idx in range(self.num_seg[1])\n ]\n frames_idx_s = offsets_s\n frames_idx_f = offsets_f\n else:\n for i in range(self.num_seg[0]):\n idx = 0\n if not self.valid_mode:\n if average_dur1 >= self.seg_len:\n idx = random.randint(0, average_dur1 - self.seg_len)\n idx += i * average_dur1\n elif average_dur1 >= 1:\n idx += i * average_dur1\n else:\n idx = i\n else:\n if average_dur1 >= self.seg_len:\n idx = (average_dur1 - 1) // 2\n idx += i * average_dur1\n elif average_dur1 >= 1:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:136-157"
+ },
+ "5277": {
+ "file_id": 442,
+ "content": "This code calculates the offsets for segmenting frames and storing them in two lists, `frames_idx_s` and `frames_idx_f`. If `valid_mode` is set, it randomly selects the indices within the constraints of `average_dur1`, otherwise it uses sequential indexing. It also handles cases where `average_dur1` is less than 1 by setting the index to i.",
+ "type": "comment"
+ },
+ "5278": {
+ "file_id": 442,
+ "content": " idx += i * average_dur1\n else:\n idx = i\n for jj in range(idx, idx + self.seg_len):\n frames_idx_s.append(jj)\n for i in range(self.num_seg[1]):\n idx = 0\n if not self.valid_mode:\n if average_dur2 >= self.seg_len:\n idx = random.randint(0, average_dur2 - self.seg_len)\n idx += i * average_dur2\n elif average_dur2 >= 1:\n idx += i * average_dur2\n else:\n idx = i\n else:\n if average_dur2 >= self.seg_len:\n idx = (average_dur2 - 1) // 2\n idx += i * average_dur2\n elif average_dur2 >= 1:\n idx += i * average_dur2\n else:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:158-180"
+ },
+ "5279": {
+ "file_id": 442,
+ "content": "Code iterates over frames and segments, assigning frame indices based on valid mode and average durations. If valid mode is off, it determines the idx based on average duration 1 and 2, or if in valid mode, it sets the idx to half the remaining average duration 2 minus 1. Finally, it appends frame indices to frames_idx_s list for each segment length.",
+ "type": "comment"
+ },
+ "5280": {
+ "file_id": 442,
+ "content": " idx = i\n for jj in range(idx, idx + self.seg_len):\n frames_idx_f.append(jj)\n return self._get(frames_idx_s, frames_idx_f, results)\n else: # for TSM\n if not self.valid_mode:\n if average_dur2 > 0:\n offsets_s = np.multiply(list(range(\n self.num_seg[0])), average_dur1) + np.random.randint(\n average_dur1, size=self.num_seg[0])\n offsets_f = np.multiply(list(range(\n self.num_seg[1])), average_dur2) + np.random.randint(\n average_dur2, size=self.num_seg[1])\n elif frames_len > self.num_seg[1]:\n offsets_s = np.sort(\n np.random.randint(frames_len, size=self.num_seg[0]))\n offsets_f = np.sort(\n np.random.randint(frames_len, size=self.num_seg[1]))\n else:\n offsets_s = np.zeros(shape=(self.num_seg[0], ))",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:181-203"
+ },
+ "5281": {
+ "file_id": 442,
+ "content": "If not in valid mode, if average duration 2 > 0, generate offsets_s and offsets_f for TSM. If frames_len is greater than num_seg[1], randomly select offsets_s and offsets_f. Otherwise, set offsets_s to zeros.",
+ "type": "comment"
+ },
+ "5282": {
+ "file_id": 442,
+ "content": " offsets_f = np.zeros(shape=(self.num_seg[1], ))\n else:\n if frames_len > self.num_seg[1]:\n average_dur_float_s = frames_len / self.num_seg[0]\n offsets_s = np.array([\n int(average_dur_float_s / 2.0 + average_dur_float_s * x)\n for x in range(self.num_seg[0])\n ])\n average_dur_float_f = frames_len / self.num_seg[1]\n offsets_f = np.array([\n int(average_dur_float_f / 2.0 + average_dur_float_f * x)\n for x in range(self.num_seg[1])\n ])\n else:\n offsets_s = np.zeros(shape=(self.num_seg[0], ))\n offsets_f = np.zeros(shape=(self.num_seg[1], ))\n frames_idx_s = list(offsets_s)\n frames_idx_f = list(offsets_f)\n return self._get(frames_idx_s, frames_idx_f, results)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/decode_sampler_MRI.py:204-224"
+ },
+ "5283": {
+ "file_id": 442,
+ "content": "This code calculates the offsets for segmenting frames into 's' and 'f' types based on the number of segments specified. If the total number of frames is greater than the specified number of segments, it calculates the average duration per segment for both types ('s' and 'f'). It then creates arrays of frame indices for 's' and 'f' frames using these calculated offsets. Finally, it returns an object by calling a method '_get'.",
+ "type": "comment"
+ },
+ "5284": {
+ "file_id": 443,
+ "content": "/paddlevideo/loader/pipelines/mix.py",
+ "type": "filepath"
+ },
+ "5285": {
+ "file_id": 443,
+ "content": "The code introduces a VideoMix operator for data augmentation in image classification tasks, using mixup and cutmix operations with controllable parameters.",
+ "type": "summary"
+ },
+ "5286": {
+ "file_id": 443,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass Mixup(object):\n \"\"\"\n Mixup operator.\n Args:\n alpha(float): alpha value.\n \"\"\"\n def __init__(self, alpha=0.2):\n assert alpha > 0., \\\n 'parameter alpha[%f] should > 0.0' % (alpha)\n self.alpha = alpha\n def __call__(self, batch):\n imgs, labels = list(zip(*batch))\n imgs = np.array(imgs)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/mix.py:1-34"
+ },
+ "5287": {
+ "file_id": 443,
+ "content": "Mixup class implements a mixup operator for PaddleVideo. It takes an alpha value as input and ensures it is greater than 0. The __call__ method takes a batch of images and labels, combines them with random weights determined by the alpha value, and returns the mixed up image batch and label batch.",
+ "type": "comment"
+ },
+ "5288": {
+ "file_id": 443,
+ "content": " labels = np.array(labels)\n bs = len(batch)\n idx = np.random.permutation(bs)\n lam = np.random.beta(self.alpha, self.alpha)\n lams = np.array([lam] * bs, dtype=np.float32)\n imgs = lam * imgs + (1 - lam) * imgs[idx]\n return list(zip(imgs, labels, labels[idx], lams))\n@PIPELINES.register()\nclass Cutmix(object):\n \"\"\" Cutmix operator\n Args:\n alpha(float): alpha value.\n \"\"\"\n def __init__(self, alpha=0.2):\n assert alpha > 0., \\\n 'parameter alpha[%f] should > 0.0' % (alpha)\n self.alpha = alpha\n def rand_bbox(self, size, lam):\n \"\"\" rand_bbox \"\"\"\n w = size[2]\n h = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(w * cut_rat)\n cut_h = np.int(h * cut_rat)\n # uniform\n cx = np.random.randint(w)\n cy = np.random.randint(h)\n bbx1 = np.clip(cx - cut_w // 2, 0, w)\n bby1 = np.clip(cy - cut_h // 2, 0, h)\n bbx2 = np.clip(cx + cut_w // 2, 0, w)\n bby2 = np.clip(cy + cut_h // 2, 0, h)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/mix.py:35-70"
+ },
+ "5289": {
+ "file_id": 443,
+ "content": "The code defines a Cutmix class for a mixup operator. It takes an alpha value as input, and randomly generates new images by cutting out a part of the original image and pasting it on top of another image, with alpha value determining the ratio of the two. The function rand_bbox is used to determine the dimensions and location of the cutout box.",
+ "type": "comment"
+ },
+ "5290": {
+ "file_id": 443,
+ "content": " return bbx1, bby1, bbx2, bby2\n def __call__(self, batch):\n imgs, labels = list(zip(*batch))\n imgs = np.array(imgs)\n labels = np.array(labels)\n bs = len(batch)\n idx = np.random.permutation(bs)\n lam = np.random.beta(self.alpha, self.alpha)\n bbx1, bby1, bbx2, bby2 = self.rand_bbox(imgs.shape, lam)\n imgs[:, :, bbx1:bbx2, bby1:bby2] = imgs[idx, :, bbx1:bbx2, bby1:bby2]\n lam = 1 - (float(bbx2 - bbx1) * (bby2 - bby1) /\n (imgs.shape[-2] * imgs.shape[-1]))\n lams = np.array([lam] * bs, dtype=np.float32)\n return list(zip(imgs, labels, labels[idx], lams))\n@PIPELINES.register()\nclass VideoMix(object):\n \"\"\"\n VideoMix operator.\n Args:\n cutmix_prob(float): prob choose cutmix\n mixup_alpha(float): alpha for mixup aug\n cutmix_alpha(float): alpha for cutmix aug\n \"\"\"\n def __init__(self, cutmix_prob=0.5, mixup_alpha=0.2, cutmix_alpha=1.0):\n assert cutmix_prob > 0., \\\n 'parameter cutmix_prob[%f] should > 0.0' % (cutmix_prob)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/mix.py:72-103"
+ },
+ "5291": {
+ "file_id": 443,
+ "content": "This code defines a VideoMix operator that performs data augmentation by either mixing or cutting images from different samples in the batch. The mixup_alpha and cutmix_alpha parameters control the degree of blending between samples, while the cutmix_prob parameter determines the probability of applying the cutmix operation.",
+ "type": "comment"
+ },
+ "5292": {
+ "file_id": 443,
+ "content": " assert mixup_alpha > 0., \\\n 'parameter mixup_alpha[%f] should > 0.0' % (mixup_alpha)\n assert cutmix_alpha > 0., \\\n 'parameter cutmix_alpha[%f] should > 0.0' % (cutmix_alpha)\n self.cutmix_prob = cutmix_prob\n self.mixup = Mixup(mixup_alpha)\n self.cutmix = Cutmix(cutmix_alpha)\n def __call__(self, batch):\n if np.random.random() < self.cutmix_prob:\n return self.cutmix(batch)\n else:\n return self.mixup(batch)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/mix.py:104-116"
+ },
+ "5293": {
+ "file_id": 443,
+ "content": "This code asserts that mixup_alpha and cutmix_alpha are greater than 0.0, sets the cutmix_prob, creates Mixup and Cutmix objects with the provided alphas, and defines a __call__ method to randomly choose between applying either Mixup or Cutmix to the batch.",
+ "type": "comment"
+ },
+ "5294": {
+ "file_id": 444,
+ "content": "/paddlevideo/loader/pipelines/multimodal.py",
+ "type": "filepath"
+ },
+ "5295": {
+ "file_id": 444,
+ "content": "The code introduces a new \"FeaturePadding\" class to PaddlePaddle library, handles data preprocessing for multimodal tasks, and provides masking, region selection, and action perturbation functions for PaddleVideo.",
+ "type": "summary"
+ },
+ "5296": {
+ "file_id": 444,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport numpy as np\nfrom PIL import Image\nimport decord as de\nimport copy\nimport json\nfrom ..registry import PIPELINES\ntry:\n from paddlenlp.transformers import BertTokenizer\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [paddlenlp] package and it's dependencies is required for ActBERT.\"\n )\n@PIPELINES.register()\nclass FeaturePadding(object):\n \"\"\"\n Padding feature to target shape.\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:1-35"
+ },
+ "5297": {
+ "file_id": 444,
+ "content": "This code is part of a PaddlePaddle video analysis library. It registers a new class called \"FeaturePadding\" which performs feature padding to target shape. It imports necessary libraries and packages including decord, PIL, numpy, json, paddlenlp for ActBERT, and the PIPELINES registry.",
+ "type": "comment"
+ },
+ "5298": {
+ "file_id": 444,
+ "content": " def __init__(self, max_region_num=36, max_action_num=5):\n self.max_region_num = max_region_num\n self.max_action_num = max_action_num\n def __call__(self, results):\n \"\"\"\n Padding feature.\n \"\"\"\n pack_feature = results['feature']\n tokenizer = results['tokenizer']\n image_feature_wp, image_target_wp, image_location_wp, \\\n num_boxes, image_h, image_w, image_id, caption, \\\n action_feature_wp, action_target_wp, num_actions = pack_feature\n image_feature = np.zeros((self.max_region_num, 2048), dtype=np.float32)\n image_target = np.zeros((self.max_region_num, 1601), dtype=np.float32)\n image_location = np.zeros((self.max_region_num, 5), dtype=np.float32)\n action_feature = np.zeros((self.max_action_num, 2048), dtype=np.float32)\n action_target = np.zeros((self.max_action_num, ), dtype=np.int64)\n num_boxes = int(num_boxes)\n image_feature[:num_boxes] = image_feature_wp\n image_target[:num_boxes] = image_target_wp",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:36-59"
+ },
+ "5299": {
+ "file_id": 444,
+ "content": "This code defines a class with an __init__ method and a __call__ method. The __init__ method initializes the maximum number of regions (36) and actions (5). The __call__ method takes in results as input, including feature packs for image and action data. It pads the features to their maximum allowed dimensions with zeroes if there are less than the specified maximum. This is useful for maintaining consistent input sizes in machine learning models.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/53.json b/docs/data/53.json
new file mode 100644
index 000000000..97f3120c2
--- /dev/null
+++ b/docs/data/53.json
@@ -0,0 +1,548 @@
+{
+ "5300": {
+ "file_id": 444,
+ "content": " image_location[:num_boxes, :4] = image_location_wp\n image_location[:, 4] = (image_location[:, 3] - image_location[:, 1]) * (\n image_location[:, 2] - image_location[:, 0]) / (float(image_w) *\n float(image_h))\n image_location[:, 0] = image_location[:, 0] / float(image_w)\n image_location[:, 1] = image_location[:, 1] / float(image_h)\n image_location[:, 2] = image_location[:, 2] / float(image_w)\n image_location[:, 3] = image_location[:, 3] / float(image_h)\n image_feature = copy.deepcopy(image_feature)\n image_target = copy.deepcopy(image_target)\n num_actions = int(num_actions)\n action_feature[:num_actions] = action_feature_wp\n action_target[:num_actions] = action_target_wp\n action_feature = copy.deepcopy(action_feature)\n action_target = copy.deepcopy(action_target)\n results = dict(image_feat=image_feature,\n image_target=image_target,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:60-81"
+ },
+ "5301": {
+ "file_id": 444,
+ "content": "This code segment is responsible for resizing and normalizing the image and action feature coordinates, as well as deep copying the features. It also initializes the results dictionary with keys for image_feat and image_target. This appears to be part of a data preprocessing step in a multimodal pipeline.",
+ "type": "comment"
+ },
+ "5302": {
+ "file_id": 444,
+ "content": " caption=caption,\n image_loc=image_location,\n num_boxes=int(num_boxes),\n action_feat=action_feature,\n action_target=action_target,\n num_actions=int(num_actions),\n tokenizer=tokenizer)\n return results\n@PIPELINES.register()\nclass RandomCap(object):\n def __init__(self, caption_path):\n \"\"\"\n Random Caption for NSP task\n \"\"\"\n self.caption_path = caption_path\n def select_caption(self, caption):\n captions = caption.split('!')\n rind = random.randint(0, len(captions) - 1)\n caption = captions[rind]\n return caption\n def get_random_caption(self, all_captions):\n num_caps = len(all_captions)\n rand_doc_idx = random.randint(0, num_caps - 1)\n caption = all_captions[rand_doc_idx]\n caption = self.select_caption(caption)\n return caption\n def random_cap(self, caption, all_captions):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:82-113"
+ },
+ "5303": {
+ "file_id": 444,
+ "content": "The code defines a pipeline that randomly selects captions for the NSP task. It takes caption paths as input and returns random captions. The class has an `__init__` method to initialize the caption path, a `select_caption` method to randomly choose one from multiple captions, a `get_random_caption` method to select a random caption from all provided captions, and finally a `random_cap` method that combines these functionalities.",
+ "type": "comment"
+ },
+ "5304": {
+ "file_id": 444,
+ "content": " if random.random() > 0.5:\n label = 0\n else:\n caption = self.get_random_caption(all_captions)\n label = 1\n return caption, label\n def __call__(self, results):\n caption = results['caption']\n all_captions = list(json.load(open(self.caption_path, 'r')))\n caption = self.select_caption(caption)\n caption, label = self.random_cap(caption, all_captions)\n results['caption'] = caption\n results['is_next'] = label\n return results\n@PIPELINES.register()\nclass Tokenize(object):\n def __init__(self, ):\n \"\"\"\n Tokenize caption\n \"\"\"\n pass\n def __call__(self, results):\n caption = results['caption']\n tokenizer = results['tokenizer']\n tokens_caption = tokenizer.tokenize(caption)\n results['caption'] = tokens_caption\n return results\n@PIPELINES.register()\nclass RandomMask(object):\n def __init__(self,\n max_seq_length=36,\n max_action_length=5,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:114-151"
+ },
+ "5305": {
+ "file_id": 444,
+ "content": "The code is part of a multi-modal pipeline, where it randomly generates labels (0 or 1) and selects captions from a list. It also includes classes for tokenizing captions and applying random masks on the text data.",
+ "type": "comment"
+ },
+ "5306": {
+ "file_id": 444,
+ "content": " max_region_length=36):\n self.max_seq_length = max_seq_length\n self.max_action_length = max_action_length\n self.max_region_length = max_region_length\n def get_image_global_feature(self, image_feat, image_loc, image_mask):\n g_image_feat = np.sum(image_feat, axis=0) / np.sum(\n image_mask, axis=0, keepdims=True)\n image_feat = np.concatenate(\n [np.expand_dims(g_image_feat, axis=0), image_feat],\n axis=0).astype(\"float32\")\n g_image_loc = np.array([0, 0, 1, 1, 1]).astype(\"float32\")\n image_loc = np.concatenate(\n [np.expand_dims(g_image_loc, axis=0), image_loc], axis=0)\n g_image_mask = np.array([1])\n image_mask = np.concatenate([g_image_mask, image_mask], axis=0)\n return image_feat, image_loc, image_mask\n def _truncate_seq_pair(self, tokens_b, max_length):\n \"\"\"Truncates a sequence pair in place to the maximum length.\n This is a simple heuristic which will always truncate the longer sequence",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:152-175"
+ },
+ "5307": {
+ "file_id": 444,
+ "content": "This code defines a class for loading multimodal data, including images and text, into TensorFlow datasets. The constructor takes the maximum sequence length, action length, and region length as arguments. It also includes functions to generate global image features and truncate a sequence pair if they exceed the maximum length.",
+ "type": "comment"
+ },
+ "5308": {
+ "file_id": 444,
+ "content": " one token at a time. This makes more sense than truncating an equal percent\n of tokens from each, since if one sequence is very short then each token\n that's truncated likely contains more information than a longer sequence.\n \"\"\"\n while True:\n total_length = len(tokens_b)\n if total_length <= max_length:\n break\n tokens_b.pop()\n def random_word(self, tokens, tokenizer):\n \"\"\"\n Masking some random tokens for Language Model task with probabilities as in the original BERT paper.\n Args:\n tokens: list of str, tokenized sentence.\n tokenizer: Tokenizer, object used for tokenization (we need it's vocab here)\n Return:\n (list of str, list of int), masked tokens and related labels for LM prediction\n \"\"\"\n output_label = []\n for i, token in enumerate(tokens):\n prob = random.random()\n # mask token with 15% probability\n if prob < 0.15:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:176-201"
+ },
+ "5309": {
+ "file_id": 444,
+ "content": "The code is implementing a method to mask random tokens in a sentence for Language Model (LM) tasks. It first ensures that all sequences have equal length by truncating one token at a time from the longer sequence, then randomly masks 15% of the tokens in each sequence. The method also includes logic to handle tokenizer and produces masked tokens along with their related labels for LM prediction.",
+ "type": "comment"
+ },
+ "5310": {
+ "file_id": 444,
+ "content": " prob /= 0.15\n # 80% randomly change token to mask token\n if prob < 0.8:\n tokens[i] = \"[MASK]\"\n # 10% randomly change token to random token\n elif prob < 0.9:\n #tok = random.choice(list(tokenizer.vocab.items()))[0]\n tok = tokenizer.vocab.idx_to_token[random.randint(\n 0,\n tokenizer.vocab_size,\n )]\n tokens[i] = tok\n # rest 10% randomly keep current token\n # append current token to output (we will predict these later)\n try:\n output_label.append(tokenizer.vocab[token])\n except KeyError:\n # For unknown words (should not occur with BPE vocab)\n output_label.append(tokenizer.vocab[\"[UNK]\"])\n print(\n \"Cannot find token '{}' in vocab. Using [UNK] insetad\".",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:202-225"
+ },
+ "5311": {
+ "file_id": 444,
+ "content": "This code modifies tokens in a given input sequence by randomly replacing them with mask tokens, random tokens from the vocabulary, or keeping them unchanged. The probability of each action is controlled by a variable 'prob', which is normalized to ensure the total probability sums up to 1.0. The resulting modified sequence is appended to 'output_label' for further prediction. Additionally, it handles unknown words by replacing them with '[UNK]'.",
+ "type": "comment"
+ },
+ "5312": {
+ "file_id": 444,
+ "content": " format(token))\n else:\n # no masking token (will be ignored by loss function later)\n output_label.append(-1)\n return tokens, output_label\n def random_region(self, image_feat, image_loc, num_boxes):\n output_label = []\n for i in range(num_boxes):\n prob = random.random()\n # mask token with 15% probability\n if prob < 0.15:\n prob /= 0.15\n # 80% randomly change token to mask token\n if prob < 0.9:\n image_feat[i] = 0\n # rest 20% randomly keep current token\n # append current token to output (we will predict these later)\n output_label.append(1)\n else:\n # no masking token (will be ignored by loss function later)\n output_label.append(-1)\n return image_feat, image_loc, output_label\n def random_action(self, action_feat, action_target, num_actions):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:226-255"
+ },
+ "5313": {
+ "file_id": 444,
+ "content": "The code defines three functions: \"random_region\", \"mask_token\", and \"random_action\". These functions are responsible for randomly masking tokens, selecting a random region from an image feature map, and randomly perturbing action features respectively. The random_region function masks 15% of the tokens in the input, while the random_action function perturbs 20% of the action features.",
+ "type": "comment"
+ },
+ "5314": {
+ "file_id": 444,
+ "content": " output_label = []\n for i in range(num_actions):\n prob = random.random()\n # mask token with 15% probability\n if prob < 0.15:\n prob /= 0.15\n # 90% randomly change token to mask token\n if prob < 0.9:\n action_feat[i] = 0\n # rest 10% randomly keep current token\n # append current token to output (we will predict these later)\n output_label.append(action_target[i])\n else:\n # no masking token (will be ignored by loss function later)\n output_label.append(-1)\n return action_feat, output_label\n def __call__(self, results):\n caption = results['caption']\n tokenizer = results['tokenizer']\n image_feat = results['image_feat']\n image_loc = results['image_loc']\n num_boxes = results['num_boxes']\n action_feat = results['action_feat']\n action_target = results['action_target']\n num_actions = results['num_actions']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:256-285"
+ },
+ "5315": {
+ "file_id": 444,
+ "content": "This code defines a function that applies random masking to an input sequence of actions. It randomly chooses to either replace 90% of the tokens with mask tokens, keep them unchanged (10%), or ignore them for loss calculation by setting their value to -1. The function takes as input various results from a pipeline and returns the masked action features and labels.",
+ "type": "comment"
+ },
+ "5316": {
+ "file_id": 444,
+ "content": " is_next = results['is_next']\n image_target = results['image_target']\n self._truncate_seq_pair(caption, self.max_seq_length - 2)\n caption, caption_label = self.random_word(caption, tokenizer)\n image_feat, image_loc, image_label = self.random_region(\n image_feat, image_loc, num_boxes)\n action_feat, action_label = self.random_action(action_feat,\n action_target,\n num_actions)\n # concatenate lm labels and account for CLS, SEP, SEP\n lm_label_ids = [-1] + caption_label + [-1]\n # The convention in BERT is:\n # (a) For sequence pairs:\n # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1\n # (b) For single sequences:\n # tokens: [CLS] the dog is hairy . [SEP]\n # type_ids: 0 0 0 0 0 0 0\n #",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:286-308"
+ },
+ "5317": {
+ "file_id": 444,
+ "content": "This code is part of a multimodal pipeline that randomly selects words from the caption, regions from an image, and actions, then concatenates them using BERT's convention for sequence pairs. It also handles truncating the caption and assigning labels to the input features.",
+ "type": "comment"
+ },
+ "5318": {
+ "file_id": 444,
+ "content": " # Where \"type_ids\" are used to indicate whether this is the first\n # sequence or the second sequence. The embedding vectors for `type=0` and\n # `type=1` were learned during pre-training and are added to the wordpiece\n # embedding vector (and position vector). This is not *strictly* necessary\n # since the [SEP] token unambigiously separates the sequences, but it makes\n # it easier for the model to learn the concept of sequences.\n #\n # For classification tasks, the first vector (corresponding to [CLS]) is\n # used as as the \"sentence vector\". Note that this only makes sense because\n # the entire model is fine-tuned.\n tokens = []\n segment_ids = []\n tokens.append(\"[CLS]\")\n segment_ids.append(0)\n for token in caption:\n tokens.append(token)\n segment_ids.append(0)\n tokens.append(\"[SEP]\")\n segment_ids.append(0)\n input_ids = tokenizer.convert_tokens_to_ids(tokens)\n # The mask has 1 for real tokens and 0 for padding tokens. Only real tokens are attended to.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:309-333"
+ },
+ "5319": {
+ "file_id": 444,
+ "content": "This code prepares input data for a multimodal pipeline in PaddleVideo. It appends special tokens \"[CLS]\" and \"[SEP]\" to the token list, assigns segment ID 0 to all tokens (indicating first sequence), converts tokens to input IDs using tokenizer, and creates a mask with 1 for real tokens and 0 for padding tokens. This allows the model to learn sequences and use the [CLS] vector as a \"sentence vector\" for classification tasks.",
+ "type": "comment"
+ },
+ "5320": {
+ "file_id": 444,
+ "content": " input_mask = [1] * (len(input_ids))\n image_mask = [1] * (num_boxes)\n action_mask = [1] * (num_actions)\n # Zero-pad up to the visual sequence length.\n while len(image_mask) < self.max_region_length:\n image_mask.append(0)\n image_label.append(-1)\n while len(action_mask) < self.max_action_length:\n action_mask.append(0)\n action_label.append(-1)\n # Zero-pad up to the sequence length.\n while len(input_ids) < self.max_seq_length:\n input_ids.append(0)\n input_mask.append(0)\n segment_ids.append(0)\n lm_label_ids.append(-1)\n assert len(input_ids) == self.max_seq_length\n assert len(input_mask) == self.max_seq_length\n assert len(segment_ids) == self.max_seq_length\n assert len(lm_label_ids) == self.max_seq_length\n assert len(image_mask) == self.max_region_length\n assert len(image_label) == self.max_region_length\n assert len(action_mask) == self.max_action_length",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:334-359"
+ },
+ "5321": {
+ "file_id": 444,
+ "content": "Zero-padding visual, action, and input sequences to the maximum lengths. Asserting that all lists are of equal length after padding and match their respective max lengths.",
+ "type": "comment"
+ },
+ "5322": {
+ "file_id": 444,
+ "content": " assert len(action_label) == self.max_action_length\n image_feat, image_loc, image_mask = self.get_image_global_feature(\n image_feat, image_loc, np.array(image_mask))\n features = [\n np.array(input_ids),\n action_feat,\n image_feat,\n image_loc,\n np.array(segment_ids),\n np.array(input_mask),\n image_mask,\n np.array(action_mask),\n np.array(lm_label_ids),\n np.array(action_label),\n np.array(is_next),\n np.array(image_label),\n image_target,\n ]\n results['features'] = features\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/multimodal.py:360-380"
+ },
+ "5323": {
+ "file_id": 444,
+ "content": "This code snippet is part of a pipeline function that asserts the length of 'action_label' matches the maximum allowed action length. It then calls another function to get global image features, and forms a list of feature arrays including input ids, action feature, image feature, location, segment ids, input mask, image mask, action label, lm_label_ids, is_next, image label, and image target. The results dictionary is updated with these features before the function returns.",
+ "type": "comment"
+ },
+ "5324": {
+ "file_id": 445,
+ "content": "/paddlevideo/loader/pipelines/sample.py",
+ "type": "filepath"
+ },
+ "5325": {
+ "file_id": 445,
+ "content": "The code utilizes PaddleVideo's image processing pipeline for efficient frame sampling, defines a sampler class for video decoding and data conversion, and calculates sampling positions, offsets, and generates frame indices for video sequences.",
+ "type": "summary"
+ },
+ "5326": {
+ "file_id": 445,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport random\nimport numpy as np\nfrom PIL import Image\ntry:\n import SimpleITK as sitk\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [SimpleITK] package and it's dependencies is required for PP-Care.\"\n )\nimport cv2\nfrom ..registry import PIPELINES\ntry:\n import cPickle as pickle\n from cStringIO import StringIO\nexcept ImportError:\n import pickle\n from io import BytesIO\n@PIPELINES.register()",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:1-38"
+ },
+ "5327": {
+ "file_id": 445,
+ "content": "This code is a Python module that imports various libraries and defines an image processing pipeline for PaddleVideo. It checks if SimpleITK is installed, handles pickling, and registers the pipeline using PaddleVideo's registry.",
+ "type": "comment"
+ },
+ "5328": {
+ "file_id": 445,
+ "content": "class Sampler(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_seg(int): number of segments.\n seg_len(int): number of sampled frames in each segment.\n valid_mode(bool): True or False.\n select_left: Whether to select the frame to the left in the middle when the sampling interval is even in the test mode.\n Returns:\n frames_idx: the index of sampled #frames.\n \"\"\"\n def __init__(self,\n num_seg,\n seg_len,\n frame_interval=None,\n valid_mode=False,\n select_left=False,\n dense_sample=False,\n linspace_sample=False,\n use_pil=True):\n self.num_seg = num_seg\n self.seg_len = seg_len\n self.frame_interval = frame_interval\n self.valid_mode = valid_mode\n self.select_left = select_left\n self.dense_sample = dense_sample\n self.linspace_sample = linspace_sample",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:39-66"
+ },
+ "5329": {
+ "file_id": 445,
+ "content": "The `Sampler` class is used to sample frames based on various parameters such as number of segments, length of each segment, frame interval, valid mode, select left flag and whether to use PIL for reading images. It returns the index of sampled frames.",
+ "type": "comment"
+ },
+ "5330": {
+ "file_id": 445,
+ "content": " self.use_pil = use_pil\n def _get(self, frames_idx, results):\n data_format = results['format']\n if data_format == \"frame\":\n frame_dir = results['frame_dir']\n imgs = []\n for idx in frames_idx:\n img = Image.open(\n os.path.join(frame_dir,\n results['suffix'].format(idx))).convert('RGB')\n imgs.append(img)\n elif data_format == \"MRI\":\n frame_dir = results['frame_dir']\n imgs = []\n MRI = sitk.GetArrayFromImage(sitk.ReadImage(frame_dir))\n for idx in frames_idx:\n item = MRI[idx]\n item = cv2.resize(item, (224, 224))\n imgs.append(item)\n elif data_format == \"video\":\n if results['backend'] == 'cv2':\n frames = np.array(results['frames'])\n imgs = []\n for idx in frames_idx:\n imgbuf = frames[idx]\n img = Image.fromarray(imgbuf, mode='RGB')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:67-96"
+ },
+ "5331": {
+ "file_id": 445,
+ "content": "The code defines a class with an attribute 'use_pil' that determines the image format. The '_get' method retrieves frames based on data format (frame, MRI, or video), applies necessary conversions and resizing, and stores them in 'imgs'. It uses different libraries such as Image, sitk, and cv2 for different formats.",
+ "type": "comment"
+ },
+ "5332": {
+ "file_id": 445,
+ "content": " imgs.append(img)\n elif results['backend'] == 'decord':\n container = results['frames']\n if self.use_pil:\n frames_select = container.get_batch(frames_idx)\n # dearray_to_img\n np_frames = frames_select.asnumpy()\n imgs = []\n for i in range(np_frames.shape[0]):\n imgbuf = np_frames[i]\n imgs.append(Image.fromarray(imgbuf, mode='RGB'))\n else:\n if frames_idx.ndim != 1:\n frames_idx = np.squeeze(frames_idx)\n frame_dict = {\n idx: container[idx].asnumpy()\n for idx in np.unique(frames_idx)\n }\n imgs = [frame_dict[idx] for idx in frames_idx]\n elif results['backend'] == 'pyav':\n imgs = []\n frames = np.array(results['frames'])\n for idx in frames_idx:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:97-119"
+ },
+ "5333": {
+ "file_id": 445,
+ "content": "Code is handling video decoding using different backends such as 'opencv', 'decord', and 'pyav'. It appends the frames to imgs list, converts numpy array to image using Image.fromarray method for 'decord' backend, and handles frame indexing and data structures based on backend used.",
+ "type": "comment"
+ },
+ "5334": {
+ "file_id": 445,
+ "content": " if self.dense_sample:\n idx = idx - 1\n imgbuf = frames[idx]\n imgs.append(imgbuf)\n imgs = np.stack(imgs) # thwc\n else:\n raise NotImplementedError\n else:\n raise NotImplementedError\n results['imgs'] = imgs\n return results\n def _get_train_clips(self, num_frames):\n ori_seg_len = self.seg_len * self.frame_interval\n avg_interval = (num_frames - ori_seg_len + 1) // self.num_seg\n if avg_interval > 0:\n base_offsets = np.arange(self.num_seg) * avg_interval\n clip_offsets = base_offsets + np.random.randint(avg_interval,\n size=self.num_seg)\n elif num_frames > max(self.num_seg, ori_seg_len):\n clip_offsets = np.sort(\n np.random.randint(num_frames - ori_seg_len + 1,\n size=self.num_seg))\n elif avg_interval == 0:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:120-144"
+ },
+ "5335": {
+ "file_id": 445,
+ "content": "This code snippet is responsible for sampling frames from a video sequence, and it handles different scenarios based on the input parameters. If `dense_sample` is True, it adjusts the index before accessing the frame. The frames are then appended to a list called `imgs`. If neither of the else conditions are met, it raises a `NotImplementedError`. The function also includes another method, `_get_train_clips`, which calculates clip offsets for training purposes based on the number of frames and other parameters.",
+ "type": "comment"
+ },
+ "5336": {
+ "file_id": 445,
+ "content": " ratio = (num_frames - ori_seg_len + 1.0) / self.num_seg\n clip_offsets = np.around(np.arange(self.num_seg) * ratio)\n else:\n clip_offsets = np.zeros((self.num_seg, ), dtype=np.int)\n return clip_offsets\n def _get_test_clips(self, num_frames):\n ori_seg_len = self.seg_len * self.frame_interval\n avg_interval = (num_frames - ori_seg_len + 1) / float(self.num_seg)\n if num_frames > ori_seg_len - 1:\n base_offsets = np.arange(self.num_seg) * avg_interval\n clip_offsets = (base_offsets + avg_interval / 2.0).astype(np.int)\n else:\n clip_offsets = np.zeros((self.num_seg, ), dtype=np.int)\n return clip_offsets\n def __call__(self, results):\n \"\"\"\n Args:\n frames_len: length of frames.\n return:\n sampling id.\n \"\"\"\n frames_len = int(results['frames_len'])\n frames_idx = []\n if self.frame_interval is not None:\n assert isinstance(self.frame_interval, int)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:145-171"
+ },
+ "5337": {
+ "file_id": 445,
+ "content": "The code defines a class with methods to determine clip offsets based on the number of frames and segment length. If the number of frames exceeds the original segment length, it calculates clip offsets for each segment. Otherwise, it sets all clip offsets to zero. The class also has a __call__ method that takes frames length as input and returns sampling indices.",
+ "type": "comment"
+ },
+ "5338": {
+ "file_id": 445,
+ "content": " if not self.valid_mode:\n offsets = self._get_train_clips(frames_len)\n else:\n offsets = self._get_test_clips(frames_len)\n offsets = offsets[:, None] + np.arange(\n self.seg_len)[None, :] * self.frame_interval\n offsets = np.concatenate(offsets)\n offsets = offsets.reshape((-1, self.seg_len))\n offsets = np.mod(offsets, frames_len)\n offsets = np.concatenate(offsets)\n if results['format'] == 'video':\n frames_idx = offsets\n elif results['format'] == 'frame':\n frames_idx = list(offsets + 1)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)\n if self.linspace_sample:\n if 'start_idx' in results and 'end_idx' in results:\n offsets = np.linspace(results['start_idx'], results['end_idx'],\n self.num_seg)\n else:\n offsets = np.linspace(0, frames_len - 1, self.num_seg)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:172-199"
+ },
+ "5339": {
+ "file_id": 445,
+ "content": "This code determines the sampling method for frames based on the mode (valid or train) and format ('video' or 'frame'). It calculates offsets, handles different formats, and if linspace_sample is True, it generates offsets using linear spacing.",
+ "type": "comment"
+ },
+ "5340": {
+ "file_id": 445,
+ "content": " offsets = np.clip(offsets, 0, frames_len - 1).astype(np.int64)\n if results['format'] == 'video':\n frames_idx = list(offsets)\n frames_idx = [x % frames_len for x in frames_idx]\n elif results['format'] == 'frame':\n frames_idx = list(offsets + 1)\n elif results['format'] == 'MRI':\n frames_idx = list(offsets)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)\n average_dur = int(frames_len / self.num_seg)\n if not self.select_left:\n if self.dense_sample: # For ppTSM\n if not self.valid_mode: # train\n sample_pos = max(1, 1 + frames_len - 64)\n t_stride = 64 // self.num_seg\n start_idx = 0 if sample_pos == 1 else np.random.randint(\n 0, sample_pos - 1)\n offsets = [(idx * t_stride + start_idx) % frames_len + 1\n for idx in range(self.num_seg)]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:200-223"
+ },
+ "5341": {
+ "file_id": 445,
+ "content": "This code segment calculates the frames to sample from a video, based on its format (video/frame/MRI). It also handles dense sampling for ppTSM. In non-dense mode, it selects random positions for each segment within the range of 1 to frames_len. For dense sampling in train mode, it generates a set of evenly spaced frame indices between start_idx and sample_pos, which is calculated based on frames_len and 64 (to ensure at least one frame within the window). The offsets are then used to fetch corresponding data using the _get method.",
+ "type": "comment"
+ },
+ "5342": {
+ "file_id": 445,
+ "content": " frames_idx = offsets\n else:\n sample_pos = max(1, 1 + frames_len - 64)\n t_stride = 64 // self.num_seg\n start_list = np.linspace(0,\n sample_pos - 1,\n num=10,\n dtype=int)\n offsets = []\n for start_idx in start_list.tolist():\n offsets += [\n (idx * t_stride + start_idx) % frames_len + 1\n for idx in range(self.num_seg)\n ]\n frames_idx = offsets\n else:\n for i in range(self.num_seg):\n idx = 0\n if not self.valid_mode:\n if average_dur >= self.seg_len:\n idx = random.randint(0, average_dur - self.seg_len)\n idx += i * average_dur",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:224-245"
+ },
+ "5343": {
+ "file_id": 445,
+ "content": "The code determines the sampling position based on frames length, number of segments, and valid mode. If no offsets are provided, it calculates the starting positions for each segment using a linear space. Then, it generates the offsets by multiplying the stride with the current segment index and adding the start index. Finally, if in valid mode, it randomly selects indices within the average duration per segment and adds them to the offsets list.",
+ "type": "comment"
+ },
+ "5344": {
+ "file_id": 445,
+ "content": " elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n else:\n if average_dur >= self.seg_len:\n idx = (average_dur - 1) // 2\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n for jj in range(idx, idx + self.seg_len):\n if results['format'] == 'video':\n frames_idx.append(int(jj % frames_len))\n elif results['format'] == 'frame':\n frames_idx.append(jj + 1)\n elif results['format'] == 'MRI':\n frames_idx.append(jj)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:246-268"
+ },
+ "5345": {
+ "file_id": 445,
+ "content": "Code calculates index based on average duration, then appends corresponding frame indices to frames_idx list based on the format specified in results. If the format is not recognized, it raises NotImplementedError. Finally, it returns the frames_idx and results to an unknown method.",
+ "type": "comment"
+ },
+ "5346": {
+ "file_id": 445,
+ "content": " else: # for TSM\n if not self.valid_mode:\n if average_dur > 0:\n offsets = np.multiply(list(range(self.num_seg)),\n average_dur) + np.random.randint(\n average_dur, size=self.num_seg)\n elif frames_len > self.num_seg:\n offsets = np.sort(\n np.random.randint(frames_len, size=self.num_seg))\n else:\n offsets = np.zeros(shape=(self.num_seg, ))\n else:\n if frames_len > self.num_seg:\n average_dur_float = frames_len / self.num_seg\n offsets = np.array([\n int(average_dur_float / 2.0 + average_dur_float * x)\n for x in range(self.num_seg)\n ])\n else:\n offsets = np.zeros(shape=(self.num_seg, ))\n if results['format'] == 'video':\n frames_idx = list(offsets)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:270-292"
+ },
+ "5347": {
+ "file_id": 445,
+ "content": "This code generates random offsets for selecting frames from a video. If the valid mode is not enabled, it randomly selects frame offsets within the available duration or number of frames. If the valid mode is enabled, it evenly distributes the frames across the video duration. The 'format' variable determines if the selected frames are in video format.",
+ "type": "comment"
+ },
+ "5348": {
+ "file_id": 445,
+ "content": " frames_idx = [x % frames_len for x in frames_idx]\n elif results['format'] == 'frame':\n frames_idx = list(offsets + 1)\n elif results['format'] == 'MRI':\n frames_idx = list(offsets)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)\n@PIPELINES.register()\nclass SamplerPkl(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_seg(int): number of segments.\n seg_len(int): number of sampled frames in each segment.\n mode(str): 'train', 'valid'\n Returns:\n frames_idx: the index of sampled #frames.\n \"\"\"\n def __init__(self, num_seg, seg_len, backend='pillow', valid_mode=False):\n self.num_seg = num_seg\n self.seg_len = seg_len\n self.valid_mode = valid_mode\n self.backend = backend\n def _get(self, buf):\n if isinstance(buf, str):\n img = Image.open(StringIO(buf))\n else:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:293-327"
+ },
+ "5349": {
+ "file_id": 445,
+ "content": "This code snippet defines a SamplerPkl class that samples frames' indices for video loading. It takes arguments num_seg, seg_len, and backend and returns the index of sampled frames. Depending on the results format ('frame', 'MRI', or others), it sets the frames_idx accordingly before returning it.",
+ "type": "comment"
+ },
+ "5350": {
+ "file_id": 445,
+ "content": " img = Image.open(BytesIO(buf))\n img = img.convert('RGB')\n if self.backend != 'pillow':\n img = np.array(img)\n return img\n def __call__(self, results):\n \"\"\"\n Args:\n frames_len: length of frames.\n return:\n sampling id.\n \"\"\"\n filename = results['frame_dir']\n data_loaded = pickle.load(open(filename, 'rb'), encoding='bytes')\n video_name, label, frames = data_loaded\n if isinstance(label, dict):\n label = label['动作类型']\n results['labels'] = label\n elif len(label) == 1:\n results['labels'] = int(label[0])\n else:\n results['labels'] = int(label[0]) if random.random() < 0.5 else int(\n label[1])\n results['frames_len'] = len(frames)\n frames_len = results['frames_len']\n average_dur = int(int(frames_len) / self.num_seg)\n imgs = []\n for i in range(self.num_seg):\n idx = 0\n if not self.valid_mode:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:328-358"
+ },
+ "5351": {
+ "file_id": 445,
+ "content": "This code is part of a pipeline for image sampling in video processing. It loads data from disk, converts images to RGB format, and handles labels. The `__call__` method takes results as input, retrieves the video name, label, and frames from the loaded data. If the label is a dictionary or has multiple elements, it assigns the label to '动作类型' or randomly chooses between the first two elements. It sets the 'frames_len' based on the length of frames and calculates the average duration per segment. Then, it initializes an empty list for the images and loops through the segments to create image samples. If valid mode is not enabled, it also resets the index variable.",
+ "type": "comment"
+ },
+ "5352": {
+ "file_id": 445,
+ "content": " if average_dur >= self.seg_len:\n idx = random.randint(0, average_dur - self.seg_len)\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n else:\n if average_dur >= self.seg_len:\n idx = (average_dur - 1) // 2\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n for jj in range(idx, idx + self.seg_len):\n imgbuf = frames[int(jj % results['frames_len'])]\n img = self._get(imgbuf)\n imgs.append(img)\n results['backend'] = self.backend\n results['imgs'] = imgs\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample.py:359-382"
+ },
+ "5353": {
+ "file_id": 445,
+ "content": "The code calculates the index for a segment of frames based on average duration and frame length. It then retrieves images from the frames list, appends them to imgs, sets backend type, and returns the results including the imgs and backend information.",
+ "type": "comment"
+ },
+ "5354": {
+ "file_id": 446,
+ "content": "/paddlevideo/loader/pipelines/sample_ava.py",
+ "type": "filepath"
+ },
+ "5355": {
+ "file_id": 446,
+ "content": "This code introduces SampleFrames class for PaddleVideo's loader pipelines using OpenCV, supports various modes and training, includes storage backend classes, converts images to numpy arrays, defines pipeline, and provides SampleAVAFrames class for sampling video frames.",
+ "type": "summary"
+ },
+ "5356": {
+ "file_id": 446,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nfrom PIL import Image\nfrom ..registry import PIPELINES\nimport os\nimport numpy as np\nimport io\nimport os.path as osp\nfrom abc import ABCMeta, abstractmethod\nimport cv2\nfrom cv2 import IMREAD_COLOR, IMREAD_GRAYSCALE, IMREAD_UNCHANGED\nimport inspect\nimread_backend = 'cv2'\nimread_flags = {\n 'color': IMREAD_COLOR,\n 'grayscale': IMREAD_GRAYSCALE,\n 'unchanged': IMREAD_UNCHANGED\n}\n@PIPELINES.register()\nclass SampleFrames:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:1-35"
+ },
+ "5357": {
+ "file_id": 446,
+ "content": "This code is importing necessary libraries and registering a class SampleFrames under PaddleVideo's loader pipelines. The class appears to sample frames from video, supporting different reading modes (color/grayscale/unchanged). It uses OpenCV (cv2) as the image processing backend.",
+ "type": "comment"
+ },
+ "5358": {
+ "file_id": 446,
+ "content": " \"\"\"Sample frames from the video. \"\"\"\n def __init__(self,\n clip_len,\n frame_interval=1,\n num_clips=1,\n temporal_jitter=False,\n twice_sample=False,\n out_of_bound_opt='loop',\n test_mode=False):\n self.clip_len = clip_len\n self.frame_interval = frame_interval\n self.num_clips = num_clips\n self.temporal_jitter = temporal_jitter\n self.twice_sample = twice_sample\n self.out_of_bound_opt = out_of_bound_opt\n self.test_mode = test_mode\n assert self.out_of_bound_opt in ['loop', 'repeat_last']\n def _get_train_clips(self, num_frames):\n \"\"\"Get clip offsets in train mode. \"\"\"\n ori_clip_len = self.clip_len * self.frame_interval\n avg_interval = (num_frames - ori_clip_len + 1) // self.num_clips\n if avg_interval > 0:\n base_offsets = np.arange(self.num_clips) * avg_interval\n clip_offsets = base_offsets + np.random.randint(",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:36-61"
+ },
+ "5359": {
+ "file_id": 446,
+ "content": "The function `__init__` initializes the parameters for sampling frames from a video, including clip length, frame interval, number of clips, temporal jittering options, and out-of-bound handling. The `_get_train_clips` function calculates the clip offsets in training mode by determining the average interval between clips based on the total number of frames. It then generates random base offsets and adds random offsets to create the final clip offsets.",
+ "type": "comment"
+ },
+ "5360": {
+ "file_id": 446,
+ "content": " avg_interval, size=self.num_clips)\n elif num_frames > max(self.num_clips, ori_clip_len):\n clip_offsets = np.sort(\n np.random.randint(\n num_frames - ori_clip_len + 1, size=self.num_clips))\n elif avg_interval == 0:\n ratio = (num_frames - ori_clip_len + 1.0) / self.num_clips\n clip_offsets = np.around(np.arange(self.num_clips) * ratio)\n else:\n clip_offsets = np.zeros((self.num_clips, ), dtype=np.int)\n return clip_offsets\n def _get_test_clips(self, num_frames):\n \"\"\"Get clip offsets in test mode. \"\"\"\n ori_clip_len = self.clip_len * self.frame_interval\n avg_interval = (num_frames - ori_clip_len + 1) / float(self.num_clips)\n if num_frames > ori_clip_len - 1:\n base_offsets = np.arange(self.num_clips) * avg_interval\n clip_offsets = (base_offsets + avg_interval / 2.0).astype(np.int)\n if self.twice_sample:\n clip_offsets = np.concatenate([clip_offsets, base_offsets])",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:62-82"
+ },
+ "5361": {
+ "file_id": 446,
+ "content": "This code calculates clip offsets for video sampling based on the number of frames and other parameters. It handles different scenarios, such as when the number of frames exceeds or equals the original clip length, when average interval is 0, and in test mode. The clip_offsets are returned at the end.",
+ "type": "comment"
+ },
+ "5362": {
+ "file_id": 446,
+ "content": " else:\n clip_offsets = np.zeros((self.num_clips, ), dtype=np.int)\n return clip_offsets\n def _sample_clips(self, num_frames):\n \"\"\"Choose clip offsets for the video in a given mode. \"\"\"\n if self.test_mode:\n clip_offsets = self._get_test_clips(num_frames)\n else:\n clip_offsets = self._get_train_clips(num_frames)\n return clip_offsets\n def __call__(self, results):\n \"\"\"Perform the SampleFrames loading. \"\"\"\n total_frames = results['total_frames']\n clip_offsets = self._sample_clips(total_frames)\n frame_inds = clip_offsets[:, None] + np.arange(\n self.clip_len)[None, :] * self.frame_interval\n frame_inds = np.concatenate(frame_inds)\n if self.temporal_jitter:\n perframe_offsets = np.random.randint(\n self.frame_interval, size=len(frame_inds))\n frame_inds += perframe_offsets\n frame_inds = frame_inds.reshape((-1, self.clip_len))\n if self.out_of_bound_opt == 'loop':",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:83-107"
+ },
+ "5363": {
+ "file_id": 446,
+ "content": "This code defines a class that samples video clips and loads frames based on different modes, such as testing or training. It takes the total number of frames in a video and returns the corresponding clip offsets and frame indices for loading. The sampling mode, temporal jitter, and out-of-bound options can be specified to customize the sampling process.",
+ "type": "comment"
+ },
+ "5364": {
+ "file_id": 446,
+ "content": " frame_inds = np.mod(frame_inds, total_frames)\n elif self.out_of_bound_opt == 'repeat_last':\n safe_inds = frame_inds < total_frames\n unsafe_inds = 1 - safe_inds\n last_ind = np.max(safe_inds * frame_inds, axis=1)\n new_inds = (safe_inds * frame_inds + (unsafe_inds.T * last_ind).T)\n frame_inds = new_inds\n else:\n raise ValueError('Illegal out_of_bound option.')\n start_index = results['start_index']\n frame_inds = np.concatenate(frame_inds) + start_index\n results['frame_inds'] = frame_inds.astype(np.int)\n results['clip_len'] = self.clip_len\n results['frame_interval'] = self.frame_interval\n results['num_clips'] = self.num_clips\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'clip_len={self.clip_len}, '\n f'frame_interval={self.frame_interval}, '\n f'num_clips={self.num_clips}, '",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:108-129"
+ },
+ "5365": {
+ "file_id": 446,
+ "content": "Code handles out-of-bound frame indices by wrapping them around, repeating the last frame, or throwing an error. It then updates results with frame indices, clip length, and number of clips.",
+ "type": "comment"
+ },
+ "5366": {
+ "file_id": 446,
+ "content": " f'temporal_jitter={self.temporal_jitter}, '\n f'twice_sample={self.twice_sample}, '\n f'out_of_bound_opt={self.out_of_bound_opt}, '\n f'test_mode={self.test_mode})')\n return repr_str\nclass BaseStorageBackend(metaclass=ABCMeta):\n \"\"\"Abstract class of storage backends. \"\"\"\n @abstractmethod\n def get(self, filepath):\n pass\n @abstractmethod\n def get_text(self, filepath):\n pass\nclass HardDiskBackend(BaseStorageBackend):\n \"\"\"Raw hard disks storage backend.\"\"\"\n def get(self, filepath):\n filepath = str(filepath)\n with open(filepath, 'rb') as f:\n value_buf = f.read()\n return value_buf\n def get_text(self, filepath):\n filepath = str(filepath)\n with open(filepath, 'r') as f:\n value_buf = f.read()\n return value_buf\nclass FileClient:\n \"\"\"A general file client to access files in different backend. \"\"\"\n _backends = {\n 'disk': HardDiskBackend,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:130-166"
+ },
+ "5367": {
+ "file_id": 446,
+ "content": "This code defines three classes: `BaseStorageBackend` (abstract class), `HardDiskBackend`, and a generic file client called `FileClient`. The `BaseStorageBackend` is an abstract class that provides two methods, `get()` and `get_text()`, which are expected to be implemented by subclasses. The `HardDiskBackend` implements these methods for handling files stored on the hard disk. Finally, the `FileClient` serves as a generic file client to access files in different backends.",
+ "type": "comment"
+ },
+ "5368": {
+ "file_id": 446,
+ "content": " }\n def __init__(self, backend='disk', **kwargs):\n if backend not in self._backends:\n raise ValueError(\n f'Backend {backend} is not supported. Currently supported ones'\n f' are {list(self._backends.keys())}')\n self.backend = backend\n self.client = self._backends[backend](**kwargs)\n @classmethod\n def _register_backend(cls, name, backend, force=False):\n if not isinstance(name, str):\n raise TypeError('the backend name should be a string, '\n f'but got {type(name)}')\n if not inspect.isclass(backend):\n raise TypeError(\n f'backend should be a class but got {type(backend)}')\n if not issubclass(backend, BaseStorageBackend):\n raise TypeError(\n f'backend {backend} is not a subclass of BaseStorageBackend')\n if not force and name in cls._backends:\n raise KeyError(\n f'{name} is already registered as a storage backend, '",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:167-190"
+ },
+ "5369": {
+ "file_id": 446,
+ "content": "This code defines a class with an initializer and a class method for registering backends. The initializer takes a backend argument, checks if it is supported, and initializes the client object with that backend. If the name or backend type is incorrect, it raises TypeError. The _register_backend method allows for backend registration by name, checking if it is a string and if the backend is a subclass of BaseStorageBackend. Raises KeyError if already registered.",
+ "type": "comment"
+ },
+ "5370": {
+ "file_id": 446,
+ "content": " 'add \"force=True\" if you want to override it')\n cls._backends[name] = backend\n @classmethod\n def register_backend(cls, name, backend=None, force=False):\n \"\"\"Register a backend to FileClient. \"\"\"\n if backend is not None:\n cls._register_backend(name, backend, force=force)\n return\n def _register(backend_cls):\n cls._register_backend(name, backend_cls, force=force)\n return backend_cls\n return _register\n def get(self, filepath):\n return self.client.get(filepath)\n def get_text(self, filepath):\n return self.client.get_text(filepath)\n@PIPELINES.register()\nclass RawFrameDecode:\n \"\"\"Load and decode frames with given indices. \"\"\"\n def __init__(self, io_backend='disk', decoding_backend='cv2', **kwargs):\n self.io_backend = io_backend\n self.decoding_backend = decoding_backend\n self.kwargs = kwargs\n self.file_client = None\n def _pillow2array(self,img, flag='color', channel_order='bgr'):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:191-225"
+ },
+ "5371": {
+ "file_id": 446,
+ "content": "This code defines a class called FileClient, which handles file operations like registration and retrieval. It also registers a pipeline named RawFrameDecode for loading and decoding frames using specified backends for I/O and decoding. The class has an _pillow2array method to convert PIL image to numpy array in specific channel order.",
+ "type": "comment"
+ },
+ "5372": {
+ "file_id": 446,
+ "content": " \"\"\"Convert a pillow image to numpy array. \"\"\"\n channel_order = channel_order.lower()\n if channel_order not in ['rgb', 'bgr']:\n raise ValueError('channel order must be either \"rgb\" or \"bgr\"')\n if flag == 'unchanged':\n array = np.array(img)\n if array.ndim >= 3 and array.shape[2] >= 3: # color image\n array[:, :, :3] = array[:, :, (2, 1, 0)] # RGB to BGR\n else:\n # If the image mode is not 'RGB', convert it to 'RGB' first.\n if img.mode != 'RGB':\n if img.mode != 'LA':\n # Most formats except 'LA' can be directly converted to RGB\n img = img.convert('RGB')\n else:\n # When the mode is 'LA', the default conversion will fill in\n # the canvas with black, which sometimes shadows black objects\n # in the foreground.\n #\n # Therefore, a random color (124, 117, 104) is used for canvas",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:226-247"
+ },
+ "5373": {
+ "file_id": 446,
+ "content": "This code converts a Pillow image to a numpy array. It checks the channel order and flag, then either keeps the array unchanged or converts it from RGB to BGR if necessary. If the image mode is not RGB, it converts it to RGB first using convert('RGB'). If the mode is LA, a random color is used for the canvas to avoid shadowing black objects in the foreground.",
+ "type": "comment"
+ },
+ "5374": {
+ "file_id": 446,
+ "content": " img_rgba = img.convert('RGBA')\n img = Image.new('RGB', img_rgba.size, (124, 117, 104))\n img.paste(img_rgba, mask=img_rgba.split()[3]) # 3 is alpha\n if flag == 'color':\n array = np.array(img)\n if channel_order != 'rgb':\n array = array[:, :, ::-1] # RGB to BGR\n elif flag == 'grayscale':\n img = img.convert('L')\n array = np.array(img)\n else:\n raise ValueError(\n 'flag must be \"color\", \"grayscale\" or \"unchanged\", '\n f'but got {flag}')\n return array\n def _imfrombytes(self,content, flag='color', channel_order='bgr'):#, backend=None):\n \"\"\"Read an image from bytes. \"\"\"\n img_np = np.frombuffer(content, np.uint8)\n flag = imread_flags[flag] if isinstance(flag, str) else flag\n img = cv2.imdecode(img_np, flag)\n if flag == IMREAD_COLOR and channel_order == 'rgb':",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:248-270"
+ },
+ "5375": {
+ "file_id": 446,
+ "content": "The code reads an image from bytes and converts it into a numpy array based on the provided flag (color or grayscale) and channel order. It first checks if the flag is valid, then decodes the image using OpenCV's imdecode function. If the flag is color and channel order is rgb, it returns the image as is. For other combinations, it converts the image to RGB or grayscale before returning the numpy array.",
+ "type": "comment"
+ },
+ "5376": {
+ "file_id": 446,
+ "content": " cv2.cvtColor(img, cv2.COLOR_BGR2RGB, img)\n return img\n def __call__(self, results):\n \"\"\"Perform the ``RawFrameDecode`` to pick frames given indices.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n # mmcv.use_backend(self.decoding_backend)\n directory = results['frame_dir']\n suffix = results['suffix']\n #modality = results['modality']\n if self.file_client is None:\n self.file_client = FileClient(self.io_backend, **self.kwargs)\n imgs = list()\n if results['frame_inds'].ndim != 1:\n results['frame_inds'] = np.squeeze(results['frame_inds'])\n offset = results.get('offset', 0)\n for frame_idx in results['frame_inds']:\n frame_idx += offset\n filepath = osp.join(directory, suffix.format(frame_idx))\n img_bytes = self.file_client.get(filepath) #以二进制方式读取图片\n # Get frame with channel order RGB directly.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:271-301"
+ },
+ "5377": {
+ "file_id": 446,
+ "content": "This code defines a pipeline for decoding frames using the RawFrameDecode transform. It reads image files from the specified directory and suffix, handles different frame indices, and utilizes a file client to retrieve images in binary format. The cv2.cvtColor function is used to convert the color of images from BGR to RGB. The code also checks if the frame indices have the correct dimensions and squeezes them if necessary.",
+ "type": "comment"
+ },
+ "5378": {
+ "file_id": 446,
+ "content": " cur_frame = self._imfrombytes(img_bytes, channel_order='rgb')\n imgs.append(cur_frame)\n results['imgs'] = imgs\n results['original_shape'] = imgs[0].shape[:2]\n results['img_shape'] = imgs[0].shape[:2]\n # we resize the gt_bboxes and proposals to their real scale\n h, w = results['img_shape']\n scale_factor = np.array([w, h, w, h])\n if 'gt_bboxes' in results:\n gt_bboxes = results['gt_bboxes']\n gt_bboxes_new = (gt_bboxes * scale_factor).astype(np.float32)\n results['gt_bboxes'] = gt_bboxes_new\n if 'proposals' in results and results['proposals'] is not None:\n proposals = results['proposals']\n proposals = (proposals * scale_factor).astype(np.float32)\n results['proposals'] = proposals\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'io_backend={self.io_backend}, '\n f'decoding_backend={self.decoding_backend})')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:303-326"
+ },
+ "5379": {
+ "file_id": 446,
+ "content": "Function applies image processing and resizing to input, appends frames to a list, and scales gt_bboxes and proposals accordingly. It then returns the results. The __repr__ function provides a string representation of the object's class and arguments.",
+ "type": "comment"
+ },
+ "5380": {
+ "file_id": 446,
+ "content": " return repr_str\n@PIPELINES.register()\nclass SampleAVAFrames(SampleFrames):\n def __init__(self, clip_len, frame_interval=2, test_mode=False):\n super().__init__(clip_len, frame_interval, test_mode=test_mode)\n def _get_clips(self, center_index, skip_offsets, shot_info):\n start = center_index - (self.clip_len // 2) * self.frame_interval\n end = center_index + ((self.clip_len + 1) // 2) * self.frame_interval\n frame_inds = list(range(start, end, self.frame_interval))\n frame_inds = frame_inds + skip_offsets\n frame_inds = np.clip(frame_inds, shot_info[0], shot_info[1] - 1)\n return frame_inds\n def __call__(self, results):\n fps = results['fps']\n timestamp = results['timestamp']\n timestamp_start = results['timestamp_start']\n shot_info = results['shot_info']\n #delta=(timestamp - timestamp_start) 为该帧距离15min视频开头有几秒\n #center_index=fps*delta为该帧距离15min视频开头有几帧\n #center_index+1是为了避免后续采样时出现负数? \n #后续需要以center_index为中心前后采样视频帧片段",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:327-354"
+ },
+ "5381": {
+ "file_id": 446,
+ "content": "The code defines a class called SampleAVAFrames, which inherits from SampleFrames. It takes clip length, frame interval, and test mode as arguments during initialization. The _get_clips method calculates the start and end indices for a given center index, taking into account skip offsets and shot information. The __call__ method retrieves fps, timestamp, timestamp_start, and shot_info from the results dictionary, and then calculates the center index to sample video frames around that index.",
+ "type": "comment"
+ },
+ "5382": {
+ "file_id": 446,
+ "content": " center_index = fps * (timestamp - timestamp_start) + 1\n skip_offsets = np.random.randint(\n -self.frame_interval // 2, (self.frame_interval + 1) // 2,\n size=self.clip_len)\n frame_inds = self._get_clips(center_index, skip_offsets, shot_info)\n results['frame_inds'] = np.array(frame_inds, dtype=np.int)\n results['clip_len'] = self.clip_len\n results['frame_interval'] = self.frame_interval\n results['num_clips'] = 1\n results['crop_quadruple'] = np.array([0, 0, 1, 1], dtype=np.float32)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'clip_len={self.clip_len}, '\n f'frame_interval={self.frame_interval}, '\n f'test_mode={self.test_mode})')\n return repr_str",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ava.py:355-374"
+ },
+ "5383": {
+ "file_id": 446,
+ "content": "This function samples a video clip by calculating the center index and generating random skip offsets to select frames. It returns frame indices, clip length, frame interval, number of clips, and crop quadruple in a dictionary format for further processing. The `__repr__` method provides a concise string representation of the object's attributes.",
+ "type": "comment"
+ },
+ "5384": {
+ "file_id": 447,
+ "content": "/paddlevideo/loader/pipelines/sample_ucf24.py",
+ "type": "filepath"
+ },
+ "5385": {
+ "file_id": 447,
+ "content": "The \"SamplerUCF24\" class samples frames from videos using parameters like frame count and interval, utilizes PIL library, initializes pipeline, generates frame indices, returns sampled frames.",
+ "type": "summary"
+ },
+ "5386": {
+ "file_id": 447,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport random\nfrom PIL import Image\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass SamplerUCF24(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_frames(int): The amount of frames used in a video\n frame_interval(int): Sampling rate\n valid_mode(bool): True or False.\n Returns:\n frames_idx: the index of sampled #frames.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ucf24.py:1-33"
+ },
+ "5387": {
+ "file_id": 447,
+ "content": "This code defines a class \"SamplerUCF24\" for sampling frames in videos, taking parameters such as num_frames and frame_interval. It uses PIL instead of OpenCV to read images and returns the index of sampled frames.",
+ "type": "comment"
+ },
+ "5388": {
+ "file_id": 447,
+ "content": " \"\"\"\n def __init__(self,\n num_frames=16,\n frame_interval=1,\n valid_mode=False):\n self.num_frames = num_frames\n self.frame_interval = frame_interval if valid_mode else random.randint(1, 2)\n self.valid_mode = valid_mode\n def _get(self, frames_idxs, img_folder, results):\n imgs = []\n for idx in frames_idxs:\n img = Image.open(\n os.path.join(img_folder, '{:05d}.jpg'.format(idx))).convert('RGB')\n imgs.append(img)\n results['imgs'] = imgs\n return results\n def _make_clip(self, im_ind, max_num):\n frame_idxs = []\n for i in reversed(range(self.num_frames)):\n # make it as a loop\n i_temp = im_ind - i * self.frame_interval\n if i_temp < 1:\n i_temp = 1\n elif i_temp > max_num:\n i_temp = max_num\n frame_idxs.append(i_temp)\n return frame_idxs\n def __call__(self, results):\n img_folder, key_frame = os.path.split(results['filename'])",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ucf24.py:34-65"
+ },
+ "5389": {
+ "file_id": 447,
+ "content": "This code defines a pipeline for loading and creating clips from video files. The `__init__` method initializes the number of frames, frame interval (randomly determined if valid mode is False), and valid mode flag. The `_get` method retrieves images in order, converts them to RGB, and appends them to a list. The `_make_clip` method generates a set of frame indices that create a looped clip. The pipeline is called with the results as input, extracting the image folder and filename for further processing.",
+ "type": "comment"
+ },
+ "5390": {
+ "file_id": 447,
+ "content": " frame_len = len(os.listdir(img_folder))\n key_idx = int(key_frame[0:5])\n frame_idxs = self._make_clip(key_idx, frame_len)\n return self._get(frame_idxs, img_folder, results)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/sample_ucf24.py:66-69"
+ },
+ "5391": {
+ "file_id": 447,
+ "content": "This code retrieves the number of frames in a folder, assigns a key frame index based on the input, generates frame indices for a video clip, and returns the requested frames from their folder.",
+ "type": "comment"
+ },
+ "5392": {
+ "file_id": 448,
+ "content": "/paddlevideo/loader/pipelines/segmentation.py",
+ "type": "filepath"
+ },
+ "5393": {
+ "file_id": 448,
+ "content": "The code enables image resizing, flipping, multi-scale segmentation in PaddleVideo's pipeline, with metadata addition and normalization. It performs image normalization and transposition before storing the result in a samples data structure.",
+ "type": "summary"
+ },
+ "5394": {
+ "file_id": 448,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nfrom PIL import Image\nimport copy\nimport cv2\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass MultiRestrictSize(object):\n def __init__(self,\n min_size=None,\n max_size=800,\n flip=False,\n multi_scale=[1.3]):\n self.min_size = min_size\n self.max_size = max_size\n self.multi_scale = multi_scale\n self.flip = flip",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation.py:1-32"
+ },
+ "5395": {
+ "file_id": 448,
+ "content": "This code is for PaddleVideo's segmentation pipeline. It includes the class definition MultiRestrictSize, which can be used with minimum and maximum size limits, flipping option, and multiple scales for image resizing.",
+ "type": "comment"
+ },
+ "5396": {
+ "file_id": 448,
+ "content": " assert ((min_size is None)) or ((max_size is None))\n def __call__(self, sample):\n samples = []\n image = sample['current_img']\n h, w = image.shape[:2]\n for scale in self.multi_scale:\n # Fixed range of scales\n sc = None\n # Align short edge\n if not (self.min_size is None):\n if h > w:\n short_edge = w\n else:\n short_edge = h\n if short_edge > self.min_size:\n sc = float(self.min_size) / short_edge\n else:\n if h > w:\n long_edge = h\n else:\n long_edge = w\n if long_edge > self.max_size:\n sc = float(self.max_size) / long_edge\n if sc is None:\n new_h = h\n new_w = w\n else:\n new_h = sc * h\n new_w = sc * w\n new_h = int(new_h * scale)\n new_w = int(new_w * scale)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation.py:33-65"
+ },
+ "5397": {
+ "file_id": 448,
+ "content": "This code is a function that applies image segmentation using multi-scale technique. It scales the input image based on a fixed range of scales and aligns short or long edges to meet minimum or maximum size requirements, respectively. The scaled images are stored in a list for further processing.",
+ "type": "comment"
+ },
+ "5398": {
+ "file_id": 448,
+ "content": " if (new_h - 1) % 16 != 0:\n new_h = int(np.around((new_h - 1) / 16.) * 16 + 1)\n if (new_w - 1) % 16 != 0:\n new_w = int(np.around((new_w - 1) / 16.) * 16 + 1)\n if new_h == h and new_w == w:\n samples.append(sample)\n else:\n new_sample = {}\n for elem in sample.keys():\n if 'meta' in elem:\n new_sample[elem] = sample[elem]\n continue\n tmp = sample[elem]\n if 'label' in elem:\n new_sample[elem] = sample[elem]\n continue\n else:\n flagval = cv2.INTER_CUBIC\n tmp = cv2.resize(tmp,\n dsize=(new_w, new_h),\n interpolation=flagval)\n new_sample[elem] = tmp\n samples.append(new_sample)\n if self.flip:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation.py:67-92"
+ },
+ "5399": {
+ "file_id": 448,
+ "content": "Code resizes input images to a multiple of 16x16, appends samples with matching metadata, and optionally flips the image if enabled.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/54.json b/docs/data/54.json
new file mode 100644
index 000000000..21c8d4bc5
--- /dev/null
+++ b/docs/data/54.json
@@ -0,0 +1,550 @@
+{
+ "5400": {
+ "file_id": 448,
+ "content": " now_sample = samples[-1]\n new_sample = {}\n for elem in now_sample.keys():\n if 'meta' in elem:\n new_sample[elem] = now_sample[elem].copy()\n new_sample[elem]['flip'] = True\n continue\n tmp = now_sample[elem]\n tmp = tmp[:, ::-1].copy()\n new_sample[elem] = tmp\n samples.append(new_sample)\n return samples\n@PIPELINES.register()\nclass MultiNorm(object):\n def __call__(self, samples):\n for idx in range(len(samples)):\n sample = samples[idx]\n for elem in sample.keys():\n if 'meta' in elem:\n continue\n tmp = sample[elem]\n if tmp is None:\n continue\n if tmp.ndim == 2:\n tmp = tmp[:, :, np.newaxis]\n else:\n tmp = tmp / 255.\n tmp -= (0.485, 0.456, 0.406)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation.py:93-124"
+ },
+ "5401": {
+ "file_id": 448,
+ "content": "This code segment is from the PaddleVideo library, specifically in the loader/pipelines/segmentation.py file. It appears to be a function that adds flipped image data to a list of samples, after normalizing each image by dividing it by 255 and subtracting (0.485, 0.456, 0.406). This function is part of the MultiNorm pipeline registered in the PIPELINES module.",
+ "type": "comment"
+ },
+ "5402": {
+ "file_id": 448,
+ "content": " tmp /= (0.229, 0.224, 0.225)\n tmp = tmp.transpose((2, 0, 1))\n samples[idx][elem] = tmp\n return samples",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation.py:125-130"
+ },
+ "5403": {
+ "file_id": 448,
+ "content": "This code segment performs image normalization and transposition before storing the result in a dictionary-like samples data structure. It divides each RGB channel value by the average RGB values, then transposes the image channels. Finally, it adds the transformed image to the samples dictionary for the given index and element.",
+ "type": "comment"
+ },
+ "5404": {
+ "file_id": 449,
+ "content": "/paddlevideo/loader/pipelines/segmentation_pipline.py",
+ "type": "filepath"
+ },
+ "5405": {
+ "file_id": 449,
+ "content": "This Python code defines a SegmentationSampler class in PaddleVideo, which samples data at a specified rate and registers it for the Action Segmentation Dataset. It is part of a video processing library's pipeline, likely for segmentation purposes.",
+ "type": "summary"
+ },
+ "5406": {
+ "file_id": 449,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport os\nimport numpy as np\nimport random\nimport paddle\nfrom ..registry import PIPELINES\n\"\"\"\npipeline ops for Action Segmentation Dataset.\n\"\"\"\n@PIPELINES.register()\nclass SegmentationSampler(object):\n def __init__(self, sample_rate):\n self.sample_rate = sample_rate\n def __call__(self, results):\n for key, data in results.items():\n if len(data.shape) == 1:\n data = data[::self.sample_rate]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation_pipline.py:1-35"
+ },
+ "5407": {
+ "file_id": 449,
+ "content": "This Python code is from the PaddleVideo library and defines a SegmentationSampler class. It samples data at a specified rate, only keeps every nth element in a 1D array, and registers this pipeline operation for the Action Segmentation Dataset.",
+ "type": "comment"
+ },
+ "5408": {
+ "file_id": 449,
+ "content": " results[key] = copy.deepcopy(data)\n else:\n data = data[:, ::self.sample_rate]\n results[key] = copy.deepcopy(data)\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/segmentation_pipline.py:36-40"
+ },
+ "5409": {
+ "file_id": 449,
+ "content": "This code segment appears to be part of a pipeline in a video processing library, possibly for segmentation. It selects specific data based on the sample rate and stores it in a results dictionary with deep copy.",
+ "type": "comment"
+ },
+ "5410": {
+ "file_id": 450,
+ "content": "/paddlevideo/loader/pipelines/skeleton_pipeline.py",
+ "type": "filepath"
+ },
+ "5411": {
+ "file_id": 450,
+ "content": "The code develops efficient data processing classes for PaddleVideo, including interpolation, cropping, and pipeline optimization. It performs image flipping operations, augments 'Flow' modality images, transforms data formats, collapses dimensions, includes ML data pipeline support, generates heatmaps for keypoints and limbs in image sequences using input parameters, applies data augmentation, and uses Gaussian filtering with specified sigma value.",
+ "type": "summary"
+ },
+ "5412": {
+ "file_id": 450,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport collections\nfrom itertools import repeat\nimport copy as cp\nfrom collections import abc\nimport numpy as np\nimport paddle.nn.functional as F\nimport random\nimport paddle\nfrom ..registry import PIPELINES\nfrom .augmentations_ava import iminvert, imflip_\n\"\"\"pipeline ops for Activity Net.\n\"\"\"\ndef _ntuple(n):\n def parse(x):\n if isinstance(x, collections.abc.Iterable):\n return tuple(x)\n return tuple(repeat(x, n))",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1-34"
+ },
+ "5413": {
+ "file_id": 450,
+ "content": "This code is importing necessary libraries and defining a function to create ActivityNet-style pipeline operations. It also registers these pipelines for future use.",
+ "type": "comment"
+ },
+ "5414": {
+ "file_id": 450,
+ "content": " return parse\n_single = _ntuple(1)\n_pair = _ntuple(2)\n_triple = _ntuple(3)\n_quadruple = _ntuple(4)\ndef _init_lazy_if_proper(results, lazy):\n \"\"\"Initialize lazy operation properly.\n Make sure that a lazy operation is properly initialized,\n and avoid a non-lazy operation accidentally getting mixed in.\n Required keys in results are \"imgs\" if \"img_shape\" not in results,\n otherwise, Required keys in results are \"img_shape\", add or modified keys\n are \"img_shape\", \"lazy\".\n Add or modified keys in \"lazy\" are \"original_shape\", \"crop_bbox\", \"flip\",\n \"flip_direction\", \"interpolation\".\n Args:\n results (dict): A dict stores data pipeline result.\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n if 'img_shape' not in results:\n results['img_shape'] = results['imgs'][0].shape[:2]\n if lazy:\n if 'lazy' not in results:\n img_h, img_w = results['img_shape']\n lazyop = dict()\n lazyop['original_shape'] = results['img_shape']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:36-68"
+ },
+ "5415": {
+ "file_id": 450,
+ "content": "This function initializes the lazy operation properly by checking if \"img_shape\" is in results, and adds or modifies keys \"lazy\", \"original_shape\", \"crop_bbox\", \"flip\", \"flip_direction\", and \"interpolation\" based on whether \"lazy\" is True.",
+ "type": "comment"
+ },
+ "5416": {
+ "file_id": 450,
+ "content": " lazyop['crop_bbox'] = np.array([0, 0, img_w, img_h],\n dtype=np.float32)\n lazyop['flip'] = False\n lazyop['flip_direction'] = None\n lazyop['interpolation'] = None\n results['lazy'] = lazyop\n else:\n assert 'lazy' not in results, 'Use Fuse after lazy operations'\n@PIPELINES.register()\nclass AutoPadding(object):\n \"\"\"\n Sample or Padding frame skeleton feature.\n Args:\n window_size: int, temporal size of skeleton feature.\n random_pad: bool, whether do random padding when frame length < window size. Default: False.\n \"\"\"\n def __init__(self, window_size, random_pad=False):\n self.window_size = window_size\n self.random_pad = random_pad\n def get_frame_num(self, data):\n C, T, V, M = data.shape\n for i in range(T - 1, -1, -1):\n tmp = np.sum(data[:, i, :, :])\n if tmp > 0:\n T = i + 1\n break\n return T\n def __call__(self, results):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:69-101"
+ },
+ "5417": {
+ "file_id": 450,
+ "content": "The code is defining a Pipeline class, specifically for auto-padding and skeleton feature extraction from image data. It first checks if the 'lazy' operation has been performed, then initializes necessary parameters for padding or sampling frames based on window size and random_pad setting. The get_frame_num function calculates the number of frames containing valid data, and the __call__ method applies the pipeline to the results.",
+ "type": "comment"
+ },
+ "5418": {
+ "file_id": 450,
+ "content": " data = results['data']\n C, T, V, M = data.shape\n T = self.get_frame_num(data)\n if T == self.window_size:\n data_pad = data[:, :self.window_size, :, :]\n elif T < self.window_size:\n begin = random.randint(\n 0, self.window_size - T) if self.random_pad else 0\n data_pad = np.zeros((C, self.window_size, V, M))\n data_pad[:, begin:begin + T, :, :] = data[:, :T, :, :]\n else:\n if self.random_pad:\n index = np.random.choice(\n T, self.window_size, replace=False).astype('int64')\n else:\n index = np.linspace(0, T, self.window_size).astype(\"int64\")\n data_pad = data[:, index, :, :]\n results['data'] = data_pad\n return results\n@PIPELINES.register()\nclass SkeletonNorm(object):\n \"\"\"\n Normalize skeleton feature.\n Args:\n aixs: dimensions of vertex coordinate. 2 for (x,y), 3 for (x,y,z). Default: 2.\n \"\"\"\n def __init__(self, axis=2, squeeze=False):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:102-133"
+ },
+ "5419": {
+ "file_id": 450,
+ "content": "Code snippet performs data padding to ensure consistent frame size for skeleton data in the Skeleton Pipeline. It checks the current frame number (T) and pads it with zeroes if T is smaller than the window size, or selects a subset of frames from the original data if T is larger than the window size. The result is then returned as 'results'.",
+ "type": "comment"
+ },
+ "5420": {
+ "file_id": 450,
+ "content": " self.axis = axis\n self.squeeze = squeeze\n def __call__(self, results):\n data = results['data']\n # Centralization\n data = data - data[:, :, 8:9, :]\n data = data[:self.axis, :, :, :] # get (x,y) from (x,y, acc)\n C, T, V, M = data.shape\n if self.squeeze:\n data = data.reshape((C, T, V)) # M = 1\n results['data'] = data.astype('float32')\n if 'label' in results:\n label = results['label']\n results['label'] = np.expand_dims(label, 0).astype('int64')\n return results\n@PIPELINES.register()\nclass Iden(object):\n \"\"\"\n Wrapper Pipeline\n \"\"\"\n def __init__(self, label_expand=True):\n self.label_expand = label_expand\n def __call__(self, results):\n data = results['data']\n results['data'] = data.astype('float32')\n if 'label' in results and self.label_expand:\n label = results['label']\n results['label'] = np.expand_dims(label, 0).astype('int64')\n return results",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:134-170"
+ },
+ "5421": {
+ "file_id": 450,
+ "content": "This code defines two classes, \"SkeletonPipeline\" and \"Iden\", which are used as PaddleVideo pipeline components. The SkeletonPipeline class is responsible for centralizing the data along a specified axis and reshaping it if squeeze is True. The Iden class simply converts the 'data' to float32 type and expands the shape of 'label' if it exists and label_expand is set to True. Both classes return updated results after processing.",
+ "type": "comment"
+ },
+ "5422": {
+ "file_id": 450,
+ "content": "@PIPELINES.register()\nclass RandomRotation(object):\n \"\"\"\n Random rotation sketeton.\n Args:\n argument: bool, if rotation.\n theta: float, rotation rate.\n \"\"\"\n def __init__(self, argument, theta=0.3):\n self.theta = theta\n self.argument = argument\n def _rot(self, rot):\n \"\"\"\n rot: T,3\n \"\"\"\n cos_r, sin_r = np.cos(rot), np.sin(rot) # T,3\n zeros = np.zeros((rot.shape[0], 1)) # T,1\n ones = np.ones((rot.shape[0], 1)) # T,1\n r1 = np.stack((ones, zeros, zeros), axis=-1) # T,1,3\n rx2 = np.stack((zeros, cos_r[:, 0:1], sin_r[:, 0:1]), axis=-1) # T,1,3\n rx3 = np.stack((zeros, -sin_r[:, 0:1], cos_r[:, 0:1]), axis=-1) # T,1,3\n rx = np.concatenate((r1, rx2, rx3), axis=1) # T,3,3\n ry1 = np.stack((cos_r[:, 1:2], zeros, -sin_r[:, 1:2]), axis=-1)\n r2 = np.stack((zeros, ones, zeros), axis=-1)\n ry3 = np.stack((sin_r[:, 1:2], zeros, cos_r[:, 1:2]), axis=-1)\n ry = np.concatenate((ry1, r2, ry3), axis=1)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:173-202"
+ },
+ "5423": {
+ "file_id": 450,
+ "content": "The code defines a random rotation skeleton class for applying random rotations to input data. It takes arguments for rotation and rotation rate, and has a method for performing the rotation operation on the input data. The method calculates rotation matrices for rotation around the x, y, and z axes using the given rotation rate and applies them to the input data.",
+ "type": "comment"
+ },
+ "5424": {
+ "file_id": 450,
+ "content": " rz1 = np.stack((cos_r[:, 2:3], sin_r[:, 2:3], zeros), axis=-1)\n r3 = np.stack((zeros, zeros, ones), axis=-1)\n rz2 = np.stack((-sin_r[:, 2:3], cos_r[:, 2:3], zeros), axis=-1)\n rz = np.concatenate((rz1, rz2, r3), axis=1)\n rot = np.matmul(np.matmul(rz, ry), rx)\n return rot\n def __call__(self, results):\n # C,T,V,M\n data = results['data']\n if self.argument:\n C, T, V, M = data.shape\n data_numpy = np.transpose(data, (1, 0, 2, 3)).conjugate().reshape(\n T, C, V * M) # T,3,V*M\n rot = np.random.uniform(-self.theta, self.theta, 3)\n rot = np.stack(\n [\n rot,\n ] * T, axis=0)\n rot = self._rot(rot) # T,3,3\n data_numpy = np.matmul(rot, data_numpy)\n data_numpy = data_numpy.reshape(T, C, V, M)\n data_numpy = np.transpose(data_numpy, (1, 0, 2, 3))\n data = data_numpy\n results['data'] = data.astype(np.float32)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:204-229"
+ },
+ "5425": {
+ "file_id": 450,
+ "content": "This code defines a class with two methods: `_rot` and `__call__`. The `_rot` method takes rotation angles and returns the rotation matrix. The `__call__` method applies random rotations to input data, performs the rotations using `_rot`, and adjusts the shape of the data before returning it.",
+ "type": "comment"
+ },
+ "5426": {
+ "file_id": 450,
+ "content": " return results\n@PIPELINES.register()\nclass SketeonCropSample(object):\n \"\"\"\n Sketeon Crop Sampler.\n Args:\n crop_model: str, crop model, support: ['center'].\n p_interval: list, crop len\n window_size: int, sample windows size.\n \"\"\"\n def __init__(self, window_size, crop_model='center', p_interval=1):\n assert crop_model in ['center'], \"Don't support :\" + crop_model\n self.crop_model = crop_model\n self.window_size = window_size\n self.p_interval = p_interval\n def __call__(self, results):\n if self.crop_model == 'center':\n # input: C,T,V,M\n data = results['data']\n valid_frame_num = np.sum(data.sum(0).sum(-1).sum(-1) != 0)\n C, T, V, M = data.shape\n begin = 0\n end = valid_frame_num\n valid_size = end - begin\n #crop\n if len(self.p_interval) == 1:\n p = self.p_interval[0]\n bias = int((1 - p) * valid_size / 2)\n data = data[:, begin + bias:end - bias, :, :] # center_crop",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:230-265"
+ },
+ "5427": {
+ "file_id": 450,
+ "content": "This code defines a Pipeline class for cropping sample data using the Sketeon crop model. It supports only the 'center' crop model and takes window size, crop model (default 'center'), and p_interval (default 1) as arguments. The __call__ method is used to apply the crop operation on the input results by selecting a center crop based on the crop model and p_interval values.",
+ "type": "comment"
+ },
+ "5428": {
+ "file_id": 450,
+ "content": " cropped_length = data.shape[1]\n else:\n p = np.random.rand(1) * (self.p_interval[1] - self.p_interval[0]\n ) + self.p_interval[0]\n # constraint cropped_length lower bound as 64\n cropped_length = np.minimum(\n np.maximum(int(np.floor(valid_size * p)), 64), valid_size)\n bias = np.random.randint(0, valid_size - cropped_length + 1)\n data = data[:, begin + bias:begin + bias + cropped_length, :, :]\n # resize\n data = np.transpose(data, (0, 2, 3, 1)).conjugate().reshape(\n C * V * M, cropped_length)\n data = data[None, None, :, :]\n # could perform both up sample and down sample\n data_tensor = paddle.to_tensor(data)\n data_tensor = F.interpolate(\n data_tensor,\n size=(C * V * M, self.window_size),\n mode='bilinear',\n align_corners=False).squeeze()",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:266-286"
+ },
+ "5429": {
+ "file_id": 450,
+ "content": "This code randomly selects a cropped length within a specified interval, then applies random bias to the cropped length. It reshapes and transposes the data before performing interpolation on the tensor for up or down sampling using bilinear mode.",
+ "type": "comment"
+ },
+ "5430": {
+ "file_id": 450,
+ "content": " data = paddle.transpose(\n paddle.reshape(data_tensor, (C, V, M, self.window_size)),\n (0, 3, 1, 2)).numpy()\n else:\n raise NotImplementedError\n results['data'] = data\n return results\n@PIPELINES.register()\nclass SketeonModalityTransform(object):\n \"\"\"\n Sketeon Crop Sampler.\n Args:\n crop_model: str, crop model, support: ['center'].\n p_interval: list, crop len\n window_size: int, sample windows size.\n \"\"\"\n def __init__(self, bone, motion, joint=True, graph='ntu_rgb_d'):\n self.joint = joint\n self.bone = bone\n self.motion = motion\n self.graph = graph\n if self.graph == \"ntu_rgb_d\":\n self.bone_pairs = ((1, 2), (2, 21), (3, 21), (4, 3), (5, 21),\n (6, 5), (7, 6), (8, 7), (9, 21), (10, 9),\n (11, 10), (12, 11), (13, 1), (14, 13), (15, 14),\n (16, 15), (17, 1), (18, 17), (19, 18), (20, 19),",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:287-316"
+ },
+ "5431": {
+ "file_id": 450,
+ "content": "This code is part of the PaddleVideo library and appears to be a function or class related to skeleton data transformation for video analysis tasks. The code seems to handle reshaping and transposing data based on certain parameters, such as window size, crop model, and more. This could potentially be used for video processing in computer vision applications like action recognition or pose estimation.",
+ "type": "comment"
+ },
+ "5432": {
+ "file_id": 450,
+ "content": " (22, 23), (21, 21), (23, 8), (24, 25), (25, 12))\n else:\n raise NotImplementedError\n def __call__(self, results):\n if self.joint:\n return results\n data_numpy = results['data']\n if self.bone:\n bone_data_numpy = np.zeros_like(data_numpy)\n for v1, v2 in self.bone_pairs:\n bone_data_numpy[:, :, v1 -\n 1] = data_numpy[:, :, v1 -\n 1] - data_numpy[:, :, v2 - 1]\n data_numpy = bone_data_numpy\n if self.motion:\n data_numpy[:, :-1] = data_numpy[:, 1:] - data_numpy[:, :-1]\n data_numpy[:, -1] = 0\n results['data'] = data_numpy\n return results\n@PIPELINES.register()\nclass UniformSampleFrames:\n \"\"\"Uniformly sample frames from the video.\n To sample an n-frame clip from the video. UniformSampleFrames basically\n divide the video into n segments of equal length and randomly sample one",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:317-344"
+ },
+ "5433": {
+ "file_id": 450,
+ "content": "This code defines a class for skeleton processing in PaddleVideo. If joints are enabled, it returns the results as is. If bones are enabled, it calculates bone data by subtracting corresponding bone vertices from each other. If motion is enabled, it sets the last frame's coordinates to 0. The UniformSampleFrames pipeline uniformly samples frames from a video by dividing it into equal segments and randomly selecting one from each segment.",
+ "type": "comment"
+ },
+ "5434": {
+ "file_id": 450,
+ "content": " frame from each segment. To make the testing results reproducible, a\n random seed is set during testing, to make the sampling results\n deterministic.\n Required keys are \"total_frames\", \"start_index\" , added or modified keys\n are \"frame_inds\", \"clip_len\", \"frame_interval\" and \"num_clips\".\n Args:\n clip_len (int): Frames of each sampled output clip.\n num_clips (int): Number of clips to be sampled. Default: 1.\n test_mode (bool): Store True when building test or validation dataset.\n Default: False.\n seed (int): The random seed used during test time. Default: 255.\n \"\"\"\n def __init__(self, clip_len, num_clips=1, test_mode=False, seed=255):\n self.clip_len = clip_len\n self.num_clips = num_clips\n self.test_mode = test_mode\n self.seed = seed\n def _get_train_clips(self, num_frames, clip_len):\n \"\"\"Uniformly sample indices for training clips.\n Args:\n num_frames (int): The number of frames.\n clip_len (int): The length of the clip.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:345-372"
+ },
+ "5435": {
+ "file_id": 450,
+ "content": "This code snippet defines a class with an __init__ method that initializes the clip_len, num_clips, test_mode, and seed. The _get_train_clips method uniformly samples indices for training clips based on the given number of frames and clip length. This is used in PaddleVideo for loading and processing video data.",
+ "type": "comment"
+ },
+ "5436": {
+ "file_id": 450,
+ "content": " \"\"\"\n assert self.num_clips == 1\n if num_frames < clip_len:\n start = np.random.randint(0, num_frames)\n inds = np.arange(start, start + clip_len)\n elif clip_len <= num_frames < 2 * clip_len:\n basic = np.arange(clip_len)\n inds = np.random.choice(\n clip_len + 1, num_frames - clip_len, replace=False)\n offset = np.zeros(clip_len + 1, dtype=np.int64)\n offset[inds] = 1\n offset = np.cumsum(offset)\n inds = basic + offset[:-1]\n else:\n bids = np.array(\n [i * num_frames // clip_len for i in range(clip_len + 1)])\n bsize = np.diff(bids)\n bst = bids[:clip_len]\n offset = np.random.randint(bsize)\n inds = bst + offset\n return inds\n def _get_test_clips(self, num_frames, clip_len):\n \"\"\"Uniformly sample indices for testing clips.\n Args:\n num_frames (int): The number of frames.\n clip_len (int): The length of the clip.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:373-401"
+ },
+ "5437": {
+ "file_id": 450,
+ "content": "This code determines the indices for a skeleton clip from a given number of frames and clip length. It handles three scenarios: when the number of frames is less than the clip length, between the clip length and twice the clip length, or more than twice the clip length. The function returns the sampled indices accordingly.",
+ "type": "comment"
+ },
+ "5438": {
+ "file_id": 450,
+ "content": " \"\"\"\n np.random.seed(self.seed)\n if num_frames < clip_len:\n # Then we use a simple strategy\n if num_frames < self.num_clips:\n start_inds = list(range(self.num_clips))\n else:\n start_inds = [\n i * num_frames // self.num_clips\n for i in range(self.num_clips)\n ]\n inds = np.concatenate(\n [np.arange(i, i + clip_len) for i in start_inds])\n elif clip_len <= num_frames < clip_len * 2:\n all_inds = []\n for i in range(self.num_clips):\n basic = np.arange(clip_len)\n inds = np.random.choice(\n clip_len + 1, num_frames - clip_len, replace=False)\n offset = np.zeros(clip_len + 1, dtype=np.int64)\n offset[inds] = 1\n offset = np.cumsum(offset)\n inds = basic + offset[:-1]\n all_inds.append(inds)\n inds = np.concatenate(all_inds)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:402-427"
+ },
+ "5439": {
+ "file_id": 450,
+ "content": "The code handles the random selection of frame indices for a given clip length and total number of frames. It considers three scenarios: when there are fewer frames than the clip length, exactly equal to the clip length, or between the clip length and twice the clip length. It uses list comprehension and numpy functions to generate the desired indices for each case.",
+ "type": "comment"
+ },
+ "5440": {
+ "file_id": 450,
+ "content": " else:\n bids = np.array(\n [i * num_frames // clip_len for i in range(clip_len + 1)])\n bsize = np.diff(bids)\n bst = bids[:clip_len]\n all_inds = []\n for i in range(self.num_clips):\n offset = np.random.randint(bsize)\n all_inds.append(bst + offset)\n inds = np.concatenate(all_inds)\n return inds\n def __call__(self, results):\n num_frames = results['total_frames']\n if self.test_mode:\n inds = self._get_test_clips(num_frames, self.clip_len)\n else:\n inds = self._get_train_clips(num_frames, self.clip_len)\n inds = np.mod(inds, num_frames)\n start_index = results['start_index']\n inds = inds + start_index\n results['frame_inds'] = inds.astype(np.int)\n results['clip_len'] = self.clip_len\n results['frame_interval'] = None\n results['num_clips'] = self.num_clips\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:428-459"
+ },
+ "5441": {
+ "file_id": 450,
+ "content": "This code defines a class for generating frame indices for skeleton data in PaddleVideo. It has methods to generate clips for training or testing, and returns the generated clips as results. The class takes parameters such as clip length, number of clips, total frames, etc. It ensures that the returned frame indices are within the range of total frames and converts them to integers.",
+ "type": "comment"
+ },
+ "5442": {
+ "file_id": 450,
+ "content": " f'clip_len={self.clip_len}, '\n f'num_clips={self.num_clips}, '\n f'test_mode={self.test_mode}, '\n f'seed={self.seed})')\n return repr_str\n@PIPELINES.register()\nclass PoseDecode:\n \"\"\"Load and decode pose with given indices.\n Required keys are \"keypoint\", \"frame_inds\" (optional), \"keypoint_score\"\n (optional), added or modified keys are \"keypoint\", \"keypoint_score\" (if\n applicable).\n \"\"\"\n @staticmethod\n def _load_kp(kp, frame_inds):\n \"\"\"Load keypoints given frame indices.\n Args:\n kp (np.ndarray): The keypoint coordinates.\n frame_inds (np.ndarray): The frame indices.\n \"\"\"\n return [x[frame_inds].astype(np.float32) for x in kp]\n @staticmethod\n def _load_kpscore(kpscore, frame_inds):\n \"\"\"Load keypoint scores given frame indices.\n Args:\n kpscore (np.ndarray): The confidence scores of keypoints.\n frame_inds (np.ndarray): The frame indices.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:460-493"
+ },
+ "5443": {
+ "file_id": 450,
+ "content": "This code defines a PoseDecode class that loads and decodes pose with given indices. It requires \"keypoint\" and \"frame_inds\" keys, and optionally \"keypoint_score\". The _load_kp static method loads keypoint coordinates based on frame indices, while the _load_kpscore method loads keypoint scores with frame indices. Both methods return arrays of float32 values for keypoint coordinates or scores respectively. This class is registered at PIPelines for further usage.",
+ "type": "comment"
+ },
+ "5444": {
+ "file_id": 450,
+ "content": " \"\"\"\n return [x[frame_inds].astype(np.float32) for x in kpscore]\n def __call__(self, results):\n if 'frame_inds' not in results:\n results['frame_inds'] = np.arange(results['total_frames'])\n if results['frame_inds'].ndim != 1:\n results['frame_inds'] = np.squeeze(results['frame_inds'])\n offset = results.get('offset', 0)\n frame_inds = results['frame_inds'] + offset\n if 'keypoint_score' in results:\n kpscore = results['keypoint_score']\n results['keypoint_score'] = kpscore[:, frame_inds].astype(\n np.float32)\n if 'keypoint' in results:\n results['keypoint'] = results['keypoint'][:, frame_inds].astype(\n np.float32)\n return results\n def __repr__(self):\n repr_str = f'{self.__class__.__name__}()'\n return repr_str\n@PIPELINES.register()\nclass PoseCompact:\n \"\"\"Convert the coordinates of keypoints to make it more compact.\n Specifically, it first find a tight bounding box that surrounds all joints",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:494-528"
+ },
+ "5445": {
+ "file_id": 450,
+ "content": "This code defines a PoseCompact class, which is a pipeline for converting keypoint coordinates into a more compact representation. It takes results from previous steps and processes 'keypoint_score' and 'keypoint' keys based on frame indices. If present, it extracts the keypoint scores and keypoint coordinates for the specified frames and converts them to float32 type.",
+ "type": "comment"
+ },
+ "5446": {
+ "file_id": 450,
+ "content": " in each frame, then we expand the tight box by a given padding ratio. For\n example, if 'padding == 0.25', then the expanded box has unchanged center,\n and 1.25x width and height.\n Required keys in results are \"img_shape\", \"keypoint\", add or modified keys\n are \"img_shape\", \"keypoint\", \"crop_quadruple\".\n Args:\n padding (float): The padding size. Default: 0.25.\n threshold (int): The threshold for the tight bounding box. If the width\n or height of the tight bounding box is smaller than the threshold,\n we do not perform the compact operation. Default: 10.\n hw_ratio (float | tuple[float] | None): The hw_ratio of the expanded\n box. Float indicates the specific ratio and tuple indicates a\n ratio range. If set as None, it means there is no requirement on\n hw_ratio. Default: None.\n allow_imgpad (bool): Whether to allow expanding the box outside the\n image to meet the hw_ratio requirement. Default: True.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:529-546"
+ },
+ "5447": {
+ "file_id": 450,
+ "content": "This function expands tight bounding boxes by a given padding ratio and adds new key \"crop_quadruple\". It requires keys \"img_shape\", \"keypoint\" and may modify them. The threshold determines if the box is too small to expand, hw_ratio sets the box aspect ratio (optional), and allow_imgpad allows expanding outside image for hw_ratio (optional). Default values are padding=0.25, threshold=10, hw_ratio=None, allow_imgpad=True.",
+ "type": "comment"
+ },
+ "5448": {
+ "file_id": 450,
+ "content": " Returns:\n type: Description of returned object.\n \"\"\"\n def __init__(self,\n padding=0.25,\n threshold=10,\n hw_ratio=None,\n allow_imgpad=True):\n self.padding = padding\n self.threshold = threshold\n if hw_ratio is not None:\n hw_ratio = _pair(hw_ratio)\n self.hw_ratio = hw_ratio\n self.allow_imgpad = allow_imgpad\n assert self.padding >= 0\n def _combine_quadruple(self, a, b):\n return (a[0] + a[2] * b[0], a[1] + a[3] * b[1], a[2] * b[2],\n a[3] * b[3])\n def __call__(self, results):\n img_shape = results['img_shape']\n h, w = img_shape\n kp = results['keypoint']\n # Make NaN zero\n kp[np.isnan(kp)] = 0.\n kp_x = kp[..., 0]\n kp_y = kp[..., 1]\n min_x = np.min(kp_x[kp_x != 0], initial=np.Inf)\n min_y = np.min(kp_y[kp_y != 0], initial=np.Inf)\n max_x = np.max(kp_x[kp_x != 0], initial=-np.Inf)\n max_y = np.max(kp_y[kp_y != 0], initial=-np.Inf)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:548-585"
+ },
+ "5449": {
+ "file_id": 450,
+ "content": "This code is initializing a class for skeleton pipeline. It takes parameters such as padding, threshold, hw_ratio, and allow_imgpad. The class also has methods to combine quadruples, apply transformations, and handle keypoints in the image. The code performs various operations like making NaN values zero, finding minimum and maximum keypoint coordinates, and applying padding if needed.",
+ "type": "comment"
+ },
+ "5450": {
+ "file_id": 450,
+ "content": " # The compact area is too small\n if max_x - min_x < self.threshold or max_y - min_y < self.threshold:\n return results\n center = ((max_x + min_x) / 2, (max_y + min_y) / 2)\n half_width = (max_x - min_x) / 2 * (1 + self.padding)\n half_height = (max_y - min_y) / 2 * (1 + self.padding)\n if self.hw_ratio is not None:\n half_height = max(self.hw_ratio[0] * half_width, half_height)\n half_width = max(1 / self.hw_ratio[1] * half_height, half_width)\n min_x, max_x = center[0] - half_width, center[0] + half_width\n min_y, max_y = center[1] - half_height, center[1] + half_height\n # hot update\n if not self.allow_imgpad:\n min_x, min_y = int(max(0, min_x)), int(max(0, min_y))\n max_x, max_y = int(min(w, max_x)), int(min(h, max_y))\n else:\n min_x, min_y = int(min_x), int(min_y)\n max_x, max_y = int(max_x), int(max_y)\n kp_x[kp_x != 0] -= min_x\n kp_y[kp_y != 0] -= min_y",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:587-611"
+ },
+ "5451": {
+ "file_id": 450,
+ "content": "This code checks if the compact area is too small and adjusts the bounding box parameters accordingly. It calculates the center, half-width, and half-height of the bounding box. If the aspect ratio should be maintained (hw_ratio), it ensures that by adjusting half_height based on half_width. The code then updates the minimum and maximum x and y values within the constraints of the image's width and height, unless allow_imgpad is True, in which case it doesn't limit the bounding box size. Finally, it adjusts the x and y coordinates of the keypoints by subtracting the new min_x and min_y to maintain their relative positions within the adjusted bounding box.",
+ "type": "comment"
+ },
+ "5452": {
+ "file_id": 450,
+ "content": " new_shape = (max_y - min_y, max_x - min_x)\n results['img_shape'] = new_shape\n # the order is x, y, w, h (in [0, 1]), a tuple\n crop_quadruple = results.get('crop_quadruple', (0., 0., 1., 1.))\n new_crop_quadruple = (min_x / w, min_y / h, (max_x - min_x) / w,\n (max_y - min_y) / h)\n crop_quadruple = self._combine_quadruple(crop_quadruple,\n new_crop_quadruple)\n results['crop_quadruple'] = crop_quadruple\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}(padding={self.padding}, '\n f'threshold={self.threshold}, '\n f'hw_ratio={self.hw_ratio}, '\n f'allow_imgpad={self.allow_imgpad})')\n return repr_str\nclass CropBase:\n @staticmethod\n def _crop_kps(kps, crop_bbox):\n return kps - crop_bbox[:2]\n @staticmethod\n def _crop_imgs(imgs, crop_bbox):\n x1, y1, x2, y2 = crop_bbox",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:613-640"
+ },
+ "5453": {
+ "file_id": 450,
+ "content": "This code segment is part of a pipeline for skeleton detection in images. It calculates the new image shape based on the cropping region, updates the 'crop_quadruple' in the results dictionary, and defines two static methods for cropping keypoints (_crop_kps) and cropped images (_crop_imgs). The CropBase class provides functionality to crop keypoints based on the provided crop region.",
+ "type": "comment"
+ },
+ "5454": {
+ "file_id": 450,
+ "content": " return [img[y1:y2, x1:x2] for img in imgs]\n @staticmethod\n def _box_crop(box, crop_bbox):\n \"\"\"Crop the bounding boxes according to the crop_bbox.\n Args:\n box (np.ndarray): The bounding boxes.\n crop_bbox(np.ndarray): The bbox used to crop the original image.\n \"\"\"\n x1, y1, x2, y2 = crop_bbox\n img_w, img_h = x2 - x1, y2 - y1\n box_ = box.copy()\n box_[..., 0::2] = np.clip(box[..., 0::2] - x1, 0, img_w - 1)\n box_[..., 1::2] = np.clip(box[..., 1::2] - y1, 0, img_h - 1)\n return box_\n def _all_box_crop(self, results, crop_bbox):\n \"\"\"Crop the gt_bboxes and proposals in results according to crop_bbox.\n Args:\n results (dict): All information about the sample, which contain\n 'gt_bboxes' and 'proposals' (optional).\n crop_bbox(np.ndarray): The bbox used to crop the original image.\n \"\"\"\n results['gt_bboxes'] = self._box_crop(results['gt_bboxes'], crop_bbox)\n if 'proposals' in results and results['proposals'] is not None:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:641-669"
+ },
+ "5455": {
+ "file_id": 450,
+ "content": "This code defines a function `_all_box_crop` that crops the gt_bboxes and proposals in results according to the crop_bbox. It first applies the `_box_crop` function to 'gt_bboxes', then if 'proposals' are present and not None, it also applies the `_box_crop` function to them. The `_box_crop` function crops bounding boxes by subtracting the x1, y1 coordinates from their x and y values respectively, ensuring they fall within the new image dimensions.",
+ "type": "comment"
+ },
+ "5456": {
+ "file_id": 450,
+ "content": " assert results['proposals'].shape[1] == 4\n results['proposals'] = self._box_crop(results['proposals'],\n crop_bbox)\n return results\n def __call__(self, results):\n raise NotImplementedError\n@PIPELINES.register()\nclass RandomResizedCrop_V2(CropBase):\n \"\"\"Random crop that specifics the area and height-weight ratio range.\n Required keys in results are \"img_shape\", \"crop_bbox\", \"imgs\" (optional),\n \"keypoint\" (optional), added or modified keys are \"imgs\", \"keypoint\",\n \"crop_bbox\" and \"lazy\"; Required keys in \"lazy\" are \"flip\", \"crop_bbox\",\n added or modified key is \"crop_bbox\".\n Args:\n area_range (Tuple[float]): The candidate area scales range of\n output cropped images. Default: (0.08, 1.0).\n aspect_ratio_range (Tuple[float]): The candidate aspect ratio range of\n output cropped images. Default: (3 / 4, 4 / 3).\n lazy (bool): Determine whether to apply lazy operation. Default: False.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:670-693"
+ },
+ "5457": {
+ "file_id": 450,
+ "content": "This code defines a RandomResizedCrop_V2 pipeline that randomly crops an image to a specified area and height-weight ratio range. The required keys in results are \"img_shape\", \"crop_bbox\", and \"imgs\" (optional). The modified keys are \"imgs\", \"keypoint\", \"crop_bbox\", and \"lazy\". The required keys in \"lazy\" are \"flip\", \"crop_bbox\". It provides an area range and aspect ratio range for the output cropped images.",
+ "type": "comment"
+ },
+ "5458": {
+ "file_id": 450,
+ "content": " \"\"\"\n def __init__(self,\n area_range=(0.08, 1.0),\n aspect_ratio_range=(3 / 4, 4 / 3),\n lazy=False):\n self.area_range = eval(area_range)\n self.aspect_ratio_range = aspect_ratio_range\n self.lazy = lazy\n if not is_tuple_of(self.area_range, float):\n raise TypeError(f'Area_range must be a tuple of float, '\n f'but got {type(area_range)}')\n if not is_tuple_of(self.aspect_ratio_range, float):\n raise TypeError(f'Aspect_ratio_range must be a tuple of float, '\n f'but got {type(aspect_ratio_range)}')\n @staticmethod\n def get_crop_bbox(img_shape,\n area_range,\n aspect_ratio_range,\n max_attempts=10):\n \"\"\"Get a crop bbox given the area range and aspect ratio range.\n Args:\n img_shape (Tuple[int]): Image shape\n area_range (Tuple[float]): The candidate area scales range of",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:694-719"
+ },
+ "5459": {
+ "file_id": 450,
+ "content": "This code initializes a class with area_range, aspect_ratio_range, and lazy parameters. It checks if the ranges are tuples of floats, and raises TypeError if not. The get_crop_bbox static method takes image shape, area range, aspect ratio range, and max attempts as arguments to return a crop bounding box.",
+ "type": "comment"
+ },
+ "5460": {
+ "file_id": 450,
+ "content": " output cropped images. Default: (0.08, 1.0).\n aspect_ratio_range (Tuple[float]): The candidate aspect\n ratio range of output cropped images. Default: (3 / 4, 4 / 3).\n max_attempts (int): The maximum of attempts. Default: 10.\n max_attempts (int): Max attempts times to generate random candidate\n bounding box. If it doesn't qualified one, the center bounding\n box will be used.\n Returns:\n (list[int]) A random crop bbox within the area range and aspect\n ratio range.\n \"\"\"\n assert 0 < area_range[0] <= area_range[1] <= 1\n assert 0 < aspect_ratio_range[0] <= aspect_ratio_range[1]\n img_h, img_w = img_shape\n area = img_h * img_w\n min_ar, max_ar = aspect_ratio_range\n aspect_ratios = np.exp(\n np.random.uniform(\n np.log(min_ar), np.log(max_ar), size=max_attempts))\n target_areas = np.random.uniform(*area_range, size=max_attempts) * area",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:720-741"
+ },
+ "5461": {
+ "file_id": 450,
+ "content": "This function generates a random crop bounding box within a specified area range and aspect ratio range. It takes image shape, area range, and aspect ratio range as input parameters. The function first checks the validity of the ranges, then calculates the image's total area, minimum and maximum aspect ratios from the aspect ratio range. It uses numpy to generate a list of random candidate bounding box aspect ratios and target areas within the specified ranges.",
+ "type": "comment"
+ },
+ "5462": {
+ "file_id": 450,
+ "content": " candidate_crop_w = np.round(np.sqrt(\n target_areas * aspect_ratios)).astype(np.int32)\n candidate_crop_h = np.round(np.sqrt(\n target_areas / aspect_ratios)).astype(np.int32)\n for i in range(max_attempts):\n crop_w = candidate_crop_w[i]\n crop_h = candidate_crop_h[i]\n if crop_h <= img_h and crop_w <= img_w:\n x_offset = random.randint(0, img_w - crop_w)\n y_offset = random.randint(0, img_h - crop_h)\n return x_offset, y_offset, x_offset + crop_w, y_offset + crop_h\n # Fallback\n crop_size = min(img_h, img_w)\n x_offset = (img_w - crop_size) // 2\n y_offset = (img_h - crop_size) // 2\n return x_offset, y_offset, x_offset + crop_size, y_offset + crop_size\n def __call__(self, results):\n \"\"\"Performs the RandomResizeCrop augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:742-766"
+ },
+ "5463": {
+ "file_id": 450,
+ "content": "This code calculates random crop sizes based on the aspect ratios and target areas, then attempts to find a suitable crop region within the image. If a suitable crop is found, it returns the offsets and dimensions of that crop. If not, it falls back to a centered crop with minimum size. This function is called as part of a pipeline for image augmentation.",
+ "type": "comment"
+ },
+ "5464": {
+ "file_id": 450,
+ "content": " \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n if 'keypoint' in results:\n assert not self.lazy, ('Keypoint Augmentations are not compatible '\n 'with lazy == True')\n img_h, img_w = results['img_shape']\n left, top, right, bottom = self.get_crop_bbox(\n (img_h, img_w), self.area_range, self.aspect_ratio_range)\n new_h, new_w = bottom - top, right - left\n if 'crop_quadruple' not in results:\n results['crop_quadruple'] = np.array(\n [0, 0, 1, 1], # x, y, w, h\n dtype=np.float32)\n x_ratio, y_ratio = left / img_w, top / img_h\n w_ratio, h_ratio = new_w / img_w, new_h / img_h\n old_crop_quadruple = results['crop_quadruple']\n old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]\n old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]\n new_crop_quadruple = [\n old_x_ratio + x_ratio * old_w_ratio,",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:767-791"
+ },
+ "5465": {
+ "file_id": 450,
+ "content": "This code initializes and adjusts the crop quadruple of an image based on its aspect ratio, area range, and size. It ensures that the 'keypoint' is not applied if lazy augmentation is enabled. The crop quadruple contains x, y, width, and height values representing the image's cropping region, which are updated according to the original image's dimensions and the desired aspect ratio range.",
+ "type": "comment"
+ },
+ "5466": {
+ "file_id": 450,
+ "content": " old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,\n h_ratio * old_h_ratio\n ]\n results['crop_quadruple'] = np.array(\n new_crop_quadruple, dtype=np.float32)\n crop_bbox = np.array([left, top, right, bottom])\n results['crop_bbox'] = crop_bbox\n results['img_shape'] = (new_h, new_w)\n if not self.lazy:\n if 'keypoint' in results:\n results['keypoint'] = self._crop_kps(results['keypoint'],\n crop_bbox)\n if 'imgs' in results:\n results['imgs'] = self._crop_imgs(results['imgs'], crop_bbox)\n else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Put Flip at last for now')\n # record crop_bbox in lazyop dict to ensure only crop once in Fuse\n lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']\n left = left * (lazy_right - lazy_left) / img_w",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:792-815"
+ },
+ "5467": {
+ "file_id": 450,
+ "content": "This code performs cropping on images and bboxes based on given ratios. It updates the results dictionary with cropped quadruple, crop bbox, and new image shape. If not in lazy mode, it crops keypoints and images using these values. If in lazy mode, it stores the left, top, right, and bottom crop positions for later fusion.",
+ "type": "comment"
+ },
+ "5468": {
+ "file_id": 450,
+ "content": " right = right * (lazy_right - lazy_left) / img_w\n top = top * (lazy_bottom - lazy_top) / img_h\n bottom = bottom * (lazy_bottom - lazy_top) / img_h\n lazyop['crop_bbox'] = np.array(\n [(lazy_left + left), (lazy_top + top), (lazy_left + right),\n (lazy_top + bottom)],\n dtype=np.float32)\n if 'gt_bboxes' in results:\n assert not self.lazy\n results = self._all_box_crop(results, results['crop_bbox'])\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'area_range={self.area_range}, '\n f'aspect_ratio_range={self.aspect_ratio_range}, '\n f'lazy={self.lazy})')\n return repr_str\ndef is_seq_of(seq, expected_type, seq_type=None):\n \"\"\"Check whether it is a sequence of some type.\n Args:\n seq (Sequence): The sequence to be checked.\n expected_type (type): Expected type of sequence items.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:816-843"
+ },
+ "5469": {
+ "file_id": 450,
+ "content": "This code is a pipeline for skeleton processing in PaddleVideo. It scales the bounding box and applies it to the lazy operation, performs cropping based on the new bounding box, and if 'gt_bboxes' is present in results, it crops all other boxes accordingly. The class also has a __repr__ method for string representation. There is also a helper function, is_seq_of, which checks whether a sequence contains items of a specific type.",
+ "type": "comment"
+ },
+ "5470": {
+ "file_id": 450,
+ "content": " seq_type (type, optional): Expected sequence type.\n Returns:\n bool: Whether the sequence is valid.\n \"\"\"\n if seq_type is None:\n exp_seq_type = abc.Sequence\n else:\n assert isinstance(seq_type, type)\n exp_seq_type = seq_type\n if not isinstance(seq, exp_seq_type):\n return False\n for item in seq:\n if not isinstance(item, expected_type):\n return False\n return True\ndef is_tuple_of(seq, expected_type):\n \"\"\"Check whether it is a tuple of some type.\n A partial method of :func:`is_seq_of`.\n \"\"\"\n return is_seq_of(seq, expected_type, seq_type=tuple)\n@PIPELINES.register()\nclass CenterCrop_V2(CropBase):\n \"\"\"Crop the center area from images.\n Required keys are \"img_shape\", \"imgs\" (optional), \"keypoint\" (optional),\n added or modified keys are \"imgs\", \"keypoint\", \"crop_bbox\", \"lazy\" and\n \"img_shape\". Required keys in \"lazy\" is \"crop_bbox\", added or modified key\n is \"crop_bbox\".\n Args:\n crop_size (int | tuple[int]): (w, h) of crop size.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:844-880"
+ },
+ "5471": {
+ "file_id": 450,
+ "content": "The code defines a function `is_seq_of` that checks if a given sequence is of the expected type. It also defines two partial methods, `is_tuple_of`, which uses `is_seq_of` to check if a sequence is a tuple of a certain type. Lastly, it registers a new pipeline `CenterCrop_V2` for cropping the center area from images with required keys \"img_shape\", \"imgs\" (optional), and modified or added keys as mentioned in the function description.",
+ "type": "comment"
+ },
+ "5472": {
+ "file_id": 450,
+ "content": " lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n def __init__(self, crop_size, lazy=False):\n self.crop_size = _pair(crop_size)\n self.lazy = lazy\n if not is_tuple_of(self.crop_size, int):\n raise TypeError(f'Crop_size must be int or tuple of int, '\n f'but got {type(crop_size)}')\n def __call__(self, results):\n \"\"\"Performs the CenterCrop augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n if 'keypoint' in results:\n assert not self.lazy, ('Keypoint Augmentations are not compatible '\n 'with lazy == True')\n img_h, img_w = results['img_shape']\n crop_w, crop_h = self.crop_size\n left = (img_w - crop_w) // 2\n top = (img_h - crop_h) // 2\n right = left + crop_w\n bottom = top + crop_h",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:881-909"
+ },
+ "5473": {
+ "file_id": 450,
+ "content": "This code defines a class for CenterCrop augmentation. It initializes with crop size and lazy operation flag, checks the validity of input parameters, performs CenterCrop operation on images, and handles keypoint augmentations if present in results dictionary.",
+ "type": "comment"
+ },
+ "5474": {
+ "file_id": 450,
+ "content": " new_h, new_w = bottom - top, right - left\n crop_bbox = np.array([left, top, right, bottom])\n results['crop_bbox'] = crop_bbox\n results['img_shape'] = (new_h, new_w)\n if 'crop_quadruple' not in results:\n results['crop_quadruple'] = np.array(\n [0, 0, 1, 1], # x, y, w, h\n dtype=np.float32)\n x_ratio, y_ratio = left / img_w, top / img_h\n w_ratio, h_ratio = new_w / img_w, new_h / img_h\n old_crop_quadruple = results['crop_quadruple']\n old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]\n old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]\n new_crop_quadruple = [\n old_x_ratio + x_ratio * old_w_ratio,\n old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,\n h_ratio * old_h_ratio\n ]\n results['crop_quadruple'] = np.array(\n new_crop_quadruple, dtype=np.float32)\n if not self.lazy:\n if 'keypoint' in results:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:910-936"
+ },
+ "5475": {
+ "file_id": 450,
+ "content": "This code calculates the new image shape and crop box coordinates based on the provided top, left, right, and bottom values. It then updates the 'crop_bbox' and 'img_shape' in the results dictionary. If 'crop_quadruple' is not already present in the results, it creates and appends it. The code then calculates new crop quadruple coordinates by adjusting the old ones with the ratios of the original and new image widths and heights. Finally, if 'keypoint' is present in the results, the code proceeds further (presumably for lazy mode).",
+ "type": "comment"
+ },
+ "5476": {
+ "file_id": 450,
+ "content": " results['keypoint'] = self._crop_kps(results['keypoint'],\n crop_bbox)\n if 'imgs' in results:\n results['imgs'] = self._crop_imgs(results['imgs'], crop_bbox)\n else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Put Flip at last for now')\n # record crop_bbox in lazyop dict to ensure only crop once in Fuse\n lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']\n left = left * (lazy_right - lazy_left) / img_w\n right = right * (lazy_right - lazy_left) / img_w\n top = top * (lazy_bottom - lazy_top) / img_h\n bottom = bottom * (lazy_bottom - lazy_top) / img_h\n lazyop['crop_bbox'] = np.array(\n [(lazy_left + left), (lazy_top + top), (lazy_left + right),\n (lazy_top + bottom)],\n dtype=np.float32)\n if 'gt_bboxes' in results:",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:937-957"
+ },
+ "5477": {
+ "file_id": 450,
+ "content": "This code is handling the case where 'lazyop' contains a flip operation. It records crop_bbox in lazyop to ensure only one crop operation is performed in Fuse. If 'gt_bboxes' is present in results, it indicates ground truth bbox information.",
+ "type": "comment"
+ },
+ "5478": {
+ "file_id": 450,
+ "content": " assert not self.lazy\n results = self._all_box_crop(results, results['crop_bbox'])\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}(crop_size={self.crop_size}, '\n f'lazy={self.lazy})')\n return repr_str\n@PIPELINES.register()\nclass Flip_V2:\n \"\"\"Flip the input images with a probability.\n Reverse the order of elements in the given imgs with a specific direction.\n The shape of the imgs is preserved, but the elements are reordered.\n Required keys are \"img_shape\", \"modality\", \"imgs\" (optional), \"keypoint\"\n (optional), added or modified keys are \"imgs\", \"keypoint\", \"lazy\" and\n \"flip_direction\". Required keys in \"lazy\" is None, added or modified key\n are \"flip\" and \"flip_direction\". The Flip augmentation should be placed\n after any cropping / reshaping augmentations, to make sure crop_quadruple\n is calculated properly.\n Args:\n flip_ratio (float): Probability of implementing flip. Default: 0.5.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:958-984"
+ },
+ "5479": {
+ "file_id": 450,
+ "content": "This code snippet is registering a pipeline class called \"Flip_V2\". The Flip_V2 class flips the input images with a probability and reverses the order of elements in the given imgs with a specific direction. It requires keys such as \"img_shape\", \"modality\", and \"imgs\" while adding or modifying keys like \"imgs\", \"keypoint\", \"lazy\", and \"flip_direction\". Flip_V2 should be placed after cropping/reshaping augmentations to ensure crop_quadruple is calculated properly. The flip ratio, which determines the probability of implementing flip, is set to 0.5 by default.",
+ "type": "comment"
+ },
+ "5480": {
+ "file_id": 450,
+ "content": " direction (str): Flip imgs horizontally or vertically. Options are\n \"horizontal\" | \"vertical\". Default: \"horizontal\".\n flip_label_map (Dict[int, int] | None): Transform the label of the\n flipped image with the specific label. Default: None.\n left_kp (list[int]): Indexes of left keypoints, used to flip keypoints.\n Default: None.\n right_kp (list[ind]): Indexes of right keypoints, used to flip\n keypoints. Default: None.\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n _directions = ['horizontal', 'vertical']\n def __init__(self,\n flip_ratio=0.5,\n direction='horizontal',\n flip_label_map=None,\n left_kp=None,\n right_kp=None,\n lazy=False):\n if direction not in self._directions:\n raise ValueError(f'Direction {direction} is not supported. '\n f'Currently support ones are {self._directions}')",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:985-1006"
+ },
+ "5481": {
+ "file_id": 450,
+ "content": "This code snippet is for an object called \"SkeletonPipeline\". It has parameters for direction, flip_label_map, left_kp, right_kp, and lazy. The direction parameter can be either 'horizontal' or 'vertical'. Flip_label_map is a dictionary used to transform the label of flipped images. Left_kp and right_kp are indexes used for flipping keypoints. Lazy determines whether to apply lazy operations, default set to False. The function checks if the given direction is within ['horizontal', 'vertical'].",
+ "type": "comment"
+ },
+ "5482": {
+ "file_id": 450,
+ "content": " self.flip_ratio = flip_ratio\n self.direction = direction\n self.flip_label_map = flip_label_map\n self.left_kp = left_kp\n self.right_kp = right_kp\n self.lazy = lazy\n def _flip_imgs(self, imgs, modality):\n _ = [imflip_(img, self.direction) for img in imgs]\n lt = len(imgs)\n if modality == 'Flow':\n # The 1st frame of each 2 frames is flow-x\n for i in range(0, lt, 2):\n imgs[i] = iminvert(imgs[i])\n return imgs\n def _flip_kps(self, kps, kpscores, img_width):\n kp_x = kps[..., 0]\n kp_x[kp_x != 0] = img_width - kp_x[kp_x != 0]\n new_order = list(range(kps.shape[2]))\n if self.left_kp is not None and self.right_kp is not None:\n for left, right in zip(self.left_kp, self.right_kp):\n new_order[left] = right\n new_order[right] = left\n kps = kps[:, :, new_order]\n if kpscores is not None:\n kpscores = kpscores[:, :, new_order]\n return kps, kpscores",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1007-1034"
+ },
+ "5483": {
+ "file_id": 450,
+ "content": "This code is a part of the skeleton_pipeline module in PaddleVideo. It initializes parameters such as flip ratio, direction, and flips label map, and defines a function _flip_imgs for image flipping and another function _flip_kps for keypoints flipping based on the direction provided. The code also includes conditions to handle flow images specifically by inverting the first frame of each two frames.",
+ "type": "comment"
+ },
+ "5484": {
+ "file_id": 450,
+ "content": " @staticmethod\n def _box_flip(box, img_width):\n \"\"\"Flip the bounding boxes given the width of the image.\n Args:\n box (np.ndarray): The bounding boxes.\n img_width (int): The img width.\n \"\"\"\n box_ = box.copy()\n box_[..., 0::4] = img_width - box[..., 2::4]\n box_[..., 2::4] = img_width - box[..., 0::4]\n return box_\n def __call__(self, results):\n \"\"\"Performs the Flip augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n if 'keypoint' in results:\n assert not self.lazy, ('Keypoint Augmentations are not compatible '\n 'with lazy == True')\n assert self.direction == 'horizontal', (\n 'Only horizontal flips are'\n 'supported for human keypoints')\n modality = results['modality']\n if modality == 'Flow':",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1036-1065"
+ },
+ "5485": {
+ "file_id": 450,
+ "content": "This code snippet is from the PaddleVideo library's skeleton pipeline. It defines a function for flipping bounding boxes and a method that applies horizontal flip augmentation to images, unless the image contains keypoints where only horizontal flip is supported. The code checks if the modality of the image is 'Flow'.",
+ "type": "comment"
+ },
+ "5486": {
+ "file_id": 450,
+ "content": " assert self.direction == 'horizontal'\n flip = np.random.rand() < self.flip_ratio\n results['flip'] = flip\n results['flip_direction'] = self.direction\n img_width = results['img_shape'][1]\n if self.flip_label_map is not None and flip:\n results['label'] = self.flip_label_map.get(results['label'],\n results['label'])\n if not self.lazy:\n if flip:\n if 'imgs' in results:\n results['imgs'] = self._flip_imgs(results['imgs'], modality)\n if 'keypoint' in results:\n kp = results['keypoint']\n kpscore = results.get('keypoint_score', None)\n kp, kpscore = self._flip_kps(kp, kpscore, img_width)\n results['keypoint'] = kp\n if 'keypoint_score' in results:\n results['keypoint_score'] = kpscore\n else:\n lazyop = results['lazy']",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1066-1090"
+ },
+ "5487": {
+ "file_id": 450,
+ "content": "The code checks if the direction is horizontal, flips the image randomly based on a flip ratio, and updates results accordingly. If the flip label map is not None and flip occurs, it updates the label in the results. If lazy is not set, it flips images and keypoints if necessary, updating results. Otherwise, it stores the operation for later execution.",
+ "type": "comment"
+ },
+ "5488": {
+ "file_id": 450,
+ "content": " if lazyop['flip']:\n raise NotImplementedError('Use one Flip please')\n lazyop['flip'] = flip\n lazyop['flip_direction'] = self.direction\n if 'gt_bboxes' in results and flip:\n assert not self.lazy and self.direction == 'horizontal'\n width = results['img_shape'][1]\n results['gt_bboxes'] = self._box_flip(results['gt_bboxes'], width)\n if 'proposals' in results and results['proposals'] is not None:\n assert results['proposals'].shape[1] == 4\n results['proposals'] = self._box_flip(results['proposals'],\n width)\n return results\n def __repr__(self):\n repr_str = (\n f'{self.__class__.__name__}('\n f'flip_ratio={self.flip_ratio}, direction={self.direction}, '\n f'flip_label_map={self.flip_label_map}, lazy={self.lazy})')\n return repr_str\n@PIPELINES.register()\nclass FormatShape:\n \"\"\"Format final imgs shape to the given input_format.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1091-1117"
+ },
+ "5489": {
+ "file_id": 450,
+ "content": "This code snippet is part of the \"SkeletonPipeline\" class in PaddleVideo. It checks if the 'flip' parameter is set and applies horizontal flipping to the 'gt_bboxes' and 'proposals' (if present) based on the direction specified. It also defines a __repr__ method for the class, providing information about its attributes, and registers a new pipeline called \"FormatShape\" for use in the system.",
+ "type": "comment"
+ },
+ "5490": {
+ "file_id": 450,
+ "content": " Required keys are \"imgs\", \"num_clips\" and \"clip_len\", added or modified\n keys are \"imgs\" and \"input_shape\".\n Args:\n input_format (str): Define the final imgs format.\n collapse (bool): To collpase input_format N... to ... (NCTHW to CTHW,\n etc.) if N is 1. Should be set as True when training and testing\n detectors. Default: False.\n \"\"\"\n def __init__(self, input_format, collapse=False):\n self.input_format = input_format\n self.collapse = collapse\n if self.input_format not in ['NCTHW', 'NCHW', 'NCHW_Flow', 'NPTCHW']:\n raise ValueError(\n f'The input format {self.input_format} is invalid.')\n def __call__(self, results):\n \"\"\"Performs the FormatShape formating.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n if not isinstance(results['imgs'], np.ndarray):\n results['imgs'] = np.array(results['imgs'])",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1119-1144"
+ },
+ "5491": {
+ "file_id": 450,
+ "content": "The code defines a class for formatting image data in a specific format based on the input_format parameter. The class takes an input_format and a collapse boolean argument, and checks if the input_format is valid (options are 'NCTHW', 'NCHW', 'NCHW_Flow', or 'NPTCHW'). If results['imgs'] is not of type np.ndarray, it converts it to np.ndarray.",
+ "type": "comment"
+ },
+ "5492": {
+ "file_id": 450,
+ "content": " imgs = results['imgs']\n # [M x H x W x C]\n # M = 1 * N_crops * N_clips * L\n if self.collapse:\n assert results['num_clips'] == 1\n if self.input_format == 'NCTHW':\n num_clips = results['num_clips']\n clip_len = results['clip_len']\n imgs = imgs.reshape((-1, num_clips, clip_len) + imgs.shape[1:])\n # N_crops x N_clips x L x H x W x C\n imgs = np.transpose(imgs, (0, 1, 5, 2, 3, 4))\n # N_crops x N_clips x C x L x H x W\n imgs = imgs.reshape((-1, ) + imgs.shape[2:])\n # M' x C x L x H x W\n # M' = N_crops x N_clips\n elif self.input_format == 'NCHW':\n imgs = np.transpose(imgs, (0, 3, 1, 2))\n # M x C x H x W\n elif self.input_format == 'NCHW_Flow':\n num_clips = results['num_clips']\n clip_len = results['clip_len']\n imgs = imgs.reshape((-1, num_clips, clip_len) + imgs.shape[1:])\n # N_crops x N_clips x L x H x W x C",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1145-1169"
+ },
+ "5493": {
+ "file_id": 450,
+ "content": "This code is a part of the SkeletonPipeline class in PaddleVideo. It processes images from results and reshapes them based on the input format specified. If input_format is 'NCTHW', it transposes and reshapes the images accordingly, if it's 'NCHW', it only transposes, and if it's 'NCHW_Flow', it also performs reshaping similar to 'NCTHW'. The 'collapse' check ensures that if results have multiple clips, it won't collapse them.",
+ "type": "comment"
+ },
+ "5494": {
+ "file_id": 450,
+ "content": " imgs = np.transpose(imgs, (0, 1, 2, 5, 3, 4))\n # N_crops x N_clips x L x C x H x W\n imgs = imgs.reshape((-1, imgs.shape[2] * imgs.shape[3]) +\n imgs.shape[4:])\n # M' x C' x H x W\n # M' = N_crops x N_clips\n # C' = L x C\n elif self.input_format == 'NPTCHW':\n num_proposals = results['num_proposals']\n num_clips = results['num_clips']\n clip_len = results['clip_len']\n imgs = imgs.reshape((num_proposals, num_clips * clip_len) +\n imgs.shape[1:])\n # P x M x H x W x C\n # M = N_clips x L\n imgs = np.transpose(imgs, (0, 1, 4, 2, 3))\n # P x M x C x H x W\n if self.collapse:\n assert imgs.shape[0] == 1\n imgs = imgs.squeeze(0)\n results['imgs'] = imgs\n results['input_shape'] = imgs.shape\n return results\n def __repr__(self):\n repr_str = self.__class__.__name__",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1170-1197"
+ },
+ "5495": {
+ "file_id": 450,
+ "content": "This code transforms image data into various formats depending on the input format specified. It supports 'NHWC', 'NCHW', and 'NPTCHW' formats. If the collapse parameter is True, it squeezes the first dimension of the images array. The results dictionary is updated with the transformed images and their shape.",
+ "type": "comment"
+ },
+ "5496": {
+ "file_id": 450,
+ "content": " repr_str += f\"(input_format='{self.input_format}')\"\n return repr_str\n@PIPELINES.register()\nclass Collect:\n \"\"\"Collect data from the loader relevant to the specific task.\n This keeps the items in ``keys`` as it is, and collect items in\n ``meta_keys`` into a meta item called ``meta_name``.This is usually\n the last stage of the data loader pipeline.\n For example, when keys='imgs', meta_keys=('filename', 'label',\n 'original_shape'), meta_name='img_metas', the results will be a dict with\n keys 'imgs' and 'img_metas', where 'img_metas' is a DataContainer of\n another dict with keys 'filename', 'label', 'original_shape'.\n Args:\n keys (Sequence[str]): Required keys to be collected.\n meta_name (str): The name of the key that contains meta infomation.\n This key is always populated. Default: \"img_metas\".\n meta_keys (Sequence[str]): Keys that are collected under meta_name.\n The contents of the ``meta_name`` dictionary depends on\n ``meta_keys``.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1198-1220"
+ },
+ "5497": {
+ "file_id": 450,
+ "content": "The code defines a Pipeline class called \"Collect\" that collects specific data from the loader relevant to the task. It keeps keys as is and gathers items in meta_keys into a meta item called meta_name, typically used as the last stage of the data loader pipeline. The Collect class takes keys, meta_name, and meta_keys as arguments, with default values provided for meta_name.",
+ "type": "comment"
+ },
+ "5498": {
+ "file_id": 450,
+ "content": " By default this includes:\n - \"filename\": path to the image file\n - \"label\": label of the image file\n - \"original_shape\": original shape of the image as a tuple\n (h, w, c)\n - \"img_shape\": shape of the image input to the network as a tuple\n (h, w, c). Note that images may be zero padded on the\n bottom/right, if the batch tensor is larger than this shape.\n - \"pad_shape\": image shape after padding\n - \"flip_direction\": a str in (\"horiziontal\", \"vertival\") to\n indicate if the image is fliped horizontally or vertically.\n - \"img_norm_cfg\": a dict of normalization information:\n - mean - per channel mean subtraction\n - std - per channel std divisor\n - to_rgb - bool indicating if bgr was converted to rgb\n nested (bool): If set as True, will apply data[x] = [data[x]] to all\n items in data. The arg is added for compatibility. Default: False.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1221-1238"
+ },
+ "5499": {
+ "file_id": 450,
+ "content": "This code defines a dictionary containing default parameters for image data loading. It includes fields such as \"filename\", \"label\", \"original_shape\", \"img_shape\", \"pad_shape\", \"flip_direction\", and \"img_norm_cfg\". The \"nested\" argument determines whether these parameters should be applied recursively to all items within the data dictionary.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/55.json b/docs/data/55.json
new file mode 100644
index 000000000..4e1f533af
--- /dev/null
+++ b/docs/data/55.json
@@ -0,0 +1,545 @@
+{
+ "5500": {
+ "file_id": 450,
+ "content": " \"\"\"\n def __init__(self,\n keys,\n meta_keys=('filename', 'label', 'original_shape', 'img_shape',\n 'pad_shape', 'flip_direction', 'img_norm_cfg'),\n meta_name='img_metas'):\n self.keys = keys\n self.meta_keys = meta_keys\n self.meta_name = meta_name\n def __call__(self, results):\n \"\"\"Performs the Collect formating.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n data = []\n for key in self.keys:\n data.append(results[key])\n if len(self.meta_keys) != 0:\n meta = {}\n for key in self.meta_keys:\n meta[key] = results[key]\n data.append(meta)\n return data\n def __repr__(self):\n return (f'{self.__class__.__name__}('\n f'keys={self.keys}, meta_keys={self.meta_keys}, '\n f'nested={self.nested})')\n@PIPELINES.register()",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1239-1275"
+ },
+ "5501": {
+ "file_id": 450,
+ "content": "The code defines a class that initializes with specified keys and optional metadata keys. It executes a call method to perform Collect formating on input results, appending each key's data into a list and adding any specified metadata as well. The __repr__ method provides a string representation of the object including its attributes. This pipeline component is registered in @PIPELINES using decorator.",
+ "type": "comment"
+ },
+ "5502": {
+ "file_id": 450,
+ "content": "class GeneratePoseTarget:\n \"\"\"Generate pseudo heatmaps based on joint coordinates and confidence.\n Required keys are \"keypoint\", \"img_shape\", \"keypoint_score\" (optional),\n added or modified keys are \"imgs\".\n Args:\n sigma (float): The sigma of the generated gaussian map. Default: 0.6.\n use_score (bool): Use the confidence score of keypoints as the maximum\n of the gaussian maps. Default: True.\n with_kp (bool): Generate pseudo heatmaps for keypoints. Default: True.\n with_limb (bool): Generate pseudo heatmaps for limbs. At least one of\n 'with_kp' and 'with_limb' should be True. Default: False.\n skeletons (tuple[tuple]): The definition of human skeletons.\n Default: ((0, 1), (0, 2), (1, 3), (2, 4), (0, 5), (5, 7), (7, 9),\n (0, 6), (6, 8), (8, 10), (5, 11), (11, 13), (13, 15),\n (6, 12), (12, 14), (14, 16), (11, 12)),\n which is the definition of COCO-17p skeletons.\n double (bool): Output both original heatmaps and flipped heatmaps.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1276-1294"
+ },
+ "5503": {
+ "file_id": 450,
+ "content": "This code defines a class `GeneratePoseTarget` that generates pseudo heatmaps based on joint coordinates and confidence, with optional use of score, limbs, or skeletons. It takes in required keys \"keypoint\", \"img_shape\", and \"keypoint_score\" (optional). It adds or modifies keys as \"imgs\". The class has parameters like sigma, use_score, with_kp, with_limb, skeletons, and double.",
+ "type": "comment"
+ },
+ "5504": {
+ "file_id": 450,
+ "content": " Default: False.\n left_kp (tuple[int]): Indexes of left keypoints, which is used when\n flipping heatmaps. Default: (1, 3, 5, 7, 9, 11, 13, 15),\n which is left keypoints in COCO-17p.\n right_kp (tuple[int]): Indexes of right keypoints, which is used when\n flipping heatmaps. Default: (2, 4, 6, 8, 10, 12, 14, 16),\n which is right keypoints in COCO-17p.\n \"\"\"\n def __init__(self,\n sigma=0.6,\n use_score=True,\n with_kp=True,\n with_limb=False,\n skeletons=((0, 1), (0, 2), (1, 3), (2, 4), (0, 5), (5, 7),\n (7, 9), (0, 6), (6, 8), (8, 10), (5, 11), (11, 13),\n (13, 15), (6, 12), (12, 14), (14, 16), (11, 12)),\n double=False,\n left_kp=(1, 3, 5, 7, 9, 11, 13, 15),\n right_kp=(2, 4, 6, 8, 10, 12, 14, 16)):\n self.sigma = sigma\n self.use_score = use_score\n self.with_kp = with_kp",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1295-1318"
+ },
+ "5505": {
+ "file_id": 450,
+ "content": "The function initializes skeleton parameters such as sigma, use_score, with_kp, with_limb, skeletons, double, left_kp, and right_kp. It sets default values for these parameters to be used in the skeleton pipeline process.",
+ "type": "comment"
+ },
+ "5506": {
+ "file_id": 450,
+ "content": " self.with_limb = with_limb\n self.double = double\n # an auxiliary const\n self.eps = 1e-4\n assert self.with_kp or self.with_limb, (\n 'At least one of \"with_limb\" '\n 'and \"with_kp\" should be set as True.')\n self.left_kp = left_kp\n self.right_kp = right_kp\n self.skeletons = skeletons\n def generate_a_heatmap(self, img_h, img_w, centers, sigma, max_values):\n \"\"\"Generate pseudo heatmap for one keypoint in one frame.\n Args:\n img_h (int): The height of the heatmap.\n img_w (int): The width of the heatmap.\n centers (np.ndarray): The coordinates of corresponding keypoints\n (of multiple persons).\n sigma (float): The sigma of generated gaussian.\n max_values (np.ndarray): The max values of each keypoint.\n Returns:\n np.ndarray: The generated pseudo heatmap.\n \"\"\"\n heatmap = np.zeros([img_h, img_w], dtype=np.float32)\n for center, max_value in zip(centers, max_values):",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1319-1349"
+ },
+ "5507": {
+ "file_id": 450,
+ "content": "This code is part of the SkeletonPipeline class, which appears to be related to skeleton detection or tracking. The class takes in parameters such as with_limb, double, eps, left_kp, right_kp, and skeletons. It generates a heatmap for one keypoint in one frame using the generate_a_heatmap method. This method takes in img_h, img_w, centers, sigma, and max_values as parameters to create a pseudo heatmap with a zero initial state, iterates through each center-max_value pair, and fills the heatmap accordingly.",
+ "type": "comment"
+ },
+ "5508": {
+ "file_id": 450,
+ "content": " mu_x, mu_y = center[0], center[1]\n if max_value < self.eps:\n continue\n st_x = max(int(mu_x - 3 * sigma), 0)\n ed_x = min(int(mu_x + 3 * sigma) + 1, img_w)\n st_y = max(int(mu_y - 3 * sigma), 0)\n ed_y = min(int(mu_y + 3 * sigma) + 1, img_h)\n x = np.arange(st_x, ed_x, 1, np.float32)\n y = np.arange(st_y, ed_y, 1, np.float32)\n # if the keypoint not in the heatmap coordinate system\n if not (len(x) and len(y)):\n continue\n y = y[:, None]\n patch = np.exp(-((x - mu_x)**2 + (y - mu_y)**2) / 2 / sigma**2)\n patch = patch * max_value\n heatmap[st_y:ed_y, st_x:ed_x] = np.maximum(\n heatmap[st_y:ed_y, st_x:ed_x], patch)\n return heatmap\n def generate_a_limb_heatmap(self, img_h, img_w, starts, ends, sigma,\n start_values, end_values):\n \"\"\"Generate pseudo heatmap for one limb in one frame.",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1350-1375"
+ },
+ "5509": {
+ "file_id": 450,
+ "content": "This function generates a heatmap for a limb in a frame by calculating the Gaussian kernel patch, and updating the heatmap with it. It checks if the keypoint is within the image boundaries before processing. The keypoint positions are calculated based on center coordinates and sigma values. If the keypoints are not within the image bounds, the function continues to the next iteration without updating the heatmap.",
+ "type": "comment"
+ },
+ "5510": {
+ "file_id": 450,
+ "content": " Args:\n img_h (int): The height of the heatmap.\n img_w (int): The width of the heatmap.\n starts (np.ndarray): The coordinates of one keypoint in the\n corresponding limbs (of multiple persons).\n ends (np.ndarray): The coordinates of the other keypoint in the\n corresponding limbs (of multiple persons).\n sigma (float): The sigma of generated gaussian.\n start_values (np.ndarray): The max values of one keypoint in the\n corresponding limbs.\n end_values (np.ndarray): The max values of the other keypoint in\n the corresponding limbs.\n Returns:\n np.ndarray: The generated pseudo heatmap.\n \"\"\"\n heatmap = np.zeros([img_h, img_w], dtype=np.float32)\n for start, end, start_value, end_value in zip(starts, ends,\n start_values, end_values):\n value_coeff = min(start_value, end_value)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1377-1398"
+ },
+ "5511": {
+ "file_id": 450,
+ "content": "This function takes in parameters such as image height, width, keypoint coordinates, and values for each limb. It then generates a pseudo heatmap by iterating through the inputs, calculates a value coefficient for each limb based on their start and end values, and returns a numpy array representing the generated heatmap.",
+ "type": "comment"
+ },
+ "5512": {
+ "file_id": 450,
+ "content": " if value_coeff < self.eps:\n continue\n min_x, max_x = min(start[0], end[0]), max(start[0], end[0])\n min_y, max_y = min(start[1], end[1]), max(start[1], end[1])\n min_x = max(int(min_x - 3 * sigma), 0)\n max_x = min(int(max_x + 3 * sigma) + 1, img_w)\n min_y = max(int(min_y - 3 * sigma), 0)\n max_y = min(int(max_y + 3 * sigma) + 1, img_h)\n x = np.arange(min_x, max_x, 1, np.float32)\n y = np.arange(min_y, max_y, 1, np.float32)\n if not (len(x) and len(y)):\n continue\n y = y[:, None]\n x_0 = np.zeros_like(x)\n y_0 = np.zeros_like(y)\n # distance to start keypoints\n d2_start = ((x - start[0])**2 + (y - start[1])**2)\n # distance to end keypoints\n d2_end = ((x - end[0])**2 + (y - end[1])**2)\n # the distance between start and end keypoints.\n d2_ab = ((start[0] - end[0])**2 + (start[1] - end[1])**2)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1399-1427"
+ },
+ "5513": {
+ "file_id": 450,
+ "content": "This code calculates the distance between a pair of keypoints (start and end) for every pixel in the image, based on certain conditions like value_coeff and sigma. It also adjusts the x and y coordinates to avoid out-of-bounds errors. If the resulting arrays of x and y are empty, it skips processing this pair of keypoints. The calculated distances are used for further processing in the codebase.",
+ "type": "comment"
+ },
+ "5514": {
+ "file_id": 450,
+ "content": " if d2_ab < 1:\n full_map = self.generate_a_heatmap(img_h, img_w, [start], sigma,\n [start_value])\n heatmap = np.maximum(heatmap, full_map)\n continue\n coeff = (d2_start - d2_end + d2_ab) / 2. / d2_ab\n a_dominate = coeff <= 0\n b_dominate = coeff >= 1\n seg_dominate = 1 - a_dominate - b_dominate\n position = np.stack([x + y_0, y + x_0], axis=-1)\n projection = start + np.stack([coeff, coeff],\n axis=-1) * (end - start)\n d2_line = position - projection\n d2_line = d2_line[:, :, 0]**2 + d2_line[:, :, 1]**2\n d2_seg = (a_dominate * d2_start + b_dominate * d2_end +\n seg_dominate * d2_line)\n patch = np.exp(-d2_seg / 2. / sigma**2)\n patch = patch * value_coeff\n heatmap[min_y:max_y, min_x:max_x] = np.maximum(\n heatmap[min_y:max_y, min_x:max_x], patch)",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1429-1453"
+ },
+ "5515": {
+ "file_id": 450,
+ "content": "This code calculates the dominant points and updates the heatmap by applying a Gaussian kernel. It checks if a point is within the start or end of a line segment, and if not, it computes the distance between the point and the line segment. It then uses this distance to compute a weight for each dominant point (start, end) and updates the heatmap using these weights. This ensures that the dominant points have more influence on the heatmap than the less dominant ones.",
+ "type": "comment"
+ },
+ "5516": {
+ "file_id": 450,
+ "content": " return heatmap\n def generate_heatmap(self, img_h, img_w, kps, sigma, max_values):\n \"\"\"Generate pseudo heatmap for all keypoints and limbs in one frame (if\n needed).\n Args:\n img_h (int): The height of the heatmap.\n img_w (int): The width of the heatmap.\n kps (np.ndarray): The coordinates of keypoints in this frame.\n sigma (float): The sigma of generated gaussian.\n max_values (np.ndarray): The confidence score of each keypoint.\n Returns:\n np.ndarray: The generated pseudo heatmap.\n \"\"\"\n heatmaps = []\n if self.with_kp:\n num_kp = kps.shape[1]\n for i in range(num_kp):\n heatmap = self.generate_a_heatmap(img_h, img_w, kps[:, i],\n sigma, max_values[:, i])\n heatmaps.append(heatmap)\n if self.with_limb:\n for limb in self.skeletons:\n start_idx, end_idx = limb\n starts = kps[:, start_idx]",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1455-1483"
+ },
+ "5517": {
+ "file_id": 450,
+ "content": "This code generates a heatmap for all keypoints and limbs in one frame. It takes the height, width, coordinates of keypoints, sigma value, and confidence scores as input. The function iterates over each keypoint to generate separate heatmaps if 'with_kp' is enabled, then appends these individual heatmaps to a list called 'heatmaps'. If 'with_limb' is also enabled, the code generates additional heatmaps for each limb defined in 'self.skeletons' by iterating over them and finding corresponding start and end indices.",
+ "type": "comment"
+ },
+ "5518": {
+ "file_id": 450,
+ "content": " ends = kps[:, end_idx]\n start_values = max_values[:, start_idx]\n end_values = max_values[:, end_idx]\n heatmap = self.generate_a_limb_heatmap(\n img_h, img_w, starts, ends, sigma, start_values, end_values)\n heatmaps.append(heatmap)\n return np.stack(heatmaps, axis=-1)\n def gen_an_aug(self, results):\n \"\"\"Generate pseudo heatmaps for all frames.\n Args:\n results (dict): The dictionary that contains all info of a sample.\n Returns:\n list[np.ndarray]: The generated pseudo heatmaps.\n \"\"\"\n all_kps = results['keypoint']\n kp_shape = all_kps.shape\n if 'keypoint_score' in results:\n all_kpscores = results['keypoint_score']\n else:\n all_kpscores = np.ones(kp_shape[:-1], dtype=np.float32)\n img_h, img_w = results['img_shape']\n num_frame = kp_shape[1]\n imgs = []\n for i in range(num_frame):\n sigma = self.sigma",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1484-1517"
+ },
+ "5519": {
+ "file_id": 450,
+ "content": "The code defines a function that generates pseudo heatmaps for all frames in an image sequence. It extracts keypoint coordinates and scores from the input results, and then creates heatmaps by calling another function to generate limb heatmaps for each frame. These heatmaps are appended into a list and finally stacked along a specific axis to form the output.",
+ "type": "comment"
+ },
+ "5520": {
+ "file_id": 450,
+ "content": " kps = all_kps[:, i]\n kpscores = all_kpscores[:, i]\n max_values = np.ones(kpscores.shape, dtype=np.float32)\n if self.use_score:\n max_values = kpscores\n hmap = self.generate_heatmap(img_h, img_w, kps, sigma, max_values)\n imgs.append(hmap)\n return imgs\n def __call__(self, results):\n if not self.double:\n results['imgs'] = np.stack(self.gen_an_aug(results))\n else:\n results_ = cp.deepcopy(results)\n flip = Flip_V2(\n flip_ratio=1, left_kp=self.left_kp, right_kp=self.right_kp)\n results_ = flip(results_)\n results['imgs'] = np.concatenate(\n [self.gen_an_aug(results),\n self.gen_an_aug(results_)])\n results['label'] = np.array([results['label']])\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'sigma={self.sigma}, '\n f'use_score={self.use_score}, '",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1518-1547"
+ },
+ "5521": {
+ "file_id": 450,
+ "content": "This code defines a class that generates heatmaps from keypoints and applies data augmentation to images. It takes in results as input, generates image heatmaps based on the keypoints, and optionally doubles the output by applying horizontal flipping with a specified left and right keypoint. The sigma value is used for Gaussian filtering, and the use_score flag determines whether scores are considered for generating heatmaps.",
+ "type": "comment"
+ },
+ "5522": {
+ "file_id": 450,
+ "content": " f'with_kp={self.with_kp}, '\n f'with_limb={self.with_limb}, '\n f'skeletons={self.skeletons}, '\n f'double={self.double}, '\n f'left_kp={self.left_kp}, '\n f'right_kp={self.right_kp})')\n return repr_str",
+ "type": "code",
+ "location": "/paddlevideo/loader/pipelines/skeleton_pipeline.py:1548-1554"
+ },
+ "5523": {
+ "file_id": 450,
+ "content": "This code is formatting a string that represents the parameters for a skeleton pipeline. The parameters include whether to output keypoints and limbs, the number of skeletons, and left/right keypoint options.",
+ "type": "comment"
+ },
+ "5524": {
+ "file_id": 451,
+ "content": "/paddlevideo/loader/registry.py",
+ "type": "filepath"
+ },
+ "5525": {
+ "file_id": 451,
+ "content": "This code snippet is a part of the PaddleVideo framework and contains copyright information, license details, and registry definitions for pipelines and datasets. It defines two registries, \"pipeline\" and \"datasets\", using the Registry class from the utils module, allowing the creation and management of custom pipeline and dataset classes.",
+ "type": "summary"
+ },
+ "5526": {
+ "file_id": 451,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..utils import Registry\nPIPELINES = Registry(\"pipeline\")\nDATASETS = Registry(\"datasets\")",
+ "type": "code",
+ "location": "/paddlevideo/loader/registry.py:1-18"
+ },
+ "5527": {
+ "file_id": 451,
+ "content": "This code snippet is a part of the PaddleVideo framework and contains copyright information, license details, and registry definitions for pipelines and datasets. It defines two registries, \"pipeline\" and \"datasets\", using the Registry class from the utils module, allowing the creation and management of custom pipeline and dataset classes.",
+ "type": "comment"
+ },
+ "5528": {
+ "file_id": 452,
+ "content": "/paddlevideo/metrics/ActivityNet/__init__.py",
+ "type": "filepath"
+ },
+ "5529": {
+ "file_id": 452,
+ "content": "The code imports the ANETproposal class from the \"anet_prop\" module and adds it to the __all__ list, making it a public API in the package. This allows other modules to import and use this class directly.",
+ "type": "summary"
+ },
+ "5530": {
+ "file_id": 452,
+ "content": "from .anet_prop import ANETproposal\n__all__ = ['ANETproposal']",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/__init__.py:1-3"
+ },
+ "5531": {
+ "file_id": 452,
+ "content": "The code imports the ANETproposal class from the \"anet_prop\" module and adds it to the __all__ list, making it a public API in the package. This allows other modules to import and use this class directly.",
+ "type": "comment"
+ },
+ "5532": {
+ "file_id": 453,
+ "content": "/paddlevideo/metrics/ActivityNet/anet_prop.py",
+ "type": "filepath"
+ },
+ "5533": {
+ "file_id": 453,
+ "content": "This code imports libraries, defines a class for metrics calculation, retrieves data from ActivityNet API, compares results, creates DataFrames, evaluates proposals using AUC-RC, and calculates average recall. It extracts videos, computes proposal scores, IOU scores, handles exceptions, determines recall with thresholds, and efficiently computes IoU for target and candidate segments.",
+ "type": "summary"
+ },
+ "5534": {
+ "file_id": 453,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport json\nimport numpy as np\nimport pandas as pd\nimport urllib.request as urllib2\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\nclass ANETproposal(object):\n \"\"\"\n This class is used for calculating AR@N and AUC;\n Code transfer from ActivityNet Gitub repository](https://github.com/activitynet/ActivityNet.git)\n \"\"\"\n GROUND_TRUTH_FIELDS = ['database', 'taxonomy', 'version']\n PROPOSAL_FIELDS = ['results', 'version', 'external_data']\n API = 'http://ec2-52-25-205-214.us-west-2.compute.amazonaws.com/challenge19/api.py'",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:1-29"
+ },
+ "5535": {
+ "file_id": 453,
+ "content": "This code imports necessary libraries, defines a class for calculating AR@N and AUC, and sets the API URL for accessing ActivityNet data. The class uses ground truth fields and proposal fields to compare results. Code is transferred from the ActivityNet GitHub repository.",
+ "type": "comment"
+ },
+ "5536": {
+ "file_id": 453,
+ "content": " def __init__(self,\n ground_truth_filename=None,\n proposal_filename=None,\n ground_truth_fields=GROUND_TRUTH_FIELDS,\n proposal_fields=PROPOSAL_FIELDS,\n tiou_thresholds=np.linspace(0.5, 0.95, 10),\n max_avg_nr_proposals=None,\n subset='validation',\n verbose=False,\n check_status=True):\n if not ground_truth_filename:\n raise IOError('Please input a valid ground truth file.')\n if not proposal_filename:\n raise IOError('Please input a valid proposal file.')\n self.subset = subset\n self.tiou_thresholds = tiou_thresholds\n self.max_avg_nr_proposals = max_avg_nr_proposals\n self.verbose = verbose\n self.gt_fields = ground_truth_fields\n self.pred_fields = proposal_fields\n self.recall = None\n self.avg_recall = None\n self.proposals_per_video = None\n self.check_status = check_status",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:31-54"
+ },
+ "5537": {
+ "file_id": 453,
+ "content": "Initializing the class with ground truth and proposal filenames as required, setting default parameters, and checking if both files exist.",
+ "type": "comment"
+ },
+ "5538": {
+ "file_id": 453,
+ "content": " # Retrieve blocked videos from server.\n if self.check_status:\n self.blocked_videos = self.get_blocked_videos()\n else:\n self.blocked_videos = list()\n # Import ground truth and proposals.\n self.ground_truth, self.activity_index = self._import_ground_truth(\n ground_truth_filename)\n self.proposal = self._import_proposal(proposal_filename)\n if self.verbose:\n print('[INIT] Loaded annotations from {} subset.'.format(subset))\n nr_gt = len(self.ground_truth)\n print('\\tNumber of ground truth instances: {}'.format(nr_gt))\n nr_pred = len(self.proposal)\n print('\\tNumber of proposals: {}'.format(nr_pred))\n print('\\tFixed threshold for tiou score: {}'.format(\n self.tiou_thresholds))\n def _import_ground_truth(self, ground_truth_filename):\n \"\"\"\n Reads ground truth file, checks if it is well formatted, and returns\n the ground truth instances and the activity classes.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:55-77"
+ },
+ "5539": {
+ "file_id": 453,
+ "content": "This code retrieves blocked videos from a server, imports ground truth and proposals, and checks if the ground truth file is well formatted. It also prints information about the number of ground truth instances and proposals, as well as the fixed threshold for tiou score.",
+ "type": "comment"
+ },
+ "5540": {
+ "file_id": 453,
+ "content": " Parameters:\n ground_truth_filename (str): full path to the ground truth json file.\n Returns:\n ground_truth (df): Data frame containing the ground truth instances.\n activity_index (dict): Dictionary containing class index.\n \"\"\"\n with open(ground_truth_filename, 'r') as fobj:\n data = json.load(fobj)\n # Checking format\n if not all([field in data.keys() for field in self.gt_fields]):\n raise IOError('Please input a valid ground truth file.')\n # Read ground truth data.\n activity_index, cidx = {}, 0\n video_lst, t_start_lst, t_end_lst, label_lst = [], [], [], []\n for videoid, v in data['database'].items():\n if self.subset != v['subset']:\n continue\n if videoid in self.blocked_videos:\n continue\n for ann in v['annotations']:\n if ann['label'] not in activity_index:\n activity_index[ann['label']] = cidx\n cidx += 1",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:79-102"
+ },
+ "5541": {
+ "file_id": 453,
+ "content": "This function reads a ground truth JSON file and returns a DataFrame containing the instances. It also returns a dictionary of class indices. The function checks if the input file has the required fields, skips videos not in the specified subset, and ignores blocked videos. If an activity label is not found in the activity_index, it adds it to the index and increments the counter.",
+ "type": "comment"
+ },
+ "5542": {
+ "file_id": 453,
+ "content": " video_lst.append(videoid)\n t_start_lst.append(float(ann['segment'][0]))\n t_end_lst.append(float(ann['segment'][1]))\n label_lst.append(activity_index[ann['label']])\n ground_truth = pd.DataFrame({\n 'video-id': video_lst,\n 't-start': t_start_lst,\n 't-end': t_end_lst,\n 'label': label_lst\n })\n return ground_truth, activity_index\n def _import_proposal(self, proposal_filename):\n \"\"\"\n Reads proposal file, checks if it is well formatted, and returns\n the proposal instances.\n Parameters:\n proposal_filename (str): Full path to the proposal json file.\n Returns:\n proposal (df): Data frame containing the proposal instances.\n \"\"\"\n with open(proposal_filename, 'r') as fobj:\n data = json.load(fobj)\n # Checking format...\n if not all([field in data.keys() for field in self.pred_fields]):\n raise IOError('Please input a valid proposal file.')",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:103-130"
+ },
+ "5543": {
+ "file_id": 453,
+ "content": "The code reads a proposal file, checks its format and returns proposal instances in the form of a data frame. It also generates ground truth data by appending video IDs, start and end times, and labels to lists before creating a DataFrame. The function takes a string as input for the full path to the proposal JSON file and returns a data frame containing the proposal instances.",
+ "type": "comment"
+ },
+ "5544": {
+ "file_id": 453,
+ "content": " # Read predictions.\n video_lst, t_start_lst, t_end_lst = [], [], []\n score_lst = []\n for videoid, v in data['results'].items():\n if videoid in self.blocked_videos:\n continue\n for result in v:\n video_lst.append(videoid)\n t_start_lst.append(float(result['segment'][0]))\n t_end_lst.append(float(result['segment'][1]))\n score_lst.append(result['score'])\n proposal = pd.DataFrame({\n 'video-id': video_lst,\n 't-start': t_start_lst,\n 't-end': t_end_lst,\n 'score': score_lst\n })\n return proposal\n def evaluate(self):\n \"\"\"\n Evaluates a proposal file. To measure the performance of a\n method for the proposal task, we computes the area under the\n average recall vs average number of proposals per video curve.\n \"\"\"\n recall, avg_recall, proposals_per_video = self.average_recall_vs_avg_nr_proposals(\n self.ground_truth,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:132-158"
+ },
+ "5545": {
+ "file_id": 453,
+ "content": "The code reads predictions from a data source, extracts relevant information (video IDs, start and end timestamps, scores), stores them in a DataFrame, and defines two functions: one for evaluating proposal files by computing area under the average recall vs average number of proposals per video curve. The evaluation function calls another function to compute this metric using ground truth data and the stored proposal data.",
+ "type": "comment"
+ },
+ "5546": {
+ "file_id": 453,
+ "content": " self.proposal,\n max_avg_nr_proposals=self.max_avg_nr_proposals,\n tiou_thresholds=self.tiou_thresholds)\n area_under_curve = np.trapz(avg_recall, proposals_per_video)\n if self.verbose:\n print('[RESULTS] Performance on ActivityNet proposal task.')\n with open(\"data/bmn/BMN_Test_results/auc_result.txt\",\n \"a\") as text_file:\n text_file.write(\n '\\tArea Under the AR vs AN curve: {}% \\n'.format(\n 100. * float(area_under_curve) /\n proposals_per_video[-1]))\n print('\\tArea Under the AR vs AN curve: {}%'.format(\n 100. * float(area_under_curve) / proposals_per_video[-1]))\n self.recall = recall\n self.avg_recall = avg_recall\n self.proposals_per_video = proposals_per_video\n def average_recall_vs_avg_nr_proposals(self,\n ground_truth,\n proposals,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:159-182"
+ },
+ "5547": {
+ "file_id": 453,
+ "content": "Calculates the area under the curve of recall vs average number of proposals for ActivityNet proposal task, writes result to file and stores recall, average recall, and proposals per video in class attributes.",
+ "type": "comment"
+ },
+ "5548": {
+ "file_id": 453,
+ "content": " max_avg_nr_proposals=None,\n tiou_thresholds=np.linspace(\n 0.5, 0.95, 10)):\n \"\"\"\n Computes the average recall given an average number of\n proposals per video.\n Parameters:\n ground_truth(df): Data frame containing the ground truth instances.\n Required fields: ['video-id', 't-start', 't-end']\n proposal(df): Data frame containing the proposal instances.\n Required fields: ['video-id, 't-start', 't-end', 'score']\n tiou_thresholds(1d-array | optional): array with tiou thresholds.\n Returns:\n recall(2d-array): recall[i,j] is recall at ith tiou threshold at the jth\n average number of average number of proposals per video.\n average_recall(1d-array): recall averaged over a list of tiou threshold.\n This is equivalent to recall.mean(axis=0).\n proposals_per_video(1d-array): average number of proposals per video.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:183-202"
+ },
+ "5549": {
+ "file_id": 453,
+ "content": "This code defines a function that computes average recall for given average number of proposals per video. It takes ground truth and proposal data frames as input, along with optional tiou_thresholds. It returns recall and average_recall arrays.",
+ "type": "comment"
+ },
+ "5550": {
+ "file_id": 453,
+ "content": " \"\"\"\n # Get list of videos.\n video_lst = ground_truth['video-id'].unique()\n if not max_avg_nr_proposals:\n max_avg_nr_proposals = float(\n proposals.shape[0]) / video_lst.shape[0]\n ratio = max_avg_nr_proposals * float(\n video_lst.shape[0]) / proposals.shape[0]\n # Adaptation to query faster\n ground_truth_gbvn = ground_truth.groupby('video-id')\n proposals_gbvn = proposals.groupby('video-id')\n # For each video, computes tiou scores among the retrieved proposals.\n score_lst = []\n total_nr_proposals = 0\n for videoid in video_lst:\n # Get ground-truth instances associated to this video.\n ground_truth_videoid = ground_truth_gbvn.get_group(videoid)\n this_video_ground_truth = ground_truth_videoid.loc[:, [\n 't-start', 't-end'\n ]].values\n # Get proposals for this video.\n try:\n proposals_videoid = proposals_gbvn.get_group(videoid)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:203-231"
+ },
+ "5551": {
+ "file_id": 453,
+ "content": "This code retrieves a list of videos, calculates the maximum average number of proposals per video, groups proposals and ground truth by video ID, and then computes Tiou scores between ground-truth instances and retrieved proposals for each video.",
+ "type": "comment"
+ },
+ "5552": {
+ "file_id": 453,
+ "content": " except:\n n = this_video_ground_truth.shape[0]\n score_lst.append(np.zeros((n, 1)))\n continue\n this_video_proposals = proposals_videoid.loc[:,\n ['t-start', 't-end'\n ]].values\n if this_video_proposals.shape[0] == 0:\n n = this_video_ground_truth.shape[0]\n score_lst.append(np.zeros((n, 1)))\n continue\n # Sort proposals by score.\n sort_idx = proposals_videoid['score'].argsort()[::-1]\n this_video_proposals = this_video_proposals[sort_idx, :]\n if this_video_proposals.ndim != 2:\n this_video_proposals = np.expand_dims(this_video_proposals,\n axis=0)\n if this_video_ground_truth.ndim != 2:\n this_video_ground_truth = np.expand_dims(\n this_video_ground_truth, axis=0)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:232-255"
+ },
+ "5553": {
+ "file_id": 453,
+ "content": "This code block is part of a function that handles exceptions when dealing with video proposals and ground truth. It appends a zero matrix to the score list if there are no video proposals or ground truth data for the current video. If there are proposals, it sorts them by score in descending order and expands dimensions as necessary before proceeding.",
+ "type": "comment"
+ },
+ "5554": {
+ "file_id": 453,
+ "content": " nr_proposals = np.minimum(\n int(this_video_proposals.shape[0] * ratio),\n this_video_proposals.shape[0])\n total_nr_proposals += nr_proposals\n this_video_proposals = this_video_proposals[:nr_proposals, :]\n # Compute tiou scores.\n tiou = self.wrapper_segment_iou(this_video_proposals,\n this_video_ground_truth)\n score_lst.append(tiou)\n # Given that the length of the videos is really varied, we\n # compute the number of proposals in terms of a ratio of the total\n # proposals retrieved, i.e. average recall at a percentage of proposals\n # retrieved per video.\n # Computes average recall.\n pcn_lst = np.arange(1, 101) / 100.0 * (max_avg_nr_proposals * float(\n video_lst.shape[0]) / total_nr_proposals)\n matches = np.empty((video_lst.shape[0], pcn_lst.shape[0]))\n positives = np.empty(video_lst.shape[0])\n recall = np.empty((tiou_thresholds.shape[0], pcn_lst.shape[0]))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:257-278"
+ },
+ "5555": {
+ "file_id": 453,
+ "content": "This code calculates average recall for a set of video proposals. It sets the number of proposals based on a ratio, computes IOU scores, and stores the results in lists. The average recall is computed using a predetermined maximum number of proposals and the total number of proposals retrieved, considering the variable length of videos.",
+ "type": "comment"
+ },
+ "5556": {
+ "file_id": 453,
+ "content": " # Iterates over each tiou threshold.\n for ridx, tiou in enumerate(tiou_thresholds):\n # Inspect positives retrieved per video at different\n # number of proposals (percentage of the total retrieved).\n for i, score in enumerate(score_lst):\n # Total positives per video.\n positives[i] = score.shape[0]\n # Find proposals that satisfies minimum tiou threshold.\n true_positives_tiou = score >= tiou\n # Get number of proposals as a percentage of total retrieved.\n pcn_proposals = np.minimum(\n (score.shape[1] * pcn_lst).astype(int), score.shape[1])\n for j, nr_proposals in enumerate(pcn_proposals):\n # Compute the number of matches for each percentage of the proposals\n matches[i, j] = np.count_nonzero(\n (true_positives_tiou[:, :nr_proposals]).sum(axis=1))\n # Computes recall given the set of matches per video.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:279-298"
+ },
+ "5557": {
+ "file_id": 453,
+ "content": "Code iterates over different tiou thresholds and positive scores, computing the number of true positives based on threshold and percentage of proposals. It calculates matches per video and computes recall for each set of matches.",
+ "type": "comment"
+ },
+ "5558": {
+ "file_id": 453,
+ "content": " recall[ridx, :] = matches.sum(axis=0) / positives.sum()\n # Recall is averaged.\n avg_recall = recall.mean(axis=0)\n # Get the average number of proposals per video.\n proposals_per_video = pcn_lst * (float(total_nr_proposals) /\n video_lst.shape[0])\n return recall, avg_recall, proposals_per_video\n def get_blocked_videos(self, api=API):\n api_url = '{}?action=get_blocked'.format(api)\n req = urllib2.Request(api_url)\n response = urllib2.urlopen(req)\n return json.loads(response.read())\n def wrapper_segment_iou(self, target_segments, candidate_segments):\n \"\"\"\n Compute intersection over union btw segments\n Parameters:\n target_segments(nd-array): 2-dim array in format [m x 2:=[init, end]]\n candidate_segments(nd-array): 2-dim array in format [n x 2:=[init, end]]\n Returns:\n tiou(nd-array): 2-dim array [n x m] with IOU ratio.\n Note: It assumes that candidate-segments are more scarce that target-segments",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:299-324"
+ },
+ "5559": {
+ "file_id": 453,
+ "content": "The function calculates recall and average recall for detected objects in videos, based on the number of true positives and total proposals. It also returns the average number of proposals per video. The second function retrieves a list of blocked videos from an API. The third function computes intersection over union between target and candidate segments efficiently.",
+ "type": "comment"
+ },
+ "5560": {
+ "file_id": 453,
+ "content": " \"\"\"\n if candidate_segments.ndim != 2 or target_segments.ndim != 2:\n raise ValueError('Dimension of arguments is incorrect')\n n, m = candidate_segments.shape[0], target_segments.shape[0]\n tiou = np.empty((n, m))\n for i in range(m):\n tiou[:, i] = self.segment_iou(target_segments[i, :],\n candidate_segments)\n return tiou\n def segment_iou(self, target_segment, candidate_segments):\n \"\"\"\n Compute the temporal intersection over union between a\n target segment and all the test segments.\n Parameters:\n target_segment(1d-array): Temporal target segment containing [starting, ending] times.\n candidate_segments(2d-array): Temporal candidate segments containing N x [starting, ending] times.\n Returns:\n tiou(1d-array): Temporal intersection over union score of the N's candidate segments.\n \"\"\"\n tt1 = np.maximum(target_segment[0], candidate_segments[:, 0])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:325-349"
+ },
+ "5561": {
+ "file_id": 453,
+ "content": "This function calculates the temporal intersection over union (TIOU) between a target segment and multiple candidate segments. If the dimensions of arguments are not 2, it raises a ValueError. It loops through each candidate segment, compares their starting and ending times with the target segment's times using np.maximum, and stores the TIOU results in a 2D array.",
+ "type": "comment"
+ },
+ "5562": {
+ "file_id": 453,
+ "content": " tt2 = np.minimum(target_segment[1], candidate_segments[:, 1])\n # Intersection including Non-negative overlap score.\n segments_intersection = (tt2 - tt1).clip(0)\n # Segment union.\n segments_union = (candidate_segments[:, 1] - candidate_segments[:, 0]) \\\n + (target_segment[1] - target_segment[0]) - segments_intersection\n # Compute overlap as the ratio of the intersection\n # over union of two segments.\n tIoU = segments_intersection.astype(float) / segments_union\n return tIoU",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ActivityNet/anet_prop.py:350-359"
+ },
+ "5563": {
+ "file_id": 453,
+ "content": "Computes intersection over union (IoU) of two segments by finding the minimum endpoints, calculating intersection and union, and dividing intersection by union.",
+ "type": "comment"
+ },
+ "5564": {
+ "file_id": 454,
+ "content": "/paddlevideo/metrics/__init__.py",
+ "type": "filepath"
+ },
+ "5565": {
+ "file_id": 454,
+ "content": "The code imports various metrics from different modules for video analysis and evaluation, including AVAMetric, VOSMetric, BMNMetric, MSRVTTMetric, SkeletonMetric, TransNetV2Metric, DepthMetric, CenterCropMetric, MultiCropMetric, HitOneMetric, and SegmentationMetric. It also imports the METRIC registry for managing these metrics.",
+ "type": "summary"
+ },
+ "5566": {
+ "file_id": 454,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .bmn_metric import BMNMetric\nfrom .build import build_metric\nfrom .center_crop_metric import CenterCropMetric\nfrom .depth_metric import DepthMetric\nfrom .msrvtt_metric import MSRVTTMetric\nfrom .multi_crop_metric import MultiCropMetric\nfrom .registry import METRIC\nfrom .skeleton_metric import SkeletonMetric\nfrom .transnetv2_metric import TransNetV2Metric\nfrom .youtube8m.eval_util import HitOneMetric\nfrom .segmentation_metric import SegmentationMetric",
+ "type": "code",
+ "location": "/paddlevideo/metrics/__init__.py:1-25"
+ },
+ "5567": {
+ "file_id": 454,
+ "content": "The code is importing various metrics from different metric classes for video analysis and evaluation. It includes BMNMetric, MSRVTTMetric, SkeletonMetric, TransNetV2Metric, DepthMetric, CenterCropMetric, MultiCropMetric, HitOneMetric, and SegmentationMetric. The METRIC registry is also imported for managing these metrics.",
+ "type": "comment"
+ },
+ "5568": {
+ "file_id": 454,
+ "content": "from .ava_metric import AVAMetric\nfrom .vos_metric import VOSMetric\nfrom .center_crop_metric_MRI import CenterCropMetric_MRI\nfrom .yowo_metric import YOWOMetric\n__all__ = [\n 'METRIC', 'build_metric', 'MultiCropMetric', 'BMNMetric',\n 'CenterCropMetric', 'SkeletonMetric', 'HitOneMetric', 'TransNetV2Metric',\n 'DepthMetric', 'MSRVTTMetric', 'VOSMetric', 'CenterCropMetric_MRI','AVAMetric',\n 'SegmentationMetric', 'YOWOMetric'\n]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/__init__.py:26-36"
+ },
+ "5569": {
+ "file_id": 454,
+ "content": "This code imports various metrics from different modules and adds them to the __all__ list for easy access, including AVAMetric, VOSMetric, CenterCropMetric_MRI, YOWOMetric, METRIC, build_metric, MultiCropMetric, BMNMetric, CenterCropMetric, SkeletonMetric, HitOneMetric, TransNetV2Metric, DepthMetric, MSRVTTMetric.",
+ "type": "comment"
+ },
+ "5570": {
+ "file_id": 455,
+ "content": "/paddlevideo/metrics/ava_evaluation/README.md",
+ "type": "filepath"
+ },
+ "5571": {
+ "file_id": 455,
+ "content": "This code is from the ActivityNet repo and has been modified to reduce length, possibly for efficiency or better readability. It uses metrics and evaluation methods for action recognition tasks, likely in video analysis applications.",
+ "type": "summary"
+ },
+ "5572": {
+ "file_id": 455,
+ "content": "The code under this folder is from the official [ActivityNet repo](https://github.com/activitynet/ActivityNet).\nSome unused codes are removed to minimize the length of codes added.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/README.md:1-2"
+ },
+ "5573": {
+ "file_id": 455,
+ "content": "This code is from the ActivityNet repo and has been modified to reduce length, possibly for efficiency or better readability. It uses metrics and evaluation methods for action recognition tasks, likely in video analysis applications.",
+ "type": "comment"
+ },
+ "5574": {
+ "file_id": 456,
+ "content": "/paddlevideo/metrics/ava_evaluation/metrics.py",
+ "type": "filepath"
+ },
+ "5575": {
+ "file_id": 456,
+ "content": "This function calculates precision and recall metrics from scores, labels, and ground truth instances, raising ValueError for incorrect inputs. It computes average precision using valid precision and recall arrays and calculates CorLoc performance metrics for object detection with given ground truth and detected images per class.",
+ "type": "summary"
+ },
+ "5576": {
+ "file_id": 456,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Functions for computing metrics like precision, recall, CorLoc and etc.\"\"\"\nimport numpy as np\ndef compute_precision_recall(scores, labels, num_gt):\n \"\"\"Compute precision and recall.\n Args:\n scores: A float numpy array representing detection score\n labels: A boolean numpy array representing true/false positive labels\n num_gt: Number of ground truth instances\n Raises:\n ValueError: if the input is not of the correct format",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/metrics.py:1-30"
+ },
+ "5577": {
+ "file_id": 456,
+ "content": "This code defines a function to compute precision and recall metrics based on input scores, labels, and the number of ground truth instances. It also raises a ValueError if the input is in the incorrect format.",
+ "type": "comment"
+ },
+ "5578": {
+ "file_id": 456,
+ "content": " Returns:\n precision: Fraction of positive instances over detected ones. This\n value is None if no ground truth labels are present.\n recall: Fraction of detected positive instance over all positive\n instances. This value is None if no ground truth labels are\n present.\n \"\"\"\n if (not isinstance(labels, np.ndarray) or labels.dtype != np.bool\n or len(labels.shape) != 1):\n raise ValueError('labels must be single dimension bool numpy array')\n if not isinstance(scores, np.ndarray) or len(scores.shape) != 1:\n raise ValueError('scores must be single dimension numpy array')\n if num_gt < np.sum(labels):\n raise ValueError(\n 'Number of true positives must be smaller than num_gt.')\n if len(scores) != len(labels):\n raise ValueError('scores and labels must be of the same size.')\n if num_gt == 0:\n return None, None\n sorted_indices = np.argsort(scores)\n sorted_indices = sorted_indices[::-1]\n labels = labels.astype(int)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/metrics.py:32-58"
+ },
+ "5579": {
+ "file_id": 456,
+ "content": "This code checks if input 'labels' and 'scores' are valid arrays. It verifies that 'labels' is a one-dimensional boolean numpy array, 'scores' is a one-dimensional numpy array, the number of true positives is less than num_gt (number of ground truth labels), and the lengths of 'scores' and 'labels' are equal. If any conditions are not met, it raises a ValueError with an appropriate error message. If all checks pass and there are no ground truth labels, the function returns None for both precision and recall.",
+ "type": "comment"
+ },
+ "5580": {
+ "file_id": 456,
+ "content": " true_positive_labels = labels[sorted_indices]\n false_positive_labels = 1 - true_positive_labels\n cum_true_positives = np.cumsum(true_positive_labels)\n cum_false_positives = np.cumsum(false_positive_labels)\n precision = cum_true_positives.astype(float) / (\n cum_true_positives + cum_false_positives)\n recall = cum_true_positives.astype(float) / num_gt\n return precision, recall\ndef compute_average_precision(precision, recall):\n \"\"\"Compute Average Precision according to the definition in VOCdevkit.\n Precision is modified to ensure that it does not decrease as recall\n decrease.\n Args:\n precision: A float [N, 1] numpy array of precisions\n recall: A float [N, 1] numpy array of recalls\n Raises:\n ValueError: if the input is not of the correct format\n Returns:\n average_precison: The area under the precision recall curve. NaN if\n precision and recall are None.\n \"\"\"\n if precision is None:\n if recall is not None:\n raise ValueError('If precision is None, recall must also be None')",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/metrics.py:59-88"
+ },
+ "5581": {
+ "file_id": 456,
+ "content": "Computes precision and recall from sorted labels, returns both values.\nDefines a function to compute average precision using precision and recall arrays.",
+ "type": "comment"
+ },
+ "5582": {
+ "file_id": 456,
+ "content": " return np.NAN\n if not isinstance(precision, np.ndarray) or not isinstance(\n recall, np.ndarray):\n raise ValueError('precision and recall must be numpy array')\n if precision.dtype != np.float or recall.dtype != np.float:\n raise ValueError('input must be float numpy array.')\n if len(precision) != len(recall):\n raise ValueError('precision and recall must be of the same size.')\n if not precision.size:\n return 0.0\n if np.amin(precision) < 0 or np.amax(precision) > 1:\n raise ValueError('Precision must be in the range of [0, 1].')\n if np.amin(recall) < 0 or np.amax(recall) > 1:\n raise ValueError('recall must be in the range of [0, 1].')\n if not all(recall[i] <= recall[i + 1] for i in range(len(recall) - 1)):\n raise ValueError('recall must be a non-decreasing array')\n recall = np.concatenate([[0], recall, [1]])\n precision = np.concatenate([[0], precision, [0]])\n # Preprocess precision to be a non-decreasing array\n for i in range(len(precision) - 2, -1, -1):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/metrics.py:89-111"
+ },
+ "5583": {
+ "file_id": 456,
+ "content": "This function checks the data types and ranges of precision and recall arrays, ensuring they are numpy float arrays within the range [0,1] and have the same size. If all conditions pass, it then concatenates recall and precision arrays with 0 and 1 at the end respectively before preprocessing precision to be a non-decreasing array.",
+ "type": "comment"
+ },
+ "5584": {
+ "file_id": 456,
+ "content": " precision[i] = np.maximum(precision[i], precision[i + 1])\n indices = np.where(recall[1:] != recall[:-1])[0] + 1\n average_precision = np.sum(\n (recall[indices] - recall[indices - 1]) * precision[indices])\n return average_precision\ndef compute_cor_loc(num_gt_imgs_per_class,\n num_images_correctly_detected_per_class):\n \"\"\"Compute CorLoc according to the definition in the following paper.\n https://www.robots.ox.ac.uk/~vgg/rg/papers/deselaers-eccv10.pdf\n Returns nans if there are no ground truth images for a class.\n Args:\n num_gt_imgs_per_class: 1D array, representing number of images\n containing at least one object instance of a particular class\n num_images_correctly_detected_per_class: 1D array, representing number\n of images that are correctly detected at least one object instance\n of a particular class\n Returns:\n corloc_per_class: A float numpy array represents the corloc score of\n each class",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/metrics.py:112-137"
+ },
+ "5585": {
+ "file_id": 456,
+ "content": "This code computes the average precision and CorLoc, which is a metric used to evaluate object detection performance. It takes in arrays of ground truth images per class and correctly detected images per class. The average precision function calculates the average precision by comparing recall values, while the compute_cor_loc function calculates the CorLoc score for each class based on these inputs. If there are no ground truth images for a class, it returns NaN.",
+ "type": "comment"
+ },
+ "5586": {
+ "file_id": 456,
+ "content": " \"\"\"\n # Divide by zero expected for classes with no gt examples.\n with np.errstate(divide='ignore', invalid='ignore'):\n return np.where(\n num_gt_imgs_per_class == 0, np.nan,\n num_images_correctly_detected_per_class / num_gt_imgs_per_class)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/metrics.py:138-143"
+ },
+ "5587": {
+ "file_id": 456,
+ "content": "Divides the number of images correctly detected by the number of ground truth images per class, ignoring division by zero for classes with no examples.",
+ "type": "comment"
+ },
+ "5588": {
+ "file_id": 457,
+ "content": "/paddlevideo/metrics/ava_evaluation/np_box_list.py",
+ "type": "filepath"
+ },
+ "5589": {
+ "file_id": 457,
+ "content": "The \"BoxList\" class manages bounding boxes, and the _is_valid_boxes function checks if data array of shape [N, 4] representing box coordinates adheres to the correct format. The function returns a boolean indicating whether all ymax are greater than or equal to ymin, all xmax are greater than or equal to xmin, and the data is not empty.",
+ "type": "summary"
+ },
+ "5590": {
+ "file_id": 457,
+ "content": "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# =============================================================================\n\"\"\"Numpy BoxList classes and functions.\"\"\"\nimport numpy as np\nclass BoxList:\n \"\"\"Box collection.\n BoxList represents a list of bounding boxes as numpy array, where each\n bounding box is represented as a row of 4 numbers,\n [y_min, x_min, y_max, x_max]. It is assumed that all bounding boxes within\n a given list correspond to a single image.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_list.py:1-26"
+ },
+ "5591": {
+ "file_id": 457,
+ "content": "The code defines a class called \"BoxList\" that represents a collection of bounding boxes as a numpy array. Each box is represented by 4 numbers: y_min, x_min, y_max, and x_max. It assumes all boxes in the list correspond to a single image.",
+ "type": "comment"
+ },
+ "5592": {
+ "file_id": 457,
+ "content": " Optionally, users can add additional related fields (such as\n objectness/classification scores).\n \"\"\"\n def __init__(self, data):\n \"\"\"Constructs box collection.\n Args:\n data: a numpy array of shape [N, 4] representing box coordinates\n Raises:\n ValueError: if bbox data is not a numpy array\n ValueError: if invalid dimensions for bbox data\n \"\"\"\n if not isinstance(data, np.ndarray):\n raise ValueError('data must be a numpy array.')\n if len(data.shape) != 2 or data.shape[1] != 4:\n raise ValueError('Invalid dimensions for box data.')\n if data.dtype != np.float32 and data.dtype != np.float64:\n raise ValueError(\n 'Invalid data type for box data: float is required.')\n if not self._is_valid_boxes(data):\n raise ValueError('Invalid box data. data must be a numpy array of '\n 'N*[y_min, x_min, y_max, x_max]')\n self.data = {'boxes': data}",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_list.py:28-52"
+ },
+ "5593": {
+ "file_id": 457,
+ "content": "This code defines a class for box collections, where users can optionally add additional related fields such as objectness or classification scores. The `__init__` method checks if the input data is a numpy array, has valid dimensions and data type (float), and raises a ValueError if any of these conditions are not met. It then stores the data in a dictionary with key \"boxes\".",
+ "type": "comment"
+ },
+ "5594": {
+ "file_id": 457,
+ "content": " def num_boxes(self):\n \"\"\"Return number of boxes held in collections.\"\"\"\n return self.data['boxes'].shape[0]\n def get_extra_fields(self):\n \"\"\"Return all non-box fields.\"\"\"\n return [k for k in self.data if k != 'boxes']\n def has_field(self, field):\n return field in self.data\n def add_field(self, field, field_data):\n \"\"\"Add data to a specified field.\n Args:\n field: a string parameter used to speficy a related field to be\n accessed.\n field_data: a numpy array of [N, ...] representing the data\n associated with the field.\n Raises:\n ValueError: if the field is already exist or the dimension of the\n field data does not matches the number of boxes.\n \"\"\"\n if self.has_field(field):\n raise ValueError('Field ' + field + 'already exists')\n if len(field_data.shape) < 1 or field_data.shape[0] != self.num_boxes(\n ):\n raise ValueError('Invalid dimensions for field data')",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_list.py:54-81"
+ },
+ "5595": {
+ "file_id": 457,
+ "content": "This code defines a class with methods to handle box collections. It provides functionality for counting the number of boxes, retrieving non-box fields, checking if a specific field exists, and adding data to an existing or new field while handling errors related to field existence and data dimensions.",
+ "type": "comment"
+ },
+ "5596": {
+ "file_id": 457,
+ "content": " self.data[field] = field_data\n def get(self):\n \"\"\"Convenience function for accesssing box coordinates.\n Returns:\n a numpy array of shape [N, 4] representing box corners\n \"\"\"\n return self.get_field('boxes')\n def get_field(self, field):\n \"\"\"Accesses data associated with the specified field in the box\n collection.\n Args:\n field: a string parameter used to speficy a related field to be\n accessed.\n Returns:\n a numpy 1-d array representing data of an associated field\n Raises:\n ValueError: if invalid field\n \"\"\"\n if not self.has_field(field):\n raise ValueError(f'field {field} does not exist')\n return self.data[field]\n def get_coordinates(self):\n \"\"\"Get corner coordinates of boxes.\n Returns:\n a list of 4 1-d numpy arrays [y_min, x_min, y_max, x_max]\n \"\"\"\n box_coordinates = self.get()\n y_min = box_coordinates[:, 0]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_list.py:82-117"
+ },
+ "5597": {
+ "file_id": 457,
+ "content": "The code defines a class with methods to access box coordinates from a stored dataset. The \"get\" method returns a numpy array of shape [N, 4] representing box corners. The \"get_field\" method is used to access data related to a specific field in the box collection. If an invalid field is provided, it raises a ValueError. The \"get_coordinates\" method returns a list of 4 1-d numpy arrays containing y_min, x_min, y_max, and x_max values for each box.",
+ "type": "comment"
+ },
+ "5598": {
+ "file_id": 457,
+ "content": " x_min = box_coordinates[:, 1]\n y_max = box_coordinates[:, 2]\n x_max = box_coordinates[:, 3]\n return [y_min, x_min, y_max, x_max]\n def _is_valid_boxes(self, data):\n \"\"\"Check whether data fullfills the format of N*[ymin, xmin, ymax,\n xmin].\n Args:\n data: a numpy array of shape [N, 4] representing box coordinates\n Returns:\n a boolean indicating whether all ymax of boxes are equal or greater\n than ymin, and all xmax of boxes are equal or greater than xmin.\n \"\"\"\n if len(data):\n for v in data:\n if v[0] > v[2] or v[1] > v[3]:\n return False\n return True",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_list.py:118-138"
+ },
+ "5599": {
+ "file_id": 457,
+ "content": "This code defines a function `_is_valid_boxes` which checks if the data array of shape [N, 4] representing box coordinates fulfills the format N*[ymin, xmin, ymax, xmax]. It returns a boolean indicating whether all ymax of boxes are equal or greater than ymin and all xmax of boxes are equal or greater than xmin. The function also checks if the data is not empty.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/56.json b/docs/data/56.json
new file mode 100644
index 000000000..6b24e5532
--- /dev/null
+++ b/docs/data/56.json
@@ -0,0 +1,549 @@
+{
+ "5600": {
+ "file_id": 458,
+ "content": "/paddlevideo/metrics/ava_evaluation/np_box_ops.py",
+ "type": "filepath"
+ },
+ "5601": {
+ "file_id": 458,
+ "content": "The code defines functions for numpy array operations on bounding boxes, including area calculation and intersection-over-union scores useful in computer vision tasks. It computes pairwise IoU scores using numpy arrays by dividing the intersection by the second set's box areas.",
+ "type": "summary"
+ },
+ "5602": {
+ "file_id": 458,
+ "content": "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"Operations for [N, 4] numpy arrays representing bounding boxes.\nExample box operations that are supported:\n * Areas: compute bounding box areas\n * IOU: pairwise intersection-over-union scores\n\"\"\"\nimport numpy as np\ndef area(boxes):\n \"\"\"Computes area of boxes.\n Args:\n boxes: Numpy array with shape [N, 4] holding N boxes",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_ops.py:1-29"
+ },
+ "5603": {
+ "file_id": 458,
+ "content": "The code defines functions for performing operations on numpy arrays of bounding boxes. It includes functionality to compute areas and intersection-over-union scores between pairs of boxes. The array holds N boxes and is expected to have shape [N, 4]. These operations are useful in computer vision tasks like object detection and tracking.",
+ "type": "comment"
+ },
+ "5604": {
+ "file_id": 458,
+ "content": " Returns:\n a numpy array with shape [N*1] representing box areas\n \"\"\"\n return (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])\ndef intersection(boxes1, boxes2):\n \"\"\"Compute pairwise intersection areas between boxes.\n Args:\n boxes1: a numpy array with shape [N, 4] holding N boxes\n boxes2: a numpy array with shape [M, 4] holding M boxes\n Returns:\n a numpy array with shape [N*M] representing pairwise intersection area\n \"\"\"\n [y_min1, x_min1, y_max1, x_max1] = np.split(boxes1, 4, axis=1)\n [y_min2, x_min2, y_max2, x_max2] = np.split(boxes2, 4, axis=1)\n all_pairs_min_ymax = np.minimum(y_max1, np.transpose(y_max2))\n all_pairs_max_ymin = np.maximum(y_min1, np.transpose(y_min2))\n intersect_heights = np.maximum(\n np.zeros(all_pairs_max_ymin.shape),\n all_pairs_min_ymax - all_pairs_max_ymin)\n all_pairs_min_xmax = np.minimum(x_max1, np.transpose(x_max2))\n all_pairs_max_xmin = np.maximum(x_min1, np.transpose(x_min2))\n intersect_widths = np.maximum(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_ops.py:31-57"
+ },
+ "5605": {
+ "file_id": 458,
+ "content": "Computes box areas by multiplying width and height (lines 30-34).\nCalculates pairwise intersection areas between boxes (lines 36-51).",
+ "type": "comment"
+ },
+ "5606": {
+ "file_id": 458,
+ "content": " np.zeros(all_pairs_max_xmin.shape),\n all_pairs_min_xmax - all_pairs_max_xmin)\n return intersect_heights * intersect_widths\ndef iou(boxes1, boxes2):\n \"\"\"Computes pairwise intersection-over-union between box collections.\n Args:\n boxes1: a numpy array with shape [N, 4] holding N boxes.\n boxes2: a numpy array with shape [M, 4] holding N boxes.\n Returns:\n a numpy array with shape [N, M] representing pairwise iou scores.\n \"\"\"\n intersect = intersection(boxes1, boxes2)\n area1 = area(boxes1)\n area2 = area(boxes2)\n union = (\n np.expand_dims(area1, axis=1) + np.expand_dims(area2, axis=0) -\n intersect)\n return intersect / union\ndef ioa(boxes1, boxes2):\n \"\"\"Computes pairwise intersection-over-area between box collections.\n Intersection-over-area (ioa) between two boxes box1 and box2 is defined as\n their intersection area over box2's area. Note that ioa is not symmetric,\n that is, IOA(box1, box2) != IOA(box2, box1).\n Args:\n boxes1: a numpy array with shape [N, 4] holding N boxes.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_ops.py:58-90"
+ },
+ "5607": {
+ "file_id": 458,
+ "content": "The code defines functions for computing pairwise intersection-over-union (iou) and intersection-over-area (ioa) between box collections. The iou function takes two numpy arrays of boxes, computes their intersection using the intersection function, calculates the union by adding the areas of both boxes and subtracting the intersection, and finally returns the pairwise iou scores. The ioa function also takes two numpy arrays of boxes, defines intersection-over-area as the intersection area divided by box2's area, and does not consider symmetry between box1 and box2.",
+ "type": "comment"
+ },
+ "5608": {
+ "file_id": 458,
+ "content": " boxes2: a numpy array with shape [M, 4] holding N boxes.\n Returns:\n a numpy array with shape [N, M] representing pairwise ioa scores.\n \"\"\"\n intersect = intersection(boxes1, boxes2)\n areas = np.expand_dims(area(boxes2), axis=0)\n return intersect / areas",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/np_box_ops.py:91-98"
+ },
+ "5609": {
+ "file_id": 458,
+ "content": "This code calculates pairwise Intersection over Union (IoU) scores between two sets of bounding boxes represented by numpy arrays. It first computes the intersection of the two sets, then calculates the area of each box in the second set, and finally divides the intersection by the areas to obtain the IoU scores.",
+ "type": "comment"
+ },
+ "5610": {
+ "file_id": 459,
+ "content": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py",
+ "type": "filepath"
+ },
+ "5611": {
+ "file_id": 459,
+ "content": "PaddleVideo's \"object_detection_evaluation\" module offers `ObjectDetectionEvaluator` for evaluating object detection outcomes, including metrics like mAP and mean correct localization, considering IOU threshold. It handles AVA dataset and computes AVA metrics for average precision and mean average precision.",
+ "type": "summary"
+ },
+ "5612": {
+ "file_id": 459,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# =============================================================================\n\"\"\"object_detection_evaluation module.\nObjectDetectionEvaluation is a class which manages ground truth information of\na object detection dataset, and computes frequently used detection metrics such\nas Precision, Recall, CorLoc of the provided detection results.\nIt supports the following operations:\n1) Add ground truth information of images sequentially.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:1-21"
+ },
+ "5613": {
+ "file_id": 459,
+ "content": "The provided code is a part of the \"object_detection_evaluation\" module in the PaddleVideo library. This module provides a class called \"ObjectDetectionEvaluation\" that manages ground truth information for object detection datasets, computes frequently used metrics like Precision, Recall, and CorLoc from detection results. The class supports adding ground truth information sequentially and various operations for evaluation.",
+ "type": "comment"
+ },
+ "5614": {
+ "file_id": 459,
+ "content": "2) Add detection result of images sequentially.\n3) Evaluate detection metrics on already inserted detection results.\n4) Write evaluation result into a pickle file for future processing or\n visualization.\nNote: This module operates on numpy boxes and box lists.\n\"\"\"\nimport collections\nimport logging\nfrom abc import ABCMeta, abstractmethod\nimport numpy as np\nfrom . import metrics, per_image_evaluation, standard_fields\nclass DetectionEvaluator:\n \"\"\"Interface for object detection evalution classes.\n Example usage of the Evaluator:\n ------------------------------\n evaluator = DetectionEvaluator(categories)\n # Detections and groundtruth for image 1.\n evaluator.add_single_groundtruth_image_info(...)\n evaluator.add_single_detected_image_info(...)\n # Detections and groundtruth for image 2.\n evaluator.add_single_groundtruth_image_info(...)\n evaluator.add_single_detected_image_info(...)\n metrics_dict = evaluator.evaluate()\n \"\"\"\n __metaclass__ = ABCMeta\n def __init__(self, categories):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:22-58"
+ },
+ "5615": {
+ "file_id": 459,
+ "content": "This code defines an abstract class `DetectionEvaluator` for evaluating object detection results. It takes categories as input and allows adding single ground truth and detected image information. After adding all the data, it can be evaluated to get a metrics dictionary. This evaluation is done on numpy boxes and box lists.",
+ "type": "comment"
+ },
+ "5616": {
+ "file_id": 459,
+ "content": " \"\"\"Constructor.\n Args:\n categories: A list of dicts, each of which has the following keys -\n 'id': (required) an integer id uniquely identifying this\n category.\n 'name': (required) string representing category name e.g.,\n 'cat', 'dog'.\n \"\"\"\n self._categories = categories\n @abstractmethod\n def add_single_ground_truth_image_info(self, image_id, groundtruth_dict):\n \"\"\"Adds groundtruth for a single image to be used for evaluation.\n Args:\n image_id: A unique string/integer identifier for the image.\n groundtruth_dict: A dictionary of groundtruth numpy arrays required\n for evaluations.\n \"\"\"\n @abstractmethod\n def add_single_detected_image_info(self, image_id, detections_dict):\n \"\"\"Adds detections for a single image to be used for evaluation.\n Args:\n image_id: A unique string/integer identifier for the image.\n detections_dict: A dictionary of detection numpy arrays required",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:59-86"
+ },
+ "5617": {
+ "file_id": 459,
+ "content": "This code defines a class constructor that takes categories as input and provides two abstract methods for adding ground truth and detected image information for evaluation. The categories are used to uniquely identify different objects in the images.",
+ "type": "comment"
+ },
+ "5618": {
+ "file_id": 459,
+ "content": " for evaluation.\n \"\"\"\n @abstractmethod\n def evaluate(self):\n \"\"\"Evaluates detections and returns a dictionary of metrics.\"\"\"\n @abstractmethod\n def clear(self):\n \"\"\"Clears the state to prepare for a fresh evaluation.\"\"\"\nclass ObjectDetectionEvaluator(DetectionEvaluator):\n \"\"\"A class to evaluate detections.\"\"\"\n def __init__(\n self,\n categories,\n matching_iou_threshold=0.5,\n evaluate_corlocs=False,\n metric_prefix=None,\n use_weighted_mean_ap=False,\n evaluate_masks=False,\n ):\n \"\"\"Constructor.\n Args:\n categories: A list of dicts, each of which has the following keys -\n 'id': (required) an integer id uniquely identifying this\n category.\n 'name': (required) string representing category name e.g.,\n 'cat', 'dog'.\n matching_iou_threshold: IOU threshold to use for matching\n groundtruth boxes to detection boxes.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:87-120"
+ },
+ "5619": {
+ "file_id": 459,
+ "content": "This code defines an `ObjectDetectionEvaluator` class that evaluates object detection results. It takes categories, IOU threshold, options for evaluating corner localizations and masks. The `evaluate()` method returns a dictionary of metrics, while the `clear()` method clears the state for a new evaluation.",
+ "type": "comment"
+ },
+ "5620": {
+ "file_id": 459,
+ "content": " evaluate_corlocs: (optional) boolean which determines if corloc\n scores are to be returned or not.\n metric_prefix: (optional) string prefix for metric name; if None,\n no prefix is used.\n use_weighted_mean_ap: (optional) boolean which determines if the\n mean average precision is computed directly from the scores and\n tp_fp_labels of all classes.\n evaluate_masks: If False, evaluation will be performed based on\n boxes. If True, mask evaluation will be performed instead.\n Raises:\n ValueError: If the category ids are not 1-indexed.\n \"\"\"\n super(ObjectDetectionEvaluator, self).__init__(categories)\n self._num_classes = max([cat['id'] for cat in categories])\n if min(cat['id'] for cat in categories) < 1:\n raise ValueError('Classes should be 1-indexed.')\n self._matching_iou_threshold = matching_iou_threshold\n self._use_weighted_mean_ap = use_weighted_mean_ap",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:121-139"
+ },
+ "5621": {
+ "file_id": 459,
+ "content": "This code is initializing the ObjectDetectionEvaluator class, which evaluates object detection performance. It takes in optional parameters for corloc scores, metric prefix, and weighted mean AP computation. It checks if category IDs are 1-indexed and raises a ValueError if not.",
+ "type": "comment"
+ },
+ "5622": {
+ "file_id": 459,
+ "content": " self._label_id_offset = 1\n self._evaluate_masks = evaluate_masks\n self._evaluation = ObjectDetectionEvaluation(\n num_groundtruth_classes=self._num_classes,\n matching_iou_threshold=self._matching_iou_threshold,\n use_weighted_mean_ap=self._use_weighted_mean_ap,\n label_id_offset=self._label_id_offset,\n )\n self._image_ids = set([])\n self._evaluate_corlocs = evaluate_corlocs\n self._metric_prefix = (metric_prefix + '_') if metric_prefix else ''\n def add_single_ground_truth_image_info(self, image_id, groundtruth_dict):\n \"\"\"Adds groundtruth for a single image to be used for evaluation.\n Args:\n image_id: A unique string/integer identifier for the image.\n groundtruth_dict: A dictionary containing -\n standard_fields.InputDataFields.groundtruth_boxes: float32\n numpy array of shape [num_boxes, 4] containing `num_boxes`\n groundtruth boxes of the format [ymin, xmin, ymax, xmax] in",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:140-160"
+ },
+ "5623": {
+ "file_id": 459,
+ "content": "This code is initializing an object detection evaluation module, specifically for the Aggregated Average Precision metric. The module takes in parameters such as the number of ground truth classes, matching IOU threshold, and a label offset. It also adds a single image's ground truth information for evaluation purposes. The function expects an image ID and a dictionary containing ground truth boxes information.",
+ "type": "comment"
+ },
+ "5624": {
+ "file_id": 459,
+ "content": " absolute image coordinates.\n standard_fields.InputDataFields.groundtruth_classes: integer\n numpy array of shape [num_boxes] containing 1-indexed\n groundtruth classes for the boxes.\n standard_fields.InputDataFields.groundtruth_difficult: Optional\n length M numpy boolean array denoting whether a ground\n truth box is a difficult instance or not. This field is\n optional to support the case that no boxes are difficult.\n standard_fields.InputDataFields.groundtruth_instance_masks:\n Optional numpy array of shape [num_boxes, height, width]\n with values in {0, 1}.\n Raises:\n ValueError: On adding groundtruth for an image more than once. Will\n also raise error if instance masks are not in groundtruth\n dictionary.\n \"\"\"\n if image_id in self._image_ids:\n raise ValueError(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:161-179"
+ },
+ "5625": {
+ "file_id": 459,
+ "content": "The code is defining the `add_groundtruth` function that takes in an image ID and a groundtruth dictionary. It checks if the image ID has been added before, and raises a ValueError if it has. The groundtruth dictionary should contain 'boxes', 'groundtruth_classes', 'groundtruth_difficult' (optional), and 'groundtruth_instance_masks' (if difficult instances). If the groundtruth is valid, it adds the information to the _image_ids set and initializes corresponding arrays for that image ID. If instance masks are not in the groundtruth dictionary, it raises a ValueError.",
+ "type": "comment"
+ },
+ "5626": {
+ "file_id": 459,
+ "content": " 'Image with id {} already added.'.format(image_id))\n groundtruth_classes = (\n groundtruth_dict[\n standard_fields.InputDataFields.groundtruth_classes] -\n self._label_id_offset)\n # If the key is not present in the groundtruth_dict or the array is\n # empty (unless there are no annotations for the groundtruth on this\n # image) use values from the dictionary or insert None otherwise.\n if (standard_fields.InputDataFields.groundtruth_difficult\n in groundtruth_dict.keys()) and (groundtruth_dict[\n standard_fields.InputDataFields.groundtruth_difficult].size\n or\n not groundtruth_classes.size):\n groundtruth_difficult = groundtruth_dict[\n standard_fields.InputDataFields.groundtruth_difficult]\n else:\n groundtruth_difficult = None\n if not len(self._image_ids) % 1000:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:180-198"
+ },
+ "5627": {
+ "file_id": 459,
+ "content": "This code is checking if an image with a specific id already exists. If not, it retrieves the groundtruth classes and difficult labels from the dictionary, either from existing keys or by setting them to None if not present or empty. It also checks if the image_id is already added to avoid duplicates.",
+ "type": "comment"
+ },
+ "5628": {
+ "file_id": 459,
+ "content": " logging.warn(('image %s does not have groundtruth difficult '\n 'flag specified'), image_id)\n groundtruth_masks = None\n if self._evaluate_masks:\n if (standard_fields.InputDataFields.groundtruth_instance_masks\n not in groundtruth_dict):\n raise ValueError(\n 'Instance masks not in groundtruth dictionary.')\n groundtruth_masks = groundtruth_dict[\n standard_fields.InputDataFields.groundtruth_instance_masks]\n self._evaluation.add_single_ground_truth_image_info(\n image_key=image_id,\n groundtruth_boxes=groundtruth_dict[\n standard_fields.InputDataFields.groundtruth_boxes],\n groundtruth_class_labels=groundtruth_classes,\n groundtruth_is_difficult_list=groundtruth_difficult,\n groundtruth_masks=groundtruth_masks,\n )\n self._image_ids.update([image_id])\n def add_single_detected_image_info(self, image_id, detections_dict):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:199-219"
+ },
+ "5629": {
+ "file_id": 459,
+ "content": "This code block checks if the ground truth difficult flag is specified for an image and raises a warning if not. It then adds single ground truth image information, including bounding boxes, class labels, and mask (if available), to the evaluation object. This allows for evaluating the performance of the object detection model on the given image.",
+ "type": "comment"
+ },
+ "5630": {
+ "file_id": 459,
+ "content": " \"\"\"Adds detections for a single image to be used for evaluation.\n Args:\n image_id: A unique string/integer identifier for the image.\n detections_dict: A dictionary containing -\n standard_fields.DetectionResultFields.detection_boxes: float32\n numpy array of shape [num_boxes, 4] containing `num_boxes`\n detection boxes of the format [ymin, xmin, ymax, xmax] in\n absolute image coordinates.\n standard_fields.DetectionResultFields.detection_scores: float32\n numpy array of shape [num_boxes] containing detection\n scores for the boxes.\n standard_fields.DetectionResultFields.detection_classes:\n integer numpy array of shape [num_boxes] containing\n 1-indexed detection classes for the boxes.\n standard_fields.DetectionResultFields.detection_masks: uint8\n numpy array of shape [num_boxes, height, width] containing",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:220-236"
+ },
+ "5631": {
+ "file_id": 459,
+ "content": "This code snippet adds detections for a single image to be used in evaluation. It takes in image_id and a dictionary containing detection boxes, scores, classes, and masks as input. The detection boxes are represented by a float32 numpy array of shape [num_boxes, 4] with the format [ymin, xmin, ymax, xmax] in absolute image coordinates. Detection scores and classes are integer numpy arrays representing the scores and classes for each box respectively, while detection masks are represented by a uint8 numpy array of shape [num_boxes, height, width].",
+ "type": "comment"
+ },
+ "5632": {
+ "file_id": 459,
+ "content": " `num_boxes` masks of values ranging between 0 and 1.\n Raises:\n ValueError: If detection masks are not in detections dictionary.\n \"\"\"\n detection_classes = (\n detections_dict[\n standard_fields.DetectionResultFields.detection_classes] -\n self._label_id_offset)\n detection_masks = None\n if self._evaluate_masks:\n if (standard_fields.DetectionResultFields.detection_masks\n not in detections_dict):\n raise ValueError(\n 'Detection masks not in detections dictionary.')\n detection_masks = detections_dict[\n standard_fields.DetectionResultFields.detection_masks]\n self._evaluation.add_single_detected_image_info(\n image_key=image_id,\n detected_boxes=detections_dict[\n standard_fields.DetectionResultFields.detection_boxes],\n detected_scores=detections_dict[\n standard_fields.DetectionResultFields.detection_scores],",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:237-259"
+ },
+ "5633": {
+ "file_id": 459,
+ "content": "This code block retrieves detection classes and masks from the \"detections_dict\" dictionary. If _evaluate_Masks is True, it checks if detection masks are present in detections_dict. If not, it raises a ValueError. Then, it adds single detected image information to _evaluation using detected boxes, scores, and (optionally) detection masks.",
+ "type": "comment"
+ },
+ "5634": {
+ "file_id": 459,
+ "content": " detected_class_labels=detection_classes,\n detected_masks=detection_masks,\n )\n def create_category_index(self, categories):\n \"\"\"Creates dictionary of COCO compatible categories keyed by category\n id.\n Args:\n categories: a list of dicts, each of which has the following keys:\n 'id': (required) an integer id uniquely identifying this\n category.\n 'name': (required) string representing category name\n e.g., 'cat', 'dog', 'pizza'.\n Returns:\n category_index: a dict containing the same entries as categories,\n but keyed by the 'id' field of each category.\n \"\"\"\n category_index = {}\n for cat in categories:\n category_index[cat['id']] = cat\n return category_index\n def evaluate(self):\n \"\"\"Compute evaluation result.\n Returns:\n A dictionary of metrics with the following fields -\n 1. summary_metrics:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:260-290"
+ },
+ "5635": {
+ "file_id": 459,
+ "content": "This code is related to object detection evaluation in the AVA dataset. The `create_category_index` function creates a dictionary of COCO compatible categories, keyed by category id. The `evaluate` function computes the evaluation results, returning a dictionary of metrics including summary_metrics. The code also uses `detection_classes` and `detection_masks` for evaluation.",
+ "type": "comment"
+ },
+ "5636": {
+ "file_id": 459,
+ "content": " 'Precision/mAP@IOU': mean average\n precision at the specified IOU threshold\n 2. per_category_ap: category specific results with keys of the form\n 'PerformanceByCategory/mAP@IOU/category'\n \"\"\"\n (\n per_class_ap,\n mean_ap,\n _,\n _,\n per_class_corloc,\n mean_corloc,\n ) = self._evaluation.evaluate()\n metric = f'mAP@{self._matching_iou_threshold}IOU'\n pascal_metrics = {self._metric_prefix + metric: mean_ap}\n if self._evaluate_corlocs:\n pascal_metrics[self._metric_prefix +\n 'Precision/meanCorLoc@{}IOU'.format(\n self._matching_iou_threshold)] = mean_corloc\n category_index = self.create_category_index(self._categories)\n for idx in range(per_class_ap.size):\n if idx + self._label_id_offset in category_index:\n display_name = (",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:291-315"
+ },
+ "5637": {
+ "file_id": 459,
+ "content": "This code calculates the mean average precision (mAP) and optionally, the mean correct localization score (meanCorLoc), at a specified IOU threshold for object detection evaluation. It creates metrics under different categories using category-specific results. The calculated values are then stored in the pascal_metrics dictionary.",
+ "type": "comment"
+ },
+ "5638": {
+ "file_id": 459,
+ "content": " self._metric_prefix +\n 'PerformanceByCategory/AP@{}IOU/{}'.format(\n self._matching_iou_threshold,\n category_index[idx + self._label_id_offset]['name'],\n ))\n pascal_metrics[display_name] = per_class_ap[idx]\n # Optionally add CorLoc metrics.classes\n if self._evaluate_corlocs: #False\n display_name = (\n self._metric_prefix +\n 'PerformanceByCategory/CorLoc@{}IOU/{}'.format(\n self._matching_iou_threshold,\n category_index[idx +\n self._label_id_offset]['name'],\n ))\n pascal_metrics[display_name] = per_class_corloc[idx]\n return pascal_metrics\n def clear(self):\n \"\"\"Clears the state to prepare for a fresh evaluation.\"\"\"\n self._evaluation = ObjectDetectionEvaluation(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:316-338"
+ },
+ "5639": {
+ "file_id": 459,
+ "content": "This code calculates average precision (AP) and optional correct localization (CorLoc) metrics for object detection by category. It appends these metrics to the pascal_metrics dictionary based on the matching IOU threshold and category names from the category_index. The clear() function resets the evaluation state for a new evaluation.",
+ "type": "comment"
+ },
+ "5640": {
+ "file_id": 459,
+ "content": " num_groundtruth_classes=self._num_classes,\n matching_iou_threshold=self._matching_iou_threshold,\n use_weighted_mean_ap=self._use_weighted_mean_ap,\n label_id_offset=self._label_id_offset,\n )\n self._image_ids.clear()\nclass PascalDetectionEvaluator(ObjectDetectionEvaluator):\n \"\"\"A class to evaluate detections using PASCAL metrics.\"\"\"\n def __init__(self, categories, matching_iou_threshold=0.5):\n super(PascalDetectionEvaluator, self).__init__(\n categories,\n matching_iou_threshold=matching_iou_threshold,\n evaluate_corlocs=False,\n use_weighted_mean_ap=False,\n )\nObjectDetectionEvalMetrics = collections.namedtuple(\n 'ObjectDetectionEvalMetrics',\n [\n 'average_precisions',\n 'mean_ap',\n 'precisions',\n 'recalls',\n 'corlocs',\n 'mean_corloc',\n ],\n)\nclass ObjectDetectionEvaluation:\n \"\"\"Internal implementation of Pascal object detection metrics.\"\"\"\n def __init__(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:339-375"
+ },
+ "5641": {
+ "file_id": 459,
+ "content": "The given code is a part of an object detection evaluation module in PaddleVideo. It defines classes and functions to evaluate detections using PASCAL metrics. The ObjectDetectionEvaluator class initializes with categories, matching IoU threshold, evaluating corlocs flag, and use weighted mean AP flag. PascalDetectionEvaluator is a subclass of ObjectDetectionEvaluator specifically for PASCAL evaluation. The code also defines the ObjectDetectionEvalMetrics namedtuple which includes average_precisions, mean_ap, precisions, recalls, corlocs, and mean_corloc attributes.",
+ "type": "comment"
+ },
+ "5642": {
+ "file_id": 459,
+ "content": " self,\n num_groundtruth_classes,\n matching_iou_threshold=0.5,\n nms_iou_threshold=1.0,\n nms_max_output_boxes=10000,\n use_weighted_mean_ap=False,\n label_id_offset=0,\n ):\n if num_groundtruth_classes < 1:\n raise ValueError(\n 'Need at least 1 groundtruth class for evaluation.')\n self.per_image_eval = per_image_evaluation.PerImageEvaluation(\n num_groundtruth_classes=num_groundtruth_classes,\n matching_iou_threshold=matching_iou_threshold,\n )\n self.num_class = num_groundtruth_classes\n self.use_weighted_mean_ap = use_weighted_mean_ap\n self.label_id_offset = label_id_offset\n self.groundtruth_boxes = {}\n self.groundtruth_class_labels = {}\n self.groundtruth_masks = {}\n self.groundtruth_is_difficult_list = {}\n self.groundtruth_is_group_of_list = {}\n self.num_gt_instances_per_class = np.zeros(self.num_class, dtype=int)\n self.num_gt_imgs_per_class = np.zeros(self.num_class, dtype=int)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:376-402"
+ },
+ "5643": {
+ "file_id": 459,
+ "content": "This function initializes the necessary attributes for object detection evaluation. It requires 'self', number of ground truth classes, matching and nms iou thresholds, maximum output boxes, whether to use weighted mean AP, label offset, and sets up dictionaries to store ground truth information. It also initializes counters for the number of instances and images per class.",
+ "type": "comment"
+ },
+ "5644": {
+ "file_id": 459,
+ "content": " self._initialize_detections()\n def _initialize_detections(self):\n self.detection_keys = set()\n self.scores_per_class = [[] for _ in range(self.num_class)]\n self.tp_fp_labels_per_class = [[] for _ in range(self.num_class)]\n self.num_images_correctly_detected_per_class = np.zeros(self.num_class)\n self.average_precision_per_class = np.empty(\n self.num_class, dtype=float)\n self.average_precision_per_class.fill(np.nan)\n self.precisions_per_class = []\n self.recalls_per_class = []\n self.corloc_per_class = np.ones(self.num_class, dtype=float)\n def clear_detections(self):\n self._initialize_detections()\n def add_single_ground_truth_image_info(\n self,\n image_key,\n groundtruth_boxes,\n groundtruth_class_labels,\n groundtruth_is_difficult_list=None,\n groundtruth_is_group_of_list=None,\n groundtruth_masks=None,\n ):\n \"\"\"Adds groundtruth for a single image to be used for evaluation.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:404-430"
+ },
+ "5645": {
+ "file_id": 459,
+ "content": "This code initializes the detection variables and provides functions to clear detections, add single ground truth image info, and perform evaluation. The average precision per class is initialized with nan values, and these functions manage the data for object detection evaluation.",
+ "type": "comment"
+ },
+ "5646": {
+ "file_id": 459,
+ "content": " Args:\n image_key: A unique string/integer identifier for the image.\n groundtruth_boxes: float32 numpy array of shape [num_boxes, 4]\n containing `num_boxes` groundtruth boxes of the format\n [ymin, xmin, ymax, xmax] in absolute image coordinates.\n groundtruth_class_labels: integer numpy array of shape [num_boxes]\n containing 0-indexed groundtruth classes for the boxes.\n groundtruth_is_difficult_list: A length M numpy boolean array\n denoting whether a ground truth box is a difficult instance or\n not. To support the case that no boxes are difficult, it is by\n default set as None.\n groundtruth_is_group_of_list: A length M numpy boolean array\n denoting whether a ground truth box is a group-of box or not.\n To support the case that no boxes are groups-of, it is by\n default set as None.\n groundtruth_masks: uint8 numpy array of shape",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:432-447"
+ },
+ "5647": {
+ "file_id": 459,
+ "content": "The function takes in an image identifier, ground truth boxes coordinates, class labels for the boxes, a boolean array denoting difficult instances, and another boolean array for group-of boxes. It calculates average precision and recall for object detection using these inputs. The function also supports cases where no boxes are difficult or groups-of.",
+ "type": "comment"
+ },
+ "5648": {
+ "file_id": 459,
+ "content": " [num_boxes, height, width] containing `num_boxes` groundtruth\n masks. The mask values range from 0 to 1.\n \"\"\"\n if image_key in self.groundtruth_boxes:\n logging.warn(('image %s has already been added to the ground '\n 'truth database.'), image_key)\n return\n self.groundtruth_boxes[image_key] = groundtruth_boxes\n self.groundtruth_class_labels[image_key] = groundtruth_class_labels\n self.groundtruth_masks[image_key] = groundtruth_masks\n if groundtruth_is_difficult_list is None:\n num_boxes = groundtruth_boxes.shape[0]\n groundtruth_is_difficult_list = np.zeros(num_boxes, dtype=bool)\n self.groundtruth_is_difficult_list[\n image_key] = groundtruth_is_difficult_list.astype(dtype=bool)\n if groundtruth_is_group_of_list is None:\n num_boxes = groundtruth_boxes.shape[0]\n groundtruth_is_group_of_list = np.zeros(num_boxes, dtype=bool)\n self.groundtruth_is_group_of_list[",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:448-467"
+ },
+ "5649": {
+ "file_id": 459,
+ "content": "This function adds the ground truth boxes, class labels, and masks to the database for a given image key. If the groundtruth_is_difficult_list or groundtruth_is_group_of_list are None, it creates them with default values. It stores these lists as well in the database for the specified image key.",
+ "type": "comment"
+ },
+ "5650": {
+ "file_id": 459,
+ "content": " image_key] = groundtruth_is_group_of_list.astype(dtype=bool)\n self._update_ground_truth_statistics(\n groundtruth_class_labels,\n groundtruth_is_difficult_list.astype(dtype=bool),\n groundtruth_is_group_of_list.astype(dtype=bool),\n )\n def add_single_detected_image_info(\n self,\n image_key,\n detected_boxes,\n detected_scores,\n detected_class_labels,\n detected_masks=None,\n ):\n \"\"\"Adds detections for a single image to be used for evaluation.\n Args:\n image_key: A unique string/integer identifier for the image.\n detected_boxes: float32 numpy array of shape [num_boxes, 4]\n containing `num_boxes` detection boxes of the format\n [ymin, xmin, ymax, xmax] in absolute image coordinates.\n detected_scores: float32 numpy array of shape [num_boxes]\n containing detection scores for the boxes.\n detected_class_labels: integer numpy array of shape [num_boxes]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:468-493"
+ },
+ "5651": {
+ "file_id": 459,
+ "content": "This function adds detections for a single image to be used for evaluation. It requires an image key, detected boxes, detected scores, and detected class labels as input. The detected boxes should be in the format [ymin, xmin, ymax, xmax] and the detected scores and detected class labels should be numpy arrays of the specified shapes. The function calls a _update_ground_truth_statistics method with groundtruth class labels, difficult list, and group of list as input. This method updates the ground truth statistics for evaluation.",
+ "type": "comment"
+ },
+ "5652": {
+ "file_id": 459,
+ "content": " containing 0-indexed detection classes for the boxes.\n detected_masks: np.uint8 numpy array of shape\n [num_boxes, height, width] containing `num_boxes` detection\n masks with values ranging between 0 and 1.\n Raises:\n ValueError: if the number of boxes, scores and class labels differ\n in length.\n \"\"\"\n if len(detected_boxes) != len(detected_scores) or len(\n detected_boxes) != len(detected_class_labels):\n raise ValueError(\n 'detected_boxes, detected_scores and '\n 'detected_class_labels should all have same lengths. Got'\n '[%d, %d, %d]' % len(detected_boxes),\n len(detected_scores),\n len(detected_class_labels),\n )\n if image_key in self.detection_keys:\n logging.warn(('image %s has already been added to the ground '\n 'truth database.'), image_key)\n return",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:494-516"
+ },
+ "5653": {
+ "file_id": 459,
+ "content": "This function creates a numpy array of detection masks based on detected boxes, scores and class labels. It raises a ValueError if the lengths of these lists are not equal. If an image key already exists in the detection keys list, it logs a warning message and returns without adding the image to the database.",
+ "type": "comment"
+ },
+ "5654": {
+ "file_id": 459,
+ "content": " self.detection_keys.add(image_key)\n if image_key in self.groundtruth_boxes:\n groundtruth_boxes = self.groundtruth_boxes[image_key]\n groundtruth_class_labels = self.groundtruth_class_labels[image_key]\n # Masks are popped instead of look up. The reason is that we do not\n # want to keep all masks in memory which can cause memory overflow.\n groundtruth_masks = self.groundtruth_masks.pop(image_key)\n groundtruth_is_difficult_list = self.groundtruth_is_difficult_list[\n image_key]\n groundtruth_is_group_of_list = self.groundtruth_is_group_of_list[\n image_key]\n else:\n groundtruth_boxes = np.empty(shape=[0, 4], dtype=float)\n groundtruth_class_labels = np.array([], dtype=int)\n if detected_masks is None:\n groundtruth_masks = None\n else:\n groundtruth_masks = np.empty(shape=[0, 1, 1], dtype=float)\n groundtruth_is_difficult_list = np.array([], dtype=bool)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:518-536"
+ },
+ "5655": {
+ "file_id": 459,
+ "content": "This code is initializing ground truth values for object detection evaluation. If an image key exists in the ground truth boxes dictionary, it retrieves the corresponding ground truth values (boxes, class labels, masks) and removes them from their respective dictionaries to avoid memory overflow. If no image key exists, it initializes empty arrays or None values for the ground truth values.",
+ "type": "comment"
+ },
+ "5656": {
+ "file_id": 459,
+ "content": " groundtruth_is_group_of_list = np.array([], dtype=bool)\n (\n scores,\n tp_fp_labels,\n ) = self.per_image_eval.compute_object_detection_metrics(\n detected_boxes=detected_boxes,\n detected_scores=detected_scores,\n detected_class_labels=detected_class_labels,\n groundtruth_boxes=groundtruth_boxes,\n groundtruth_class_labels=groundtruth_class_labels,\n groundtruth_is_difficult_list=groundtruth_is_difficult_list,\n groundtruth_is_group_of_list=groundtruth_is_group_of_list,\n detected_masks=detected_masks,\n groundtruth_masks=groundtruth_masks,\n )\n for i in range(self.num_class):\n if scores[i].shape[0] > 0:\n self.scores_per_class[i].append(scores[i])\n self.tp_fp_labels_per_class[i].append(tp_fp_labels[i])\n def _update_ground_truth_statistics(\n self,\n groundtruth_class_labels,\n groundtruth_is_difficult_list,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:537-561"
+ },
+ "5657": {
+ "file_id": 459,
+ "content": "This code is part of the PaddleVideo library and it computes object detection metrics. It takes in detected boxes, scores, class labels, ground truth boxes, class labels, masks etc., and calculates true positive and false positive labels for each image. The computed values are then stored per class in separate lists (scores_per_class and tp_fp_labels_per_class). Additionally, the function updates ground truth statistics by appending new ground truth class labels and difficult list to existing ones.",
+ "type": "comment"
+ },
+ "5658": {
+ "file_id": 459,
+ "content": " groundtruth_is_group_of_list,\n ):\n \"\"\"Update grouth truth statitistics.\n 1. Difficult boxes are ignored when counting the number of ground truth\n instances as done in Pascal VOC devkit.\n 2. Difficult boxes are treated as normal boxes when computing CorLoc\n related statitistics.\n Args:\n groundtruth_class_labels: An integer numpy array of length M,\n representing M class labels of object instances in ground truth\n groundtruth_is_difficult_list: A boolean numpy array of length M\n denoting whether a ground truth box is a difficult instance or\n not\n groundtruth_is_group_of_list: A boolean numpy array of length M\n denoting whether a ground truth box is a group-of box or not\n \"\"\"\n for class_index in range(self.num_class):\n num_gt_instances = np.sum(groundtruth_class_labels[\n ~groundtruth_is_difficult_list\n & ~groundtruth_is_group_of_list] == class_index)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:562-583"
+ },
+ "5659": {
+ "file_id": 459,
+ "content": "This function updates ground truth statistics for object detection by counting instances, excluding difficult boxes and treating them as normal ones for CorLoc computations. It iterates through class indices to determine the number of instances for each class label, excluding difficult or group-of boxes.",
+ "type": "comment"
+ },
+ "5660": {
+ "file_id": 459,
+ "content": " self.num_gt_instances_per_class[class_index] += num_gt_instances\n if np.any(groundtruth_class_labels == class_index):\n self.num_gt_imgs_per_class[class_index] += 1\n def evaluate(self):\n \"\"\"Compute evaluation result.\n Returns:\n A named tuple with the following fields -\n average_precision: float numpy array of average precision for\n each class.\n mean_ap: mean average precision of all classes, float scalar\n precisions: List of precisions, each precision is a float numpy\n array\n recalls: List of recalls, each recall is a float numpy array\n corloc: numpy float array\n mean_corloc: Mean CorLoc score for each class, float scalar\n \"\"\"\n if (self.num_gt_instances_per_class == 0).any():\n print(\n 'The following classes have no ground truth examples: %s',\n np.squeeze(np.argwhere(self.num_gt_instances_per_class == 0)) +",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:584-605"
+ },
+ "5661": {
+ "file_id": 459,
+ "content": "The code calculates average precision, mean average precision, precisions, recalls, and CorLoc scores for object detection evaluation. It checks if any ground truth instances exist for each class and returns a named tuple with evaluation results. If there are classes with no ground truth examples, it prints a warning message.",
+ "type": "comment"
+ },
+ "5662": {
+ "file_id": 459,
+ "content": " self.label_id_offset, \"self.detection_keys:\",self.detection_keys\n )\n if self.use_weighted_mean_ap:\n all_scores = np.array([], dtype=float)\n all_tp_fp_labels = np.array([], dtype=bool)\n for class_index in range(self.num_class):\n if self.num_gt_instances_per_class[class_index] == 0:\n continue\n if not self.scores_per_class[class_index]:\n scores = np.array([], dtype=float)\n tp_fp_labels = np.array([], dtype=bool)\n else:\n scores = np.concatenate(self.scores_per_class[class_index])\n tp_fp_labels = np.concatenate(\n self.tp_fp_labels_per_class[class_index])\n if self.use_weighted_mean_ap:\n all_scores = np.append(all_scores, scores)\n all_tp_fp_labels = np.append(all_tp_fp_labels, tp_fp_labels)\n precision, recall = metrics.compute_precision_recall(\n scores,\n tp_fp_labels,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:606-629"
+ },
+ "5663": {
+ "file_id": 459,
+ "content": "This code is part of a class that performs object detection evaluation using Average Vehicle Accuracy (AVA) metrics. It checks for the number of ground truth instances per class and concatenates scores and true positive/false positive labels per class. If weighted mean average precision (AP) calculation is enabled, it appends the scores and labels to the total arrays. The code uses the compute_precision_recall function from the metrics module to calculate precision and recall values.",
+ "type": "comment"
+ },
+ "5664": {
+ "file_id": 459,
+ "content": " self.num_gt_instances_per_class[class_index],\n )\n self.precisions_per_class.append(precision)\n self.recalls_per_class.append(recall)\n average_precision = metrics.compute_average_precision(\n precision, recall)\n self.average_precision_per_class[class_index] = average_precision\n self.corloc_per_class = metrics.compute_cor_loc(\n self.num_gt_imgs_per_class,\n self.num_images_correctly_detected_per_class,\n )\n if self.use_weighted_mean_ap:\n num_gt_instances = np.sum(self.num_gt_instances_per_class)\n precision, recall = metrics.compute_precision_recall(\n all_scores, all_tp_fp_labels, num_gt_instances)\n mean_ap = metrics.compute_average_precision(precision, recall)\n else:\n mean_ap = np.nanmean(self.average_precision_per_class)\n mean_corloc = np.nanmean(self.corloc_per_class)\n return ObjectDetectionEvalMetrics(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:630-651"
+ },
+ "5665": {
+ "file_id": 459,
+ "content": "This function calculates average precision and correlation localization for object detection evaluation. It stores the precision, recall, average precision per class, and correlation localization per class. If weighted mean AP is enabled, it computes precision, recall, mean AP, and mean correlation localization.",
+ "type": "comment"
+ },
+ "5666": {
+ "file_id": 459,
+ "content": " self.average_precision_per_class,\n mean_ap,\n self.precisions_per_class,\n self.recalls_per_class,\n self.corloc_per_class,\n mean_corloc,\n )",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/object_detection_evaluation.py:652-658"
+ },
+ "5667": {
+ "file_id": 459,
+ "content": "This code snippet appears to be part of a class function that returns several evaluation metrics for object detection. The metrics include average precision per class, mean average precision, precisions and recalls per class, and mean corloc values. These metrics are commonly used in evaluating object detection models' performance.",
+ "type": "comment"
+ },
+ "5668": {
+ "file_id": 460,
+ "content": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py",
+ "type": "filepath"
+ },
+ "5669": {
+ "file_id": 460,
+ "content": "The code measures object detection performance, handles class labels and non-maximum suppression, and calculates true/false positives using an IoU threshold. It is used for AVA evaluation and contains functions to select class-specific data, remove invalid boxes, and filter input arrays.",
+ "type": "summary"
+ },
+ "5670": {
+ "file_id": 460,
+ "content": "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# =============================================================================\n\"\"\"Evaluate Object Detection result on a single image.\nAnnotate each detected result as true positives or false positive according to\na predefined IOU ratio. Non Maximum Supression is used by default. Multi class\ndetection is supported by default. Based on the settings, per image evaluation\nis either performed on boxes or on object masks.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:1-20"
+ },
+ "5671": {
+ "file_id": 460,
+ "content": "This code file is for evaluating object detection results on a single image. It determines true positives or false positives based on a predefined IOU ratio. Non Maximum Supression and multi-class detection are supported. The evaluation can be performed either on boxes or object masks, depending on the settings.",
+ "type": "comment"
+ },
+ "5672": {
+ "file_id": 460,
+ "content": "\"\"\"\nimport numpy as np\nfrom . import np_box_list, np_box_ops\nclass PerImageEvaluation:\n \"\"\"Evaluate detection result of a single image.\"\"\"\n def __init__(self, num_groundtruth_classes, matching_iou_threshold=0.5):\n \"\"\"Initialized PerImageEvaluation by evaluation parameters.\n Args:\n num_groundtruth_classes: Number of ground truth object classes\n matching_iou_threshold: A ratio of area intersection to union,\n which is the threshold to consider whether a detection is true\n positive or not\n \"\"\"\n self.matching_iou_threshold = matching_iou_threshold\n self.num_groundtruth_classes = num_groundtruth_classes\n def compute_object_detection_metrics(\n self,\n detected_boxes,\n detected_scores,\n detected_class_labels,\n groundtruth_boxes,\n groundtruth_class_labels,\n groundtruth_is_difficult_list,\n groundtruth_is_group_of_list,\n detected_masks=None,\n groundtruth_masks=None,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:21-53"
+ },
+ "5673": {
+ "file_id": 460,
+ "content": "This code initializes a class for evaluating detection results of a single image. It takes in parameters such as the number of ground truth classes and matching IOU threshold, and computes object detection metrics using detected boxes, scores, class labels, etc.",
+ "type": "comment"
+ },
+ "5674": {
+ "file_id": 460,
+ "content": " ):\n \"\"\"Evaluates detections as being tp, fp or ignored from a single image.\n The evaluation is done in two stages:\n 1. All detections are matched to non group-of boxes; true positives\n are determined and detections matched to difficult boxes are\n ignored.\n 2. Detections that are determined as false positives are matched\n against group-of boxes and ignored if matched.\n Args:\n detected_boxes: A float numpy array of shape [N, 4], representing N\n regions of detected object regions.\n Each row is of the format [y_min, x_min, y_max, x_max]\n detected_scores: A float numpy array of shape [N, 1], representing\n the confidence scores of the detected N object instances.\n detected_class_labels: A integer numpy array of shape [N, 1],\n repreneting the class labels of the detected N object\n instances.\n groundtruth_boxes: A float numpy array of shape [M, 4],",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:54-73"
+ },
+ "5675": {
+ "file_id": 460,
+ "content": "This function evaluates detections as true positives, false positives or ignored based on the detected and ground truth boxes. It works in two stages: 1) matching all detections to non group-of boxes for true positives, ignoring difficult ones; and 2) ignoring detections matched to group-of boxes. The inputs are numpy arrays of detected boxes, scores, class labels, and ground truth boxes.",
+ "type": "comment"
+ },
+ "5676": {
+ "file_id": 460,
+ "content": " representing M regions of object instances in ground truth\n groundtruth_class_labels: An integer numpy array of shape [M, 1],\n representing M class labels of object instances in ground truth\n groundtruth_is_difficult_list: A boolean numpy array of length M\n denoting whether a ground truth box is a difficult instance or\n not\n groundtruth_is_group_of_list: A boolean numpy array of length M\n denoting whether a ground truth box has group-of tag\n detected_masks: (optional) A uint8 numpy array of shape\n [N, height, width]. If not None, the metrics will be computed\n based on masks.\n groundtruth_masks: (optional) A uint8 numpy array of shape\n [M, height, width].\n Returns:\n scores: A list of C float numpy arrays. Each numpy array is of\n shape [K, 1], representing K scores detected with object class\n label c",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:74-91"
+ },
+ "5677": {
+ "file_id": 460,
+ "content": "This code function accepts multiple parameters including ground truth regions, class labels, difficult instances, group-of tags, optional detected masks and ground truth masks. It returns a list of scores representing K scores detected with object class label c.",
+ "type": "comment"
+ },
+ "5678": {
+ "file_id": 460,
+ "content": " tp_fp_labels: A list of C boolean numpy arrays. Each numpy array\n is of shape [K, 1], representing K True/False positive label of\n object instances detected with class label c\n \"\"\"\n (\n detected_boxes,\n detected_scores,\n detected_class_labels,\n detected_masks,\n ) = self._remove_invalid_boxes(\n detected_boxes,\n detected_scores,\n detected_class_labels,\n detected_masks,\n )\n scores, tp_fp_labels = self._compute_tp_fp(\n detected_boxes=detected_boxes,\n detected_scores=detected_scores,\n detected_class_labels=detected_class_labels,\n groundtruth_boxes=groundtruth_boxes,\n groundtruth_class_labels=groundtruth_class_labels,\n groundtruth_is_difficult_list=groundtruth_is_difficult_list,\n groundtruth_is_group_of_list=groundtruth_is_group_of_list,\n detected_masks=detected_masks,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:92-115"
+ },
+ "5679": {
+ "file_id": 460,
+ "content": "This function is part of the AvaEvaluation class, which evaluates object detection performance in videos. It computes true positive and false positive labels for detected object instances based on ground truth information. The function removes invalid detection boxes before computing the tp_fp_labels. This helps in evaluating the accuracy of detected objects.",
+ "type": "comment"
+ },
+ "5680": {
+ "file_id": 460,
+ "content": " groundtruth_masks=groundtruth_masks,\n )\n return scores, tp_fp_labels\n def _compute_tp_fp(\n self,\n detected_boxes,\n detected_scores,\n detected_class_labels,\n groundtruth_boxes,\n groundtruth_class_labels,\n groundtruth_is_difficult_list,\n groundtruth_is_group_of_list,\n detected_masks=None,\n groundtruth_masks=None,\n ):\n \"\"\"Labels true/false positives of detections of an image across all\n classes.\n Args:\n detected_boxes: A float numpy array of shape [N, 4], representing N\n regions of detected object regions.\n Each row is of the format [y_min, x_min, y_max, x_max]\n detected_scores: A float numpy array of shape [N, 1], representing\n the confidence scores of the detected N object instances.\n detected_class_labels: A integer numpy array of shape [N, 1],\n repreneting the class labels of the detected N object",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:116-143"
+ },
+ "5681": {
+ "file_id": 460,
+ "content": "This code calculates true/false positives for object detection in an image across all classes. It takes detected boxes, scores, class labels, ground truth boxes, class labels, and optional masks as input, returning the computed scores and tp_fp_labels. The separate function computes tp_fp for a single image given the above inputs.",
+ "type": "comment"
+ },
+ "5682": {
+ "file_id": 460,
+ "content": " instances.\n groundtruth_boxes: A float numpy array of shape [M, 4],\n representing M regions of object instances in ground truth\n groundtruth_class_labels: An integer numpy array of shape [M, 1],\n representing M class labels of object instances in ground truth\n groundtruth_is_difficult_list: A boolean numpy array of length M\n denoting whether a ground truth box is a difficult instance or\n not\n groundtruth_is_group_of_list: A boolean numpy array of length M\n denoting whether a ground truth box has group-of tag\n detected_masks: (optional) A np.uint8 numpy array of shape\n [N, height, width]. If not None, the scores will be computed\n based on masks.\n groundtruth_masks: (optional) A np.uint8 numpy array of shape\n [M, height, width].\n Returns:\n result_scores: A list of float numpy arrays. Each numpy array is of",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:144-161"
+ },
+ "5683": {
+ "file_id": 460,
+ "content": "The function takes input parameters like instances, groundtruth_boxes, groundtruth_class_labels, groundtruth_is_difficult_list, groundtruth_is_group_of_list, detected_masks and groundtruth_masks. It returns a list of float numpy arrays representing result scores based on these inputs. The function computes scores considering masks if detected_masks is not None and groundtruth_masks if it's not None.",
+ "type": "comment"
+ },
+ "5684": {
+ "file_id": 460,
+ "content": " shape [K, 1], representing K scores detected with object class\n label c\n result_tp_fp_labels: A list of boolean numpy array. Each numpy\n array is of shape [K, 1], representing K True/False positive\n label of object instances detected with class label c\n Raises:\n ValueError: If detected masks is not None but groundtruth masks are\n None, or the other way around.\n \"\"\"\n if detected_masks is not None and groundtruth_masks is None:\n raise ValueError(\n 'Detected masks is available but groundtruth masks is not.')\n if detected_masks is None and groundtruth_masks is not None:\n raise ValueError(\n 'Groundtruth masks is available but detected masks is not.')\n result_scores = []\n result_tp_fp_labels = []\n for i in range(self.num_groundtruth_classes):\n groundtruth_is_difficult_list_at_ith_class = (\n groundtruth_is_difficult_list[groundtruth_class_labels == i])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:162-183"
+ },
+ "5685": {
+ "file_id": 460,
+ "content": "This function checks if both detected_masks and groundtruth_masks are not null. It then initializes result_scores and result_tp_fp_labels lists for storing scores and True/False positive labels of object instances respectively. If only one of the masks is available, it raises a ValueError. This function seems to be part of an AVA evaluation process where it deals with class label c, groundtruth_class_labels, detected_masks, and groundtruth_masks.",
+ "type": "comment"
+ },
+ "5686": {
+ "file_id": 460,
+ "content": " groundtruth_is_group_of_list_at_ith_class = (\n groundtruth_is_group_of_list[groundtruth_class_labels == i])\n (\n gt_boxes_at_ith_class,\n gt_masks_at_ith_class,\n detected_boxes_at_ith_class,\n detected_scores_at_ith_class,\n detected_masks_at_ith_class,\n ) = self._get_ith_class_arrays(detected_boxes, detected_scores,\n detected_masks,\n detected_class_labels,\n groundtruth_boxes,\n groundtruth_masks,\n groundtruth_class_labels, i)\n scores, tp_fp_labels = self._compute_tp_fp_for_single_class(\n detected_boxes=detected_boxes_at_ith_class,\n detected_scores=detected_scores_at_ith_class,\n groundtruth_boxes=gt_boxes_at_ith_class,\n groundtruth_is_difficult_list=(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:184-202"
+ },
+ "5687": {
+ "file_id": 460,
+ "content": "The code is extracting per-class arrays for detected and ground truth objects. It separates the data into specific classes, computes true positive and false positive labels using a single class function, and assigns them to their respective variables.",
+ "type": "comment"
+ },
+ "5688": {
+ "file_id": 460,
+ "content": " groundtruth_is_difficult_list_at_ith_class),\n groundtruth_is_group_of_list=(\n groundtruth_is_group_of_list_at_ith_class),\n detected_masks=detected_masks_at_ith_class,\n groundtruth_masks=gt_masks_at_ith_class,\n )\n result_scores.append(scores)\n result_tp_fp_labels.append(tp_fp_labels)\n return result_scores, result_tp_fp_labels\n def _get_overlaps_and_scores_box_mode(\n self,\n detected_boxes,\n detected_scores,\n groundtruth_boxes,\n groundtruth_is_group_of_list,\n ):\n \"\"\"Computes overlaps and scores between detected and groudntruth boxes.\n Args:\n detected_boxes: A numpy array of shape [N, 4] representing detected\n box coordinates\n detected_scores: A 1-d numpy array of length N representing\n classification score\n groundtruth_boxes: A numpy array of shape [M, 4] representing\n ground truth box coordinates",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:203-228"
+ },
+ "5689": {
+ "file_id": 460,
+ "content": "This code is calculating overlapping regions and scores between detected and ground truth boxes. It's taking in arrays of detected box coordinates, classification scores, ground truth box coordinates, and ground truth group indicators. The code then returns the resultant scores and true positive/false positive labels for each image. This seems to be part of an object detection or instance segmentation evaluation metric.",
+ "type": "comment"
+ },
+ "5690": {
+ "file_id": 460,
+ "content": " groundtruth_is_group_of_list: A boolean numpy array of length M\n denoting whether a ground truth box has group-of tag. If a\n groundtruth box is group-of box, every detection matching this\n box is ignored.\n Returns:\n iou: A float numpy array of size [num_detected_boxes,\n num_gt_boxes]. If gt_non_group_of_boxlist.num_boxes() == 0 it\n will be None.\n ioa: A float numpy array of size [num_detected_boxes,\n num_gt_boxes]. If gt_group_of_boxlist.num_boxes() == 0 it will\n be None.\n scores: The score of the detected boxlist.\n num_boxes: Number of non-maximum suppressed detected boxes.\n \"\"\"\n detected_boxlist = np_box_list.BoxList(detected_boxes)\n detected_boxlist.add_field('scores', detected_scores)\n gt_non_group_of_boxlist = np_box_list.BoxList(\n groundtruth_boxes[~groundtruth_is_group_of_list])\n iou = np_box_ops.iou(detected_boxlist.get(),",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:229-249"
+ },
+ "5691": {
+ "file_id": 460,
+ "content": "Code computes intersection over union (IoU) and intersection over area (IoA) between detected boxlists and ground truth boxlists. It also returns the scores of the detected boxes and the number of non-maximum suppressed detected boxes. The ground truth is_group_of_list is used to ignore group-of boxes during calculation.",
+ "type": "comment"
+ },
+ "5692": {
+ "file_id": 460,
+ "content": " gt_non_group_of_boxlist.get())\n scores = detected_boxlist.get_field('scores')\n num_boxes = detected_boxlist.num_boxes()\n return iou, None, scores, num_boxes\n def _compute_tp_fp_for_single_class(\n self,\n detected_boxes,\n detected_scores,\n groundtruth_boxes,\n groundtruth_is_difficult_list,\n groundtruth_is_group_of_list,\n detected_masks=None,\n groundtruth_masks=None,\n ):\n \"\"\"Labels boxes detected with the same class from the same image as\n tp/fp.\n Args:\n detected_boxes: A numpy array of shape [N, 4] representing detected\n box coordinates\n detected_scores: A 1-d numpy array of length N representing\n classification score\n groundtruth_boxes: A numpy array of shape [M, 4] representing\n groundtruth box coordinates\n groundtruth_is_difficult_list: A boolean numpy array of length M\n denoting whether a ground truth box is a difficult instance or",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:250-276"
+ },
+ "5693": {
+ "file_id": 460,
+ "content": "This function labels boxes detected with the same class from the same image as true positives or false positives. It takes in the detected boxes, scores, ground truth boxes, and other relevant information to perform this labeling task. The output is determined based on the intersection-over-union (IoU) threshold between detected and ground truth boxes. If a detected box has an IoU greater than 0.5 with any ground truth box in the same class and image, it is considered a true positive (tp). Otherwise, it's considered a false positive (fp). The function also computes the number of detected boxes.",
+ "type": "comment"
+ },
+ "5694": {
+ "file_id": 460,
+ "content": " not. If a groundtruth box is difficult, every detection\n matching this box is ignored.\n groundtruth_is_group_of_list: A boolean numpy array of length M\n denoting whether a ground truth box has group-of tag. If a\n groundtruth box is group-of box, every detection matching this\n box is ignored.\n detected_masks: (optional) A uint8 numpy array of shape\n [N, height, width]. If not None, the scores will be computed\n based on masks.\n groundtruth_masks: (optional) A uint8 numpy array of shape\n [M, height, width].\n Returns:\n Two arrays of the same size, containing all boxes that were\n evaluated as being true positives or false positives; if a box\n matched to a difficult box or to a group-of box, it is ignored.\n scores: A numpy array representing the detection scores.\n tp_fp_labels: a boolean numpy array indicating whether a detection",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:277-295"
+ },
+ "5695": {
+ "file_id": 460,
+ "content": "This function computes true positive (TP) and false positive (FP) labels for detected boxes based on whether they match difficult or group-of ground truth boxes. It returns scores and TP/FP labels, ignoring any detections that match these challenging boxes. Optional mask inputs are also supported to compute scores based on pixel-wise comparisons instead of bounding box overlaps.",
+ "type": "comment"
+ },
+ "5696": {
+ "file_id": 460,
+ "content": " is a true positive.\n \"\"\"\n if detected_boxes.size == 0:\n return np.array([], dtype=float), np.array([], dtype=bool)\n (\n iou,\n _,\n scores,\n num_detected_boxes,\n ) = self._get_overlaps_and_scores_box_mode(\n detected_boxes=detected_boxes,\n detected_scores=detected_scores,\n groundtruth_boxes=groundtruth_boxes,\n groundtruth_is_group_of_list=groundtruth_is_group_of_list,\n )\n if groundtruth_boxes.size == 0:\n return scores, np.zeros(num_detected_boxes, dtype=bool)\n tp_fp_labels = np.zeros(num_detected_boxes, dtype=bool)\n is_matched_to_difficult_box = np.zeros(num_detected_boxes, dtype=bool)\n is_matched_to_group_of_box = np.zeros(num_detected_boxes, dtype=bool)\n # The evaluation is done in two stages:\n # 1. All detections are matched to non group-of boxes; true positives\n # are determined and detections matched to difficult boxes are",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:296-322"
+ },
+ "5697": {
+ "file_id": 460,
+ "content": "This code checks for true positive detections by first obtaining the Intersection over Union (IoU) and scores between detected boxes and ground truth boxes. If there are no detected or ground truth boxes, it returns empty arrays. Then, it initializes variables to keep track of whether a detection is matched to a difficult box or a group-of box. The code proceeds in two stages: first, all detections are matched to non-group-of boxes, determining true positives, and then detections matched to difficult boxes are identified.",
+ "type": "comment"
+ },
+ "5698": {
+ "file_id": 460,
+ "content": " # ignored.\n # 2. Detections that are determined as false positives are matched\n # against group-of boxes and ignored if matched.\n # Tp-fp evaluation for non-group of boxes (if any).\n if iou.shape[1] > 0:\n groundtruth_nongroup_of_is_difficult_list = (\n groundtruth_is_difficult_list[~groundtruth_is_group_of_list])\n max_overlap_gt_ids = np.argmax(iou, axis=1)\n is_gt_box_detected = np.zeros(iou.shape[1], dtype=bool)\n for i in range(num_detected_boxes):\n gt_id = max_overlap_gt_ids[i]\n if iou[i, gt_id] >= self.matching_iou_threshold:\n if not groundtruth_nongroup_of_is_difficult_list[gt_id]:\n if not is_gt_box_detected[gt_id]:\n tp_fp_labels[i] = True\n is_gt_box_detected[gt_id] = True\n else:\n is_matched_to_difficult_box[i] = True\n return (\n scores[~is_matched_to_difficult_box & ~is_matched_to_group_of_box],",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:323-344"
+ },
+ "5699": {
+ "file_id": 460,
+ "content": "This code performs a TP-FP evaluation for non-group of boxes, ignoring difficult ground truth boxes and false positives matched to group-of boxes. It checks the IOU between detected boxes and ground truth boxes, and assigns labels accordingly.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/57.json b/docs/data/57.json
new file mode 100644
index 000000000..b7745e73b
--- /dev/null
+++ b/docs/data/57.json
@@ -0,0 +1,545 @@
+{
+ "5700": {
+ "file_id": 460,
+ "content": " tp_fp_labels[~is_matched_to_difficult_box\n & ~is_matched_to_group_of_box],\n )\n def _get_ith_class_arrays(\n self,\n detected_boxes,\n detected_scores,\n detected_masks,\n detected_class_labels,\n groundtruth_boxes,\n groundtruth_masks,\n groundtruth_class_labels,\n class_index,\n ):\n \"\"\"Returns numpy arrays belonging to class with index `class_index`.\n Args:\n detected_boxes: A numpy array containing detected boxes.\n detected_scores: A numpy array containing detected scores.\n detected_masks: A numpy array containing detected masks.\n detected_class_labels: A numpy array containing detected class\n labels.\n groundtruth_boxes: A numpy array containing groundtruth boxes.\n groundtruth_masks: A numpy array containing groundtruth masks.\n groundtruth_class_labels: A numpy array containing groundtruth\n class labels.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:345-371"
+ },
+ "5701": {
+ "file_id": 460,
+ "content": "This function, _get_ith_class_arrays, takes in various numpy arrays of detected and ground truth boxes, masks, and class labels. It then returns the corresponding numpy arrays for a specific class index.",
+ "type": "comment"
+ },
+ "5702": {
+ "file_id": 460,
+ "content": " class_index: An integer index.\n Returns:\n gt_boxes_at_ith_class: A numpy array containing groundtruth boxes\n labeled as ith class.\n gt_masks_at_ith_class: A numpy array containing groundtruth masks\n labeled as ith class.\n detected_boxes_at_ith_class: A numpy array containing detected\n boxes corresponding to the ith class.\n detected_scores_at_ith_class: A numpy array containing detected\n scores corresponding to the ith class.\n detected_masks_at_ith_class: A numpy array containing detected\n masks corresponding to the ith class.\n \"\"\"\n selected_groundtruth = groundtruth_class_labels == class_index\n gt_boxes_at_ith_class = groundtruth_boxes[selected_groundtruth]\n if groundtruth_masks is not None:\n gt_masks_at_ith_class = groundtruth_masks[selected_groundtruth]\n else:\n gt_masks_at_ith_class = None\n selected_detections = detected_class_labels == class_index",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:372-392"
+ },
+ "5703": {
+ "file_id": 460,
+ "content": "This function returns ground truth boxes, masks (if provided), detected boxes, scores, and masks for a given class index. It selects the data corresponding to the class index from input arrays and returns them in separate numpy arrays.",
+ "type": "comment"
+ },
+ "5704": {
+ "file_id": 460,
+ "content": " detected_boxes_at_ith_class = detected_boxes[selected_detections]\n detected_scores_at_ith_class = detected_scores[selected_detections]\n if detected_masks is not None:\n detected_masks_at_ith_class = detected_masks[selected_detections]\n else:\n detected_masks_at_ith_class = None\n return (\n gt_boxes_at_ith_class,\n gt_masks_at_ith_class,\n detected_boxes_at_ith_class,\n detected_scores_at_ith_class,\n detected_masks_at_ith_class,\n )\n def _remove_invalid_boxes(\n self,\n detected_boxes,\n detected_scores,\n detected_class_labels,\n detected_masks=None,\n ):\n \"\"\"Removes entries with invalid boxes.\n A box is invalid if either its xmax is smaller than its xmin, or its\n ymax is smaller than its ymin.\n Args:\n detected_boxes: A float numpy array of size [num_boxes, 4]\n containing box coordinates in [ymin, xmin, ymax, xmax] format.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:393-421"
+ },
+ "5705": {
+ "file_id": 460,
+ "content": "This code defines two functions: \n1. _get_class_specific_results: Extracts class-specific results from the given data and returns them in a tuple.\n2. _remove_invalid_boxes: Removes entries with invalid boxes from the given data. An invalid box is one where xmax < xmin or ymax < ymin.",
+ "type": "comment"
+ },
+ "5706": {
+ "file_id": 460,
+ "content": " detected_scores: A float numpy array of size [num_boxes].\n detected_class_labels: A int32 numpy array of size [num_boxes].\n detected_masks: A uint8 numpy array of size\n [num_boxes, height, width].\n Returns:\n valid_detected_boxes: A float numpy array of size\n [num_valid_boxes, 4] containing box coordinates in\n [ymin, xmin, ymax, xmax] format.\n valid_detected_scores: A float numpy array of size\n [num_valid_boxes].\n valid_detected_class_labels: A int32 numpy array of size\n [num_valid_boxes].\n valid_detected_masks: A uint8 numpy array of size\n [num_valid_boxes, height, width].\n \"\"\"\n valid_indices = np.logical_and(\n detected_boxes[:, 0] < detected_boxes[:, 2],\n detected_boxes[:, 1] < detected_boxes[:, 3],\n )\n detected_boxes = detected_boxes[valid_indices]\n detected_scores = detected_scores[valid_indices]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:422-443"
+ },
+ "5707": {
+ "file_id": 460,
+ "content": "This function performs a filtering operation on the input arrays (detected_boxes, detected_scores, detected_class_labels, and detected_masks). It keeps only those elements where the first element of the detected_box is less than its fourth element, and the second element of detected_box is less than its third element. The resulting valid indices are used to slice the input arrays into their valid subsets (valid_detected_boxes, valid_detected_scores, valid_detected_class_labels, and valid_detected_masks).",
+ "type": "comment"
+ },
+ "5708": {
+ "file_id": 460,
+ "content": " detected_class_labels = detected_class_labels[valid_indices]\n if detected_masks is not None:\n detected_masks = detected_masks[valid_indices]\n return [\n detected_boxes,\n detected_scores,\n detected_class_labels,\n detected_masks,\n ]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/per_image_evaluation.py:444-452"
+ },
+ "5709": {
+ "file_id": 460,
+ "content": "Function returns detected bounding boxes, scores, class labels, and masks (if available) for valid indices only.",
+ "type": "comment"
+ },
+ "5710": {
+ "file_id": 461,
+ "content": "/paddlevideo/metrics/ava_evaluation/standard_fields.py",
+ "type": "filepath"
+ },
+ "5711": {
+ "file_id": 461,
+ "content": "This code sets naming conventions for object detection, defines fields and variables for efficient communication between decoder and model, and improves dataset evaluation. It also establishes conventions for video object detector output storage and standard metrics for field evaluation.",
+ "type": "summary"
+ },
+ "5712": {
+ "file_id": 461,
+ "content": "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# =============================================================================\n\"\"\"Contains classes specifying naming conventions used for object detection.\nSpecifies:\n InputDataFields: standard fields used by reader/preprocessor/batcher.\n DetectionResultFields: standard fields returned by object detector.\n\"\"\"\nclass InputDataFields:\n \"\"\"Names for the input tensors.\n Holds the standard data field names to use for identifying input tensors.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/standard_fields.py:1-26"
+ },
+ "5713": {
+ "file_id": 461,
+ "content": "This code is defining classes for standard naming conventions in object detection. It provides InputDataFields for input tensors and DetectionResultFields for results returned by the object detector.",
+ "type": "comment"
+ },
+ "5714": {
+ "file_id": 461,
+ "content": " This should be used by the decoder to identify keys for the returned\n tensor_dict containing input tensors. And it should be used by the model to\n identify the tensors it needs.\n Attributes:\n image: image.\n original_image: image in the original input size.\n key: unique key corresponding to image.\n source_id: source of the original image.\n filename: original filename of the dataset (without common path).\n groundtruth_image_classes: image-level class labels.\n groundtruth_boxes: coordinates of the ground truth boxes in the image.\n groundtruth_classes: box-level class labels.\n groundtruth_label_types: box-level label types (e.g. explicit\n negative).\n groundtruth_is_crowd: [DEPRECATED, use groundtruth_group_of instead]\n is the groundtruth a single object or a crowd.\n groundtruth_area: area of a groundtruth segment.\n groundtruth_difficult: is a `difficult` object\n groundtruth_group_of: is a `group_of` objects, e.g. multiple objects of",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/standard_fields.py:27-46"
+ },
+ "5715": {
+ "file_id": 461,
+ "content": "This code defines standard fields used by the decoder and model for identifying keys in returned tensor_dict. Fields include image, original_image, source_id, filename, groundtruth_image_classes, groundtruth_boxes, groundtruth_classes, groundtruth_label_types, groundtruth_is_crowd, groundtruth_area, and groundtruth_difficult. It is used by the decoder to identify keys for returned tensor_dict and by model to identify necessary tensors.",
+ "type": "comment"
+ },
+ "5716": {
+ "file_id": 461,
+ "content": " the same class, forming a connected group, where instances are\n heavily occluding each other.\n proposal_boxes: coordinates of object proposal boxes.\n proposal_objectness: objectness score of each proposal.\n groundtruth_instance_masks: ground truth instance masks.\n groundtruth_instance_boundaries: ground truth instance boundaries.\n groundtruth_instance_classes: instance mask-level class labels.\n groundtruth_keypoints: ground truth keypoints.\n groundtruth_keypoint_visibilities: ground truth keypoint visibilities.\n groundtruth_label_scores: groundtruth label scores.\n groundtruth_weights: groundtruth weight factor for bounding boxes.\n num_groundtruth_boxes: number of groundtruth boxes.\n true_image_shapes: true shapes of images in the resized images, as\n resized images can be padded with zeros.\n \"\"\"\n image = 'image'\n original_image = 'original_image'\n key = 'key'\n source_id = 'source_id'",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/standard_fields.py:47-66"
+ },
+ "5717": {
+ "file_id": 461,
+ "content": "This code defines a dictionary of variables used for AVA evaluation, including image and original image keys, source IDs, and other metrics such as proposal boxes, ground truth instance masks, and more. These variables are necessary for accurately evaluating the performance of video object detection models.",
+ "type": "comment"
+ },
+ "5718": {
+ "file_id": 461,
+ "content": " filename = 'filename'\n groundtruth_image_classes = 'groundtruth_image_classes'\n groundtruth_boxes = 'groundtruth_boxes'\n groundtruth_classes = 'groundtruth_classes'\n groundtruth_label_types = 'groundtruth_label_types'\n groundtruth_is_crowd = 'groundtruth_is_crowd'\n groundtruth_area = 'groundtruth_area'\n groundtruth_difficult = 'groundtruth_difficult'\n groundtruth_group_of = 'groundtruth_group_of'\n proposal_boxes = 'proposal_boxes'\n proposal_objectness = 'proposal_objectness'\n groundtruth_instance_masks = 'groundtruth_instance_masks'\n groundtruth_instance_boundaries = 'groundtruth_instance_boundaries'\n groundtruth_instance_classes = 'groundtruth_instance_classes'\n groundtruth_keypoints = 'groundtruth_keypoints'\n groundtruth_keypoint_visibilities = 'groundtruth_keypoint_visibilities'\n groundtruth_label_scores = 'groundtruth_label_scores'\n groundtruth_weights = 'groundtruth_weights'\n num_groundtruth_boxes = 'num_groundtruth_boxes'\n true_image_shape = 'true_image_shape'",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/standard_fields.py:67-86"
+ },
+ "5719": {
+ "file_id": 461,
+ "content": "This code defines various field names for the AWA dataset, which includes attributes like ground truth image classes, bounding boxes, class labels, label types, object crowding status, and more. The fields cover aspects such as instance masks, boundaries, keypoints, visibilities, label scores, and weights. These field definitions are likely used to organize and manage data in the dataset for further processing or evaluation tasks.",
+ "type": "comment"
+ },
+ "5720": {
+ "file_id": 461,
+ "content": "class DetectionResultFields:\n \"\"\"Naming conventions for storing the output of the detector.\n Attributes:\n source_id: source of the original image.\n key: unique key corresponding to image.\n detection_boxes: coordinates of the detection boxes in the image.\n detection_scores: detection scores for the detection boxes in the\n image.\n detection_classes: detection-level class labels.\n detection_masks: contains a segmentation mask for each detection box.\n detection_boundaries: contains an object boundary for each detection\n box.\n detection_keypoints: contains detection keypoints for each detection\n box.\n num_detections: number of detections in the batch.\n \"\"\"\n source_id = 'source_id'\n key = 'key'\n detection_boxes = 'detection_boxes'\n detection_scores = 'detection_scores'\n detection_classes = 'detection_classes'\n detection_masks = 'detection_masks'\n detection_boundaries = 'detection_boundaries'",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/standard_fields.py:89-113"
+ },
+ "5721": {
+ "file_id": 461,
+ "content": "This class defines the standard naming conventions for storing the output of a video object detector. It includes attributes like source_id, key, detection boxes coordinates, scores, classes, masks, boundaries, keypoints, and number of detections in a batch.",
+ "type": "comment"
+ },
+ "5722": {
+ "file_id": 461,
+ "content": " detection_keypoints = 'detection_keypoints'\n num_detections = 'num_detections'",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_evaluation/standard_fields.py:114-115"
+ },
+ "5723": {
+ "file_id": 461,
+ "content": "These two variables, detection_keypoints and num_detections, represent metrics for storing the keypoints of detected objects and the number of detections respectively in the standard fields evaluation.",
+ "type": "comment"
+ },
+ "5724": {
+ "file_id": 462,
+ "content": "/paddlevideo/metrics/ava_metric.py",
+ "type": "filepath"
+ },
+ "5725": {
+ "file_id": 462,
+ "content": "The code imports libraries, defines AVAMetric class for PaddleVideo and prepares metrics for video object detection. It also includes methods for logging during iterations, setting dataset info, and calculating final results.",
+ "type": "summary"
+ },
+ "5726": {
+ "file_id": 462,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nfrom collections import OrderedDict\nfrom paddlevideo.utils import get_logger, load, log_batch, AverageMeter\nfrom .registry import METRIC\nfrom .base import BaseMetric\nimport time\nfrom datetime import datetime\nfrom .ava_utils import ava_evaluate_results\nlogger = get_logger(\"paddlevideo\")\n\"\"\" An example for metrics class.\n MultiCropMetric for slowfast.\n\"\"\"\n@METRIC.register\nclass AVAMetric(BaseMetric):\n def __init__(self,\n data_size,\n batch_size,\n file_path,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_metric.py:1-34"
+ },
+ "5727": {
+ "file_id": 462,
+ "content": "This code imports necessary libraries and registers a class called AVAMetric as a metric for PaddleVideo. It initializes the AVAMetric with specified data size, batch size, and file path. The class inherits from BaseMetric and has an __init__ method which sets instance variables for data_size, batch_size, file_path, result_filename, and other properties.",
+ "type": "comment"
+ },
+ "5728": {
+ "file_id": 462,
+ "content": " exclude_file,\n label_file,\n custom_classes,\n log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.file_path = file_path\n self.exclude_file = exclude_file\n self.label_file = label_file\n self.custom_classes = custom_classes\n self.results = []\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"recall@thr=0.5\", AverageMeter(\"recall@thr=0.5\", '.5f')),\n (\"prec@thr=0.5\", AverageMeter(\"prec@thr=0.5\", '.5f')),\n (\"recall@top3\", AverageMeter(\"recall@top3\", '.5f')),\n (\"prec@top3\", AverageMeter(\"prec@top3\", '.5f')),\n (\"recall@top5\", AverageMeter(\"recall@top5\", '.5f')),\n (\"prec@top5\", AverageMeter(\"prec@top5\", '.5f')),\n (\"mAP@0.5IOU\", AverageMeter(\"mAP@0.5IOU\", '.5f')),\n (\"batch_time\", AverageMeter('batch_cost', '.5f')),\n (\"reader_time\", AverageMeter('reader_cost', '.5f')),",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_metric.py:35-60"
+ },
+ "5729": {
+ "file_id": 462,
+ "content": "The code initializes a class for preparing metrics in video object detection. It takes various parameters like file path, exclude file, label file, custom classes, and log interval for initialization. The class uses AverageMeter to store metrics such as loss, recall@thr=0.5, prec@thr=0.5, recall@top3, prec@top3, recall@top5, prec@top5, mAP@0.5IOU, batch time, and reader time.",
+ "type": "comment"
+ },
+ "5730": {
+ "file_id": 462,
+ "content": " ]\n self.record_list = OrderedDict(record_list)\n self.tic = time.time()\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n self.results.extend(outputs)\n self.record_list['batch_time'].update(time.time() - self.tic)\n tic = time.time()\n ips = \"ips: {:.5f} instance/sec.\".format(\n self.batch_size / self.record_list[\"batch_time\"].val)\n log_batch(self.record_list, batch_id, 0, 0, \"test\", ips)\n def set_dataset_info(self, info, dataset_len):\n self.info = info\n self.dataset_len = dataset_len\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n test_res = ava_evaluate_results(self.info, self.dataset_len,\n self.results, None, self.label_file,\n self.file_path, self.exclude_file)\n for name, value in test_res.items():\n self.record_list[name].update(value, self.batch_size)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_metric.py:61-90"
+ },
+ "5731": {
+ "file_id": 462,
+ "content": "This code defines a class for metrics calculation and logging, with methods for updating metrics during iterations, setting dataset information, and accumulating final results. The update method extends the results list, updates batch time, logs batch time, and logs instance per second (ips). The accumulate method calculates final test results using ava_evaluate_results function and updates the record list with the final values.",
+ "type": "comment"
+ },
+ "5732": {
+ "file_id": 462,
+ "content": " return self.record_list",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_metric.py:92-92"
+ },
+ "5733": {
+ "file_id": 462,
+ "content": "The code snippet is returning the record list from a class method. It seems that this method might have been responsible for recording or storing some data in the `record_list` attribute of the class instance, and now it's returning that data.",
+ "type": "comment"
+ },
+ "5734": {
+ "file_id": 463,
+ "content": "/paddlevideo/metrics/ava_utils.py",
+ "type": "filepath"
+ },
+ "5735": {
+ "file_id": 463,
+ "content": "This code handles video sequence object detection, converting results to CSV format and evaluating AVA metrics using error handling, utility functions, and GPU-based processing.",
+ "type": "summary"
+ },
+ "5736": {
+ "file_id": 463,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport csv\nimport heapq\nimport logging\nimport time\nfrom collections import defaultdict\nfrom .ava_evaluation import object_detection_evaluation as det_eval\nfrom .ava_evaluation import standard_fields\nfrom .recall import eval_recalls\nimport shutil\nimport pickle\nimport time\nimport os\nimport os.path as osp\nfrom paddlevideo.utils import get_logger, get_dist_info\nimport paddle.distributed as dist\nimport sys\nimport numpy as np",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:1-31"
+ },
+ "5737": {
+ "file_id": 463,
+ "content": "This code imports necessary libraries and modules for evaluating AVA (Activity-driven Visual Attention) metrics. It also includes a license notice, time management functions, and error handling measures. The code uses defaultdict from collections and eval_recalls function from the same repository to perform evaluation tasks related to object detection in video sequences. Additionally, it incorporates paddlevideo's get_logger() function for logging, dist library for distributed processing, and numpy for numerical operations.",
+ "type": "comment"
+ },
+ "5738": {
+ "file_id": 463,
+ "content": "from pathlib import Path\nfrom datetime import datetime\nimport paddle\ndef det2csv(info, dataset_len, results, custom_classes):\n csv_results = []\n for idx in range(dataset_len):\n video_id = info[idx]['video_id']\n timestamp = info[idx]['timestamp']\n result = results[idx]\n for label, _ in enumerate(result):\n for bbox in result[label]:\n if type(bbox) == paddle.Tensor:\n bbox = bbox.numpy()\n bbox_ = tuple(bbox.tolist())\n if custom_classes is not None:\n actual_label = custom_classes[label + 1]\n else:\n actual_label = label + 1\n csv_results.append((\n video_id,\n timestamp,\n ) + bbox_[:4] + (actual_label, ) + bbox_[4:])\n return csv_results\n# results is organized by class\ndef results2csv(info, dataset_len, results, out_file, custom_classes=None):\n if isinstance(results[0], list):\n csv_results = det2csv(info, dataset_len, results, custom_classes)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:32-64"
+ },
+ "5739": {
+ "file_id": 463,
+ "content": "The code defines two functions: \"det2csv\" and \"results2csv\". \"det2csv\" takes in information, dataset length, results, and custom classes (if any), and returns a list of tuples representing the results in CSV format. It loops through each entry, extracts relevant data, converts tensors to numpy arrays if needed, and appends the information to the csv_results list. \"results2csv\" checks if the results are organized by class or not, then calls either \"det2csv\" or performs CSV conversion directly using it.",
+ "type": "comment"
+ },
+ "5740": {
+ "file_id": 463,
+ "content": " # save space for float\n def tostr(item):\n if isinstance(item, float):\n return f'{item:.3f}'\n return str(item)\n with open(out_file, 'w') as f:\n for csv_result in csv_results:\n f.write(','.join(map(lambda x: tostr(x), csv_result)))\n f.write('\\n')\ndef print_time(message, start):\n print('==> %g seconds to %s' % (time.time() - start, message))\ndef make_image_key(video_id, timestamp):\n \"\"\"Returns a unique identifier for a video id & timestamp.\"\"\"\n return f'{video_id},{int(timestamp):04d}'\ndef read_csv(csv_file, class_whitelist=None, capacity=0):\n \"\"\"Loads boxes and class labels from a CSV file in the AVA format.\n CSV file format described at https://research.google.com/ava/download.html.\n Args:\n csv_file: A file object.\n class_whitelist: If provided, boxes corresponding to (integer) class\n labels not in this set are skipped.\n capacity: Maximum number of labeled boxes allowed for each example.\n Default is 0 where there is no limit.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:66-97"
+ },
+ "5741": {
+ "file_id": 463,
+ "content": "This code snippet contains several utility functions used for video analysis. The \"tostr\" function converts a float to a string representation with 3 decimal places, while the \"print_time\" function calculates and prints the time elapsed since a given start point. The \"make_image_key\" function generates a unique identifier for a video ID and timestamp, and \"read_csv\" function loads boxes and class labels from a CSV file in AVA format.",
+ "type": "comment"
+ },
+ "5742": {
+ "file_id": 463,
+ "content": " Returns:\n boxes: A dictionary mapping each unique image key (string) to a list of\n boxes, given as coordinates [y1, x1, y2, x2].\n labels: A dictionary mapping each unique image key (string) to a list\n of integer class lables, matching the corresponding box in `boxes`.\n scores: A dictionary mapping each unique image key (string) to a list\n of score values lables, matching the corresponding label in `labels`.\n If scores are not provided in the csv, then they will default to 1.0.\n \"\"\"\n start = time.time()\n entries = defaultdict(list)\n boxes = defaultdict(list)\n labels = defaultdict(list)\n scores = defaultdict(list)\n reader = csv.reader(csv_file)\n for row in reader:\n assert len(row) in [7, 8], 'Wrong number of columns: ' + row\n image_key = make_image_key(row[0], row[1])\n x1, y1, x2, y2 = [float(n) for n in row[2:6]]\n action_id = int(row[6])\n if class_whitelist and action_id not in class_whitelist:\n continue",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:99-120"
+ },
+ "5743": {
+ "file_id": 463,
+ "content": "This code reads a CSV file with video frame data, and for each row, it creates dictionaries for boxes, labels, and scores. It checks the class whitelist before adding the data to the respective lists. If scores are not provided in the CSV, they default to 1.0. The time taken for this process is measured at the beginning with start = time.time().",
+ "type": "comment"
+ },
+ "5744": {
+ "file_id": 463,
+ "content": " score = 1.0\n if len(row) == 8:\n score = float(row[7])\n if capacity < 1 or len(entries[image_key]) < capacity:\n heapq.heappush(entries[image_key],\n (score, action_id, y1, x1, y2, x2))\n elif score > entries[image_key][0][0]:\n heapq.heapreplace(entries[image_key],\n (score, action_id, y1, x1, y2, x2))\n for image_key in entries:\n # Evaluation API assumes boxes with descending scores\n entry = sorted(entries[image_key], key=lambda tup: -tup[0])\n for item in entry:\n score, action_id, y1, x1, y2, x2 = item\n boxes[image_key].append([y1, x1, y2, x2])\n labels[image_key].append(action_id)\n scores[image_key].append(score)\n print_time('read file ' + csv_file.name, start)\n return boxes, labels, scores\ndef read_exclusions(exclusions_file):\n \"\"\"Reads a CSV file of excluded timestamps.\n Args:\n exclusions_file: A file object containing a csv of video-id,timestamp.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:122-147"
+ },
+ "5745": {
+ "file_id": 463,
+ "content": "This code reads a CSV file containing object detection results and stores them in three lists: boxes, labels, and scores. The code also handles exclusions by reading a separate CSV file that contains excluded timestamps. The score is determined based on the length of each row in the CSV file and added to the corresponding image key's entry in the entries dictionary if the capacity allows or if the score is higher than the current highest score for that image key. The code then sorts the entries by descending scores and appends them to the boxes, labels, and scores lists for each image key.",
+ "type": "comment"
+ },
+ "5746": {
+ "file_id": 463,
+ "content": " Returns:\n A set of strings containing excluded image keys, e.g.\n \"aaaaaaaaaaa,0904\",\n or an empty set if exclusions file is None.\n \"\"\"\n excluded = set()\n if exclusions_file:\n reader = csv.reader(exclusions_file)\n for row in reader:\n assert len(row) == 2, 'Expected only 2 columns, got: ' + row\n excluded.add(make_image_key(row[0], row[1]))\n return excluded\ndef read_labelmap(labelmap_file):\n \"\"\"Reads a labelmap without the dependency on protocol buffers.\n Args:\n labelmap_file: A file object containing a label map protocol buffer.\n Returns:\n labelmap: The label map in the form used by the\n object_detection_evaluation\n module - a list of {\"id\": integer, \"name\": classname } dicts.\n class_ids: A set containing all of the valid class id integers.\n \"\"\"\n labelmap = []\n class_ids = set()\n name = ''\n class_id = ''\n for line in labelmap_file:\n if line.startswith(' name:'):\n name = line.split('\"')[1]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:149-181"
+ },
+ "5747": {
+ "file_id": 463,
+ "content": "Function `read_excluded_images` reads an exclusions file and returns a set of image keys to exclude. The input file is read row by row, and for each row the function checks that there are exactly two columns and adds the image key (combination of column 1 and column 2) to the excluded set.\n\nFunction `read_labelmap` reads a labelmap file without using protocol buffers. It iterates over the file line by line. When it encounters a line starting with 'name:', it extracts the class name, and when it encounters a line starting with 'id:' it extracts the class id. The function then appends a dictionary containing the id and name to the labelmap list and adds the id to the set of valid class ids.",
+ "type": "comment"
+ },
+ "5748": {
+ "file_id": 463,
+ "content": " elif line.startswith(' id:') or line.startswith(' label_id:'):\n class_id = int(line.strip().split(' ')[-1])\n labelmap.append({'id': class_id, 'name': name})\n class_ids.add(class_id)\n return labelmap, class_ids\n# Seems there is at most 100 detections for each image\ndef ava_eval(result_file,\n result_type,\n label_file,\n ann_file,\n exclude_file,\n max_dets=(100, ),\n verbose=True,\n custom_classes=None):\n assert result_type in ['mAP']\n start = time.time()\n categories, class_whitelist = read_labelmap(open(label_file))\n if custom_classes is not None:\n custom_classes = custom_classes[1:]\n assert set(custom_classes).issubset(set(class_whitelist))\n class_whitelist = custom_classes\n categories = [cat for cat in categories if cat['id'] in custom_classes]\n # loading gt, do not need gt score\n gt_boxes, gt_labels, _ = read_csv(open(ann_file), class_whitelist, 0)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:182-210"
+ },
+ "5749": {
+ "file_id": 463,
+ "content": "This function ava_eval() takes several file paths as input and evaluates the results using mean average precision (mAP). It uses a label map to convert class labels from detections to their corresponding IDs. The code checks for 'id' or 'label_id' in each line of the label file, appends the ID and name to the label map, and adds the ID to a set of class_ids. The function also handles custom classes by excluding any category whose ID is not in the custom_classes list. The gt_boxes, gt_labels, and _ are loaded from the ann_file for evaluation.",
+ "type": "comment"
+ },
+ "5750": {
+ "file_id": 463,
+ "content": " if verbose:\n print_time('Reading detection results', start)\n if exclude_file is not None:\n excluded_keys = read_exclusions(open(exclude_file))\n else:\n excluded_keys = list()\n start = time.time()\n boxes, labels, scores = read_csv(open(result_file), class_whitelist, 0)\n if verbose:\n print_time('Reading detection results', start)\n if result_type == 'proposal':\n gts = [\n np.array(gt_boxes[image_key], dtype=float) for image_key in gt_boxes\n ]\n proposals = []\n for image_key in gt_boxes:\n if image_key in boxes:\n proposals.append(\n np.concatenate(\n (np.array(boxes[image_key], dtype=float),\n np.array(scores[image_key], dtype=float)[:, None]),\n axis=1))\n else:\n # if no corresponding proposal, add a fake one\n proposals.append(np.array([0, 0, 1, 1, 1]))\n # Proposals used here are with scores",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:211-240"
+ },
+ "5751": {
+ "file_id": 463,
+ "content": "The code reads detection results from a file, excludes certain keys if specified in an exclude file, and measures the time taken to read the results. It then checks if the result type is 'proposal' and creates proposals based on the gt_boxes for each image key present in boxes or adds a fake one if no corresponding proposal exists. Proposals include scores.",
+ "type": "comment"
+ },
+ "5752": {
+ "file_id": 463,
+ "content": " recalls = eval_recalls(gts, proposals, np.array(max_dets),\n np.arange(0.5, 0.96, 0.05))\n ar = recalls.mean(axis=1)\n ret = {}\n for i, num in enumerate(max_dets):\n print(f'Recall@0.5@{num}\\t={recalls[i, 0]:.4f}')\n print(f'AR@{num}\\t={ar[i]:.4f}')\n ret[f'Recall@0.5@{num}'] = recalls[i, 0]\n ret[f'AR@{num}'] = ar[i]\n return ret\n if result_type == 'mAP':\n pascal_evaluator = det_eval.PascalDetectionEvaluator(categories)\n start = time.time()\n for image_key in gt_boxes:\n if verbose and image_key in excluded_keys:\n logging.info(\n 'Found excluded timestamp in detections: %s.'\n 'It will be ignored.', image_key)\n continue\n pascal_evaluator.add_single_ground_truth_image_info(\n image_key, {\n standard_fields.InputDataFields.groundtruth_boxes:\n np.array(gt_boxes[image_key], dtype=float),",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:241-265"
+ },
+ "5753": {
+ "file_id": 463,
+ "content": "This code calculates the Average Recall (AR) and Recall@0.5 (R@0.5) for different detection numbers using the eval_recalls function. It then prints the results and stores them in a dictionary. If the result type is 'mAP', it initializes a PascalDetectionEvaluator, adds ground truth information for each image key, and calculates the mean average precision (mAP).",
+ "type": "comment"
+ },
+ "5754": {
+ "file_id": 463,
+ "content": " standard_fields.InputDataFields.groundtruth_classes:\n np.array(gt_labels[image_key], dtype=int),\n standard_fields.InputDataFields.groundtruth_difficult:\n np.zeros(len(gt_boxes[image_key]), dtype=bool)\n })\n if verbose:\n print_time('Convert groundtruth', start)\n start = time.time()\n for image_key in boxes:\n if verbose and image_key in excluded_keys:\n logging.info(\n 'Found excluded timestamp in detections: %s.'\n 'It will be ignored.', image_key)\n continue\n pascal_evaluator.add_single_detected_image_info(\n image_key, {\n standard_fields.DetectionResultFields.detection_boxes:\n np.array(boxes[image_key], dtype=float),\n standard_fields.DetectionResultFields.detection_classes:\n np.array(labels[image_key], dtype=int),",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:266-286"
+ },
+ "5755": {
+ "file_id": 463,
+ "content": "This code adds single detected image information to a Pascal evaluator. It converts groundtruth labels and boxes into appropriate data structures, handles excluded timestamps, and processes detection boxes and classes for evaluation.",
+ "type": "comment"
+ },
+ "5756": {
+ "file_id": 463,
+ "content": " standard_fields.DetectionResultFields.detection_scores:\n np.array(scores[image_key], dtype=float)\n })\n if verbose:\n print_time('convert detections', start)\n start = time.time()\n metrics = pascal_evaluator.evaluate()\n if verbose:\n print_time('run_evaluator', start)\n for display_name in metrics:\n print(f'{display_name}=\\t{metrics[display_name]}')\n ret = {\n display_name: metrics[display_name]\n for display_name in metrics if 'ByCategory' not in display_name\n }\n return ret\ndef mkdir_or_exist(dir_name, mode=0o777):\n if dir_name == '':\n return\n dir_name = osp.expanduser(dir_name)\n os.makedirs(dir_name, mode=mode, exist_ok=True)\ndef dump_to_fileobj(obj, file, **kwargs):\n kwargs.setdefault('protocol', 2)\n pickle.dump(obj, file, **kwargs)\ndef dump_to_path(obj, filepath, mode='wb'):\n with open(filepath, mode) as f:\n dump_to_fileobj(obj, f)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:287-320"
+ },
+ "5757": {
+ "file_id": 463,
+ "content": "Code snippet performs AVA evaluation for detection results, and prints or returns specific metrics. It also includes functions for creating directories, dumping objects to files using pickle library, and has a function for time measurement called print_time.",
+ "type": "comment"
+ },
+ "5758": {
+ "file_id": 463,
+ "content": "def load_from_fileobj(file, **kwargs):\n return pickle.load(file, **kwargs)\ndef load_from_path(filepath, mode='rb'):\n with open(filepath, mode) as f:\n return load_from_fileobj(f)\ndef collect_results_cpu(result_part, size):\n \"\"\"Collect results in cpu mode.\n It saves the results on different gpus to 'tmpdir' and collects\n them by the rank 0 worker.\n \"\"\"\n tmpdir = osp.join('./', 'collect_results_cpu')\n #1. load results of all parts from tmp dir\n mkdir_or_exist(tmpdir)\n rank, world_size = get_dist_info()\n dump_to_path(result_part, osp.join(tmpdir, f'part_{rank}.pkl'))\n dist.barrier()\n if rank != 0:\n return None\n #2. collect all parts\n while 1:\n all_exist = True\n for i in range(world_size):\n part_file = osp.join(tmpdir, f'part_{i}.pkl')\n if not Path(part_file).exists():\n all_exist = False\n if all_exist:\n break\n else:\n time.sleep(60)\n time.sleep(120)\n #3. load results of all parts from tmp dir",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:323-357"
+ },
+ "5759": {
+ "file_id": 463,
+ "content": "This code defines three functions: `load_from_fileobj`, `load_from_path`, and `collect_results_cpu`. The first two are used to load data from files or file paths, respectively. The third function, `collect_results_cpu`, is a CPU-based method for collecting results across multiple GPUs by saving them in a temporary directory ('tmpdir') and having the rank 0 worker collect them. It checks if all parts exist, waits if not, then loads and returns the collected results once they are all available.",
+ "type": "comment"
+ },
+ "5760": {
+ "file_id": 463,
+ "content": " part_list = []\n for i in range(world_size):\n part_file = osp.join(tmpdir, f'part_{i}.pkl')\n part_list.append(load_from_path(part_file))\n #4. sort the results\n ordered_results = []\n for res in zip(*part_list):\n ordered_results.extend(list(res))\n ordered_results = ordered_results[:\n size] #the dataloader may pad some samples\n #5. remove results of all parts from tmp dir, avoid dump_file fail to tmp dir when dir not exists.\n for i in range(world_size):\n part_file = osp.join(tmpdir, f'part_{i}.pkl')\n os.remove(part_file)\n return ordered_results\ndef ava_evaluate_results(info, dataset_len, results, custom_classes, label_file,\n file_path, exclude_file):\n # need to create a temp result file\n time_now = datetime.now().strftime('%Y%m%d_%H%M%S')\n temp_file = f'AVA_{time_now}_result.csv'\n results2csv(info, dataset_len, results, temp_file)\n ret = {}\n eval_result = ava_eval(\n temp_file,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:358-384"
+ },
+ "5761": {
+ "file_id": 463,
+ "content": "This code is used for evaluating AVA results by splitting the computation across multiple processes, then combining and ordering the partial results before deleting temporary files. It takes in information about the dataset, the evaluation results, custom class labels, and file paths for input and exclusion lists. The code creates a temporary result file, converts the results to a CSV format, performs AVA evaluation on the temporary file, and returns an evaluation result.",
+ "type": "comment"
+ },
+ "5762": {
+ "file_id": 463,
+ "content": " 'mAP',\n label_file,\n file_path, #ann_file,\n exclude_file,\n custom_classes=custom_classes)\n ret.update(eval_result)\n os.remove(temp_file)\n return ret",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ava_utils.py:385-394"
+ },
+ "5763": {
+ "file_id": 463,
+ "content": "This code is computing the mean average precision (mAP) for object detection metrics. It reads from a label file, file path, and excludes certain classes as specified. The results are stored in the 'ret' dictionary before removing a temporary file and returning the final results.",
+ "type": "comment"
+ },
+ "5764": {
+ "file_id": 464,
+ "content": "/paddlevideo/metrics/base.py",
+ "type": "filepath"
+ },
+ "5765": {
+ "file_id": 464,
+ "content": "This code defines a base class for metrics in the PaddleVideo library, initializing with data size, batch size, world size, and log interval. It includes all-gather and concatenation methods, along with abstract update and accumulate functions to be implemented by subclasses.",
+ "type": "summary"
+ },
+ "5766": {
+ "file_id": 464,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nimport paddle\nfrom paddlevideo.utils import get_dist_info\nfrom .registry import METRIC\nclass BaseMetric(object):\n def __init__(self, data_size, batch_size, log_interval=1, **kwargs):\n self.data_size = data_size\n self.batch_size = batch_size\n _, self.world_size = get_dist_info()\n self.log_interval = log_interval\n def gather_from_gpu(self,\n gather_object: paddle.Tensor,\n concat_axis=0) -> paddle.Tensor:\n \"\"\"gather Tensor from all gpus into a list and concatenate them on `concat_axis`.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/base.py:1-31"
+ },
+ "5767": {
+ "file_id": 464,
+ "content": "This code is part of the PaddleVideo library and defines a base class for metrics. It initializes the metric with data size, batch size, world size, and log interval. The gather_from_gpu method gathers Tensors from all GPUs into a list and concatenates them on a specified axis.",
+ "type": "comment"
+ },
+ "5768": {
+ "file_id": 464,
+ "content": " Args:\n gather_object (paddle.Tensor): gather object Tensor\n concat_axis (int, optional): axis for concatenation. Defaults to 0.\n Returns:\n paddle.Tensor: gatherd & concatenated Tensor\n \"\"\"\n gather_object_list = []\n paddle.distributed.all_gather(gather_object_list, gather_object.cuda())\n return paddle.concat(gather_object_list, axis=concat_axis)\n @abstractmethod\n def update(self):\n raise NotImplementedError(\n \"'update' method must be implemented in subclass\")\n @abstractmethod\n def accumulate(self):\n raise NotImplementedError(\n \"'accumulate' method must be implemented in subclass\")",
+ "type": "code",
+ "location": "/paddlevideo/metrics/base.py:33-52"
+ },
+ "5769": {
+ "file_id": 464,
+ "content": "Function that performs all-gather and concatenation on the gather object Tensor. Abstract methods for update and accumulate that must be implemented in subclasses.",
+ "type": "comment"
+ },
+ "5770": {
+ "file_id": 465,
+ "content": "/paddlevideo/metrics/bmn_metric.py",
+ "type": "filepath"
+ },
+ "5771": {
+ "file_id": 465,
+ "content": "This code calculates BMN metric for object detection in computer vision frameworks, supports batch_size and world_size as 1, initializes class variables, processes video data, logs progress, saves results, performs soft NMS, calculates proposal lists, evaluates performance using the \"cal_metrics\" function.",
+ "type": "summary"
+ },
+ "5772": {
+ "file_id": 465,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nimport multiprocessing as mp\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom .ActivityNet import ANETproposal\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ndef iou_with_anchors(anchors_min, anchors_max, box_min, box_max):\n \"\"\"Compute jaccard score between a box and the anchors.\n \"\"\"\n len_anchors = anchors_max - anchors_min\n int_xmin = np.maximum(anchors_min, box_min)\n int_xmax = np.minimum(anchors_max, box_max)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:1-32"
+ },
+ "5773": {
+ "file_id": 465,
+ "content": "This code imports necessary libraries and defines a function to compute the Intersection over Union (IoU) between a box and anchors. It appears to be related to object detection or proposal generation within a computer vision framework.",
+ "type": "comment"
+ },
+ "5774": {
+ "file_id": 465,
+ "content": " inter_len = np.maximum(int_xmax - int_xmin, 0.)\n union_len = len_anchors - inter_len + box_max - box_min\n jaccard = np.divide(inter_len, union_len)\n return jaccard\ndef boundary_choose(score_list):\n \"\"\"Choose start and end boundary from score.\n \"\"\"\n max_score = max(score_list)\n mask_high = (score_list > max_score * 0.5)\n score_list = list(score_list)\n score_middle = np.array([0.0] + score_list + [0.0])\n score_front = np.array([0.0, 0.0] + score_list)\n score_back = np.array(score_list + [0.0, 0.0])\n mask_peak = ((score_middle > score_front) & (score_middle > score_back))\n mask_peak = mask_peak[1:-1]\n mask = (mask_high | mask_peak).astype('float32')\n return mask\ndef soft_nms(df, alpha, t1, t2):\n '''\n df: proposals generated by network;\n alpha: alpha value of Gaussian decaying function;\n t1, t2: threshold for soft nms.\n '''\n df = df.sort_values(by=\"score\", ascending=False)\n tstart = list(df.xmin.values[:])\n tend = list(df.xmax.values[:])\n tscore = list(df.score.values[:])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:33-63"
+ },
+ "5775": {
+ "file_id": 465,
+ "content": "inter_len calculates the intersection length between two bounding boxes. union_len computes the total length of both boxes and jaccard index is calculated by dividing inter_len with union_len. This function returns the Jaccard index.\nThe boundary_choose() function selects start and end boundaries based on a given score list. It identifies the highest score and creates three arrays - score_list, score_front, score_back for comparison. mask_peak is created by comparing these arrays, followed by generating a binary mask of True values.\nSoft_nms function sorts proposals generated by network based on scores in descending order. It takes alpha value (Gaussian decaying function), and two threshold values t1, t2.",
+ "type": "comment"
+ },
+ "5776": {
+ "file_id": 465,
+ "content": " rstart = []\n rend = []\n rscore = []\n while len(tscore) > 1 and len(rscore) < 101:\n max_index = tscore.index(max(tscore))\n tmp_iou_list = iou_with_anchors(np.array(tstart), np.array(tend),\n tstart[max_index], tend[max_index])\n for idx in range(0, len(tscore)):\n if idx != max_index:\n tmp_iou = tmp_iou_list[idx]\n tmp_width = tend[max_index] - tstart[max_index]\n if tmp_iou > t1 + (t2 - t1) * tmp_width:\n tscore[idx] = tscore[idx] * np.exp(\n -np.square(tmp_iou) / alpha)\n rstart.append(tstart[max_index])\n rend.append(tend[max_index])\n rscore.append(tscore[max_index])\n tstart.pop(max_index)\n tend.pop(max_index)\n tscore.pop(max_index)\n newDf = pd.DataFrame()\n newDf['score'] = rscore\n newDf['xmin'] = rstart\n newDf['xmax'] = rend\n return newDf\n@METRIC.register\nclass BMNMetric(BaseMetric):\n \"\"\"\n Metrics for BMN. Two Stages in this metric:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:65-98"
+ },
+ "5777": {
+ "file_id": 465,
+ "content": "The code calculates BMN metric for object detection by iterating through a list of scores and appending the maximum score, along with its corresponding start and end positions, to new lists. It then creates a new DataFrame using these lists before returning it as the final output.",
+ "type": "comment"
+ },
+ "5778": {
+ "file_id": 465,
+ "content": " (1) Get test results using trained model, results will be saved in BMNMetric.result_path;\n (2) Calculate metrics using results file from stage (1).\n \"\"\"\n def __init__(self,\n data_size,\n batch_size,\n tscale,\n dscale,\n file_path,\n ground_truth_filename,\n subset,\n output_path,\n result_path,\n get_metrics=True,\n log_interval=1):\n \"\"\"\n Init for BMN metrics.\n Params:\n get_metrics: whether to calculate AR@N and AUC metrics or not, default True.\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n assert self.batch_size == 1, \" Now we just support batch_size==1 test\"\n assert self.world_size == 1, \" Now we just support single-card test\"\n self.tscale = tscale\n self.dscale = dscale\n self.file_path = file_path\n self.ground_truth_filename = ground_truth_filename",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:99-127"
+ },
+ "5779": {
+ "file_id": 465,
+ "content": "This code initializes an instance of BMNMetric class with various parameters such as data_size, batch_size, tscale, dscale, file_path, ground_truth_filename, subset, output_path, result_path, get_metrics, and log_interval. It also performs assertions to ensure batch_size is 1 and world_size is 1, as the code currently supports only these conditions. The class is a part of PaddleVideo library for video analysis tasks.",
+ "type": "comment"
+ },
+ "5780": {
+ "file_id": 465,
+ "content": " self.subset = subset\n self.output_path = output_path\n self.result_path = result_path\n self.get_metrics = get_metrics\n if not os.path.isdir(self.output_path):\n os.makedirs(self.output_path)\n if not os.path.isdir(self.result_path):\n os.makedirs(self.result_path)\n self.video_dict, self.video_list = self.get_dataset_dict(\n self.file_path, self.subset)\n def get_dataset_dict(self, file_path, subset):\n annos = json.load(open(file_path))\n video_dict = {}\n for video_name in annos.keys():\n video_subset = annos[video_name][\"subset\"]\n if subset in video_subset:\n video_dict[video_name] = annos[video_name]\n video_list = list(video_dict.keys())\n video_list.sort()\n return video_dict, video_list\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n fid = data[4].numpy()\n pred_bm, pred_start, pred_end = outputs",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:128-156"
+ },
+ "5781": {
+ "file_id": 465,
+ "content": "The code initializes the class variables and checks if the output and result directories exist, creating them if not. It then calls a method to get the dataset dictionary and list based on the provided file path and subset. The update method takes batch ID, data, and outputs as inputs to update metrics during each iteration.",
+ "type": "comment"
+ },
+ "5782": {
+ "file_id": 465,
+ "content": " pred_bm = pred_bm.numpy()\n pred_start = pred_start[0].numpy()\n pred_end = pred_end[0].numpy()\n snippet_xmins = [1.0 / self.tscale * i for i in range(self.tscale)]\n snippet_xmaxs = [\n 1.0 / self.tscale * i for i in range(1, self.tscale + 1)\n ]\n cols = [\"xmin\", \"xmax\", \"score\"]\n video_name = self.video_list[fid[0]]\n pred_bm = pred_bm[0, 0, :, :] * pred_bm[0, 1, :, :]\n start_mask = boundary_choose(pred_start)\n start_mask[0] = 1.\n end_mask = boundary_choose(pred_end)\n end_mask[-1] = 1.\n score_vector_list = []\n for idx in range(self.dscale):\n for jdx in range(self.tscale):\n start_index = jdx\n end_index = start_index + idx\n if end_index < self.tscale and start_mask[\n start_index] == 1 and end_mask[end_index] == 1:\n xmin = snippet_xmins[start_index]\n xmax = snippet_xmaxs[end_index]\n xmin_score = pred_start[start_index]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:157-182"
+ },
+ "5783": {
+ "file_id": 465,
+ "content": "Code snippet performs boundary detection and creates a score vector list for each detection. It uses the provided prediction, start and end values to calculate the xmin and xmax values within the defined time scale. Then it checks if the start and end mask conditions are met and adds the corresponding score value to the score vector list. This information is used for further analysis or evaluation of video frames.",
+ "type": "comment"
+ },
+ "5784": {
+ "file_id": 465,
+ "content": " xmax_score = pred_end[end_index]\n bm_score = pred_bm[idx, jdx]\n conf_score = xmin_score * xmax_score * bm_score\n score_vector_list.append([xmin, xmax, conf_score])\n score_vector_list = np.stack(score_vector_list)\n video_df = pd.DataFrame(score_vector_list, columns=cols)\n video_df.to_csv(os.path.join(self.output_path, \"%s.csv\" % video_name),\n index=False)\n if batch_id % self.log_interval == 0:\n logger.info(\"Processing................ batch {}\".format(batch_id))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n # check clip index of each video\n #Stage1\n self.bmn_post_processing(self.video_dict, self.subset, self.output_path,\n self.result_path)\n if self.get_metrics:\n logger.info(\"[TEST] calculate metrics...\")\n #Stage2\n uniform_average_nr_proposals_valid, uniform_average_recall_valid, uniform_recall_valid = self.cal_metrics(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:183-206"
+ },
+ "5785": {
+ "file_id": 465,
+ "content": "This code snippet performs post-processing on video data and calculates metrics. It first processes the video data, then accumulates the metrics for each batch during processing. The code uses numpy arrays to handle score vectors, Pandas DataFrame to store and manipulate data, and logging to provide progress updates. The results are saved in a CSV file at the specified output path.",
+ "type": "comment"
+ },
+ "5786": {
+ "file_id": 465,
+ "content": " self.ground_truth_filename,\n os.path.join(self.result_path, \"bmn_results_validation.json\"),\n max_avg_nr_proposals=100,\n tiou_thresholds=np.linspace(0.5, 0.95, 10),\n subset='validation')\n logger.info(\"AR@1; AR@5; AR@10; AR@100\")\n logger.info(\"%.02f %.02f %.02f %.02f\" %\n (100 * np.mean(uniform_recall_valid[:, 0]),\n 100 * np.mean(uniform_recall_valid[:, 4]),\n 100 * np.mean(uniform_recall_valid[:, 9]),\n 100 * np.mean(uniform_recall_valid[:, -1])))\n def bmn_post_processing(self, video_dict, subset, output_path, result_path):\n video_list = list(video_dict.keys())\n global result_dict\n result_dict = mp.Manager().dict()\n pp_num = 12\n num_videos = len(video_list)\n num_videos_per_thread = int(num_videos / pp_num)\n processes = []\n for tid in range(pp_num - 1):\n tmp_video_list = video_list[tid * num_videos_per_thread:(tid + 1) *",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:207-229"
+ },
+ "5787": {
+ "file_id": 465,
+ "content": "This code is initializing a bmn_post_processing function that will process multiple videos in parallel using multiple processes. It creates a result dictionary and divides the video list into equal parts to assign each part to a separate process. It also logs the average recall at different thresholds for different numbers of detections.",
+ "type": "comment"
+ },
+ "5788": {
+ "file_id": 465,
+ "content": " num_videos_per_thread]\n p = mp.Process(target=self.video_process,\n args=(tmp_video_list, video_dict, output_path,\n result_dict))\n p.start()\n processes.append(p)\n tmp_video_list = video_list[(pp_num - 1) * num_videos_per_thread:]\n p = mp.Process(target=self.video_process,\n args=(tmp_video_list, video_dict, output_path,\n result_dict))\n p.start()\n processes.append(p)\n for p in processes:\n p.join()\n result_dict = dict(result_dict)\n output_dict = {\n \"version\": \"VERSION 1.3\",\n \"results\": result_dict,\n \"external_data\": {}\n }\n outfile = open(\n os.path.join(result_path, \"bmn_results_%s.json\" % subset), \"w\")\n # json.dump(output_dict, outfile)\n # in case of file name in chinese\n json.dump(output_dict, outfile, ensure_ascii=False)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:230-256"
+ },
+ "5789": {
+ "file_id": 465,
+ "content": "The code creates multiple processes to handle video processing tasks in parallel, using multiprocessing. It then joins all the results together into a single output dictionary before writing it to a JSON file. This approach allows for efficient and concurrent processing of large numbers of videos.",
+ "type": "comment"
+ },
+ "5790": {
+ "file_id": 465,
+ "content": " outfile.close()\n def video_process(self,\n video_list,\n video_dict,\n output_path,\n result_dict,\n snms_alpha=0.4,\n snms_t1=0.55,\n snms_t2=0.9):\n for video_name in video_list:\n logger.info(\"Processing video........\" + video_name)\n df = pd.read_csv(os.path.join(output_path, video_name + \".csv\"))\n if len(df) > 1:\n df = soft_nms(df, snms_alpha, snms_t1, snms_t2)\n video_duration = video_dict[video_name][\"duration_second\"]\n proposal_list = []\n for idx in range(min(100, len(df))):\n tmp_prop={\"score\":df.score.values[idx], \\\n \"segment\":[max(0,df.xmin.values[idx])*video_duration, \\\n min(1,df.xmax.values[idx])*video_duration]}\n proposal_list.append(tmp_prop)\n video_name = video_name[2:] if video_name[:2] == 'v_' else video_name",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:257-282"
+ },
+ "5791": {
+ "file_id": 465,
+ "content": "This function takes a list of video names, corresponding metadata dictionaries, output path, and result dictionary. It processes each video by reading its CSV file, performs soft NMS if the dataframe has more than one row, calculates proposal list for each video, and appends them to the result dictionary.",
+ "type": "comment"
+ },
+ "5792": {
+ "file_id": 465,
+ "content": " result_dict[video_name] = proposal_list\n def cal_metrics(self,\n ground_truth_filename,\n proposal_filename,\n max_avg_nr_proposals=100,\n tiou_thresholds=np.linspace(0.5, 0.95, 10),\n subset='validation'):\n anet_proposal = ANETproposal(ground_truth_filename,\n proposal_filename,\n tiou_thresholds=tiou_thresholds,\n max_avg_nr_proposals=max_avg_nr_proposals,\n subset=subset,\n verbose=True,\n check_status=False)\n anet_proposal.evaluate()\n recall = anet_proposal.recall\n average_recall = anet_proposal.avg_recall\n average_nr_proposals = anet_proposal.proposals_per_video\n return (average_nr_proposals, average_recall, recall)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/bmn_metric.py:283-304"
+ },
+ "5793": {
+ "file_id": 465,
+ "content": "The code defines a function \"cal_metrics\" that takes in ground truth and proposal filenames, calculates the average recall, average proposals per video, and overall recall using ANETproposal class. This function is used to evaluate performance based on given thresholds and subsets of data.",
+ "type": "comment"
+ },
+ "5794": {
+ "file_id": 466,
+ "content": "/paddlevideo/metrics/build.py",
+ "type": "filepath"
+ },
+ "5795": {
+ "file_id": 466,
+ "content": "Copyright notice, Apache License v2.0, software distributed as is without warranties or conditions. Imports registry and utils modules, defines build_metric function that builds metric using provided configuration.",
+ "type": "summary"
+ },
+ "5796": {
+ "file_id": 466,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import METRIC\nfrom ..utils import build\ndef build_metric(cfg):\n return build(cfg, METRIC)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/metrics/build.py:1-20"
+ },
+ "5797": {
+ "file_id": 466,
+ "content": "Copyright notice, Apache License v2.0, software distributed as is without warranties or conditions. Imports registry and utils modules, defines build_metric function that builds metric using provided configuration.",
+ "type": "comment"
+ },
+ "5798": {
+ "file_id": 467,
+ "content": "/paddlevideo/metrics/center_crop_metric.py",
+ "type": "filepath"
+ },
+ "5799": {
+ "file_id": 467,
+ "content": "The code initializes a metric class for PaddleVideo, handling batch updates and GPU data to mitigate resampling effects while managing output accumulation, concatenation, top-k accuracy, and logging.",
+ "type": "summary"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/58.json b/docs/data/58.json
new file mode 100644
index 000000000..7f538fc7e
--- /dev/null
+++ b/docs/data/58.json
@@ -0,0 +1,543 @@
+{
+ "5800": {
+ "file_id": 467,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom typing import List\nimport paddle\nfrom paddlevideo.utils import get_logger\nfrom .base import BaseMetric\nfrom .registry import METRIC\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass CenterCropMetric(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1, **kwargs):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval, **kwargs)\n self.rest_data_size = data_size # Number of samples remaining to be tested\n self.all_outputs = []",
+ "type": "code",
+ "location": "/paddlevideo/metrics/center_crop_metric.py:1-31"
+ },
+ "5801": {
+ "file_id": 467,
+ "content": "This code registers a class called CenterCropMetric as a metric in the PaddleVideo library. It initializes the metric with data_size, batch_size, and log_interval parameters. The rest_data_size is also stored to keep track of remaining samples to be tested.",
+ "type": "comment"
+ },
+ "5802": {
+ "file_id": 467,
+ "content": " self.all_labels = []\n self.topk = kwargs.get(\"topk\", [1, 5])\n def update(self, batch_id: int, data: List, outputs: paddle.Tensor) -> None:\n \"\"\"update metrics during each iter\n Args:\n batch_id (int): iter id of current batch.\n data (List): list of batched data, such as [inputs, labels]\n outputs (paddle.Tensor): batched outputs from model\n \"\"\"\n labels = data[1]\n if self.world_size > 1:\n labels_gathered = self.gather_from_gpu(labels, concat_axis=0)\n outpus_gathered = self.gather_from_gpu(outputs, concat_axis=0)\n else:\n labels_gathered = labels\n outpus_gathered = outputs\n # Avoid resampling effects when testing with multiple cards\n labels_gathered = labels_gathered[0:min(len(labels_gathered), self.\n rest_data_size)]\n outpus_gathered = outpus_gathered[0:min(len(outpus_gathered), self.\n rest_data_size)]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/center_crop_metric.py:32-55"
+ },
+ "5803": {
+ "file_id": 467,
+ "content": "This code is initializing a metric object, allowing for batch updates, and handling data from multiple GPUs to avoid resampling effects when testing with multiple cards.",
+ "type": "comment"
+ },
+ "5804": {
+ "file_id": 467,
+ "content": " self.all_labels.append(labels_gathered)\n self.all_outputs.append(outpus_gathered)\n self.rest_data_size -= outpus_gathered.shape[0]\n # preds ensemble\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate, compute, and show metrics when finished all iters.\n \"\"\"\n self.all_outputs = paddle.concat(self.all_outputs, axis=0)\n self.all_labels = paddle.concat(self.all_labels, axis=0)\n result_str = []\n for _k in self.topk:\n topk_val = paddle.metric.accuracy(input=self.all_outputs,\n label=self.all_labels,\n k=_k).item()\n result_str.append(f\"avg_acc{_k}={topk_val}\")\n result_str = \", \".join(result_str)\n logger.info(f\"[TEST] finished, {result_str}\")",
+ "type": "code",
+ "location": "/paddlevideo/metrics/center_crop_metric.py:56-79"
+ },
+ "5805": {
+ "file_id": 467,
+ "content": "The code is part of a class that seems to be handling batch processing in a machine learning application. It accumulates and concatenates outputs and labels from multiple batches, performs top-k accuracy calculations, and logs the results. The log_interval variable controls when progress updates are displayed.",
+ "type": "comment"
+ },
+ "5806": {
+ "file_id": 468,
+ "content": "/paddlevideo/metrics/center_crop_metric_MRI.py",
+ "type": "filepath"
+ },
+ "5807": {
+ "file_id": 468,
+ "content": "The code creates a class to calculate top-1 and possibly top-5 accuracy metrics in image classification tasks, tracking and averaging them during iteration, with support for multi-GPU scenarios using all-reduce operations.",
+ "type": "summary"
+ },
+ "5808": {
+ "file_id": 468,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass CenterCropMetric_MRI(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1, if_slowfast=0):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.top1 = []\n self.if_slowfast = if_slowfast\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter",
+ "type": "code",
+ "location": "/paddlevideo/metrics/center_crop_metric_MRI.py:1-33"
+ },
+ "5809": {
+ "file_id": 468,
+ "content": "This code defines the class CenterCropMetric_MRI, a metric for a video processing framework. It initializes variables and tracks top1 accuracy during iteration.",
+ "type": "comment"
+ },
+ "5810": {
+ "file_id": 468,
+ "content": " \"\"\"\n labels = data[1]\n if self.if_slowfast:\n labels = data[2]\n top1 = paddle.metric.accuracy(input=outputs, label=labels, k=1)\n #top5 = paddle.metric.accuracy(input=outputs, label=labels, k=5)\n #NOTE(shipping): deal with multi cards validate\n if self.world_size > 1:\n top1 = paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM) / self.world_size\n # top5 = paddle.distributed.all_reduce(\n # top5, op=paddle.distributed.ReduceOp.SUM) / self.world_size\n self.top1.append(top1.numpy())\n #self.top5.append(top5.numpy())\n # preds ensemble\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n logger.info('[TEST] finished, avg_acc1= {}'.format(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/center_crop_metric_MRI.py:34-60"
+ },
+ "5811": {
+ "file_id": 468,
+ "content": "This code snippet defines a class for calculating top-1 and possibly top-5 accuracy metrics in an image classification task. It collects the metrics for each batch during testing, then averages them at the end of all iterations. The code handles multi-GPU scenarios by performing all-reduce operations on the metric values.",
+ "type": "comment"
+ },
+ "5812": {
+ "file_id": 468,
+ "content": " np.mean(np.array(self.top1))))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/center_crop_metric_MRI.py:61-61"
+ },
+ "5813": {
+ "file_id": 468,
+ "content": "Calculates mean of top-1 accuracy across all samples in the batch.",
+ "type": "comment"
+ },
+ "5814": {
+ "file_id": 469,
+ "content": "/paddlevideo/metrics/depth_metric.py",
+ "type": "filepath"
+ },
+ "5815": {
+ "file_id": 469,
+ "content": "The `DepthMetric` class inherits from `BaseMetric`, processes batches, accumulates metrics and performs distributed all-reduce operations before averaging metric values.",
+ "type": "summary"
+ },
+ "5816": {
+ "file_id": 469,
+ "content": "import numpy as np\nimport paddle\nfrom paddlevideo.utils import get_logger\nfrom .base import BaseMetric\nfrom .registry import METRIC\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass DepthMetric(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.abs_rel = []\n self.sq_rel = []\n self.rmse = []\n self.rmse_log = []\n self.a1 = []\n self.a2 = []\n self.a3 = []\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3 = outputs['abs_rel'], outputs['sq_rel'], outputs['rmse'], \\\n outputs['rmse_log'], outputs['a1'], outputs['a2'],outputs['a3']\n # preds ensemble\n if self.world_size > 1:\n abs_rel = paddle.distributed.all_reduce(\n outputs['abs_rel'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size",
+ "type": "code",
+ "location": "/paddlevideo/metrics/depth_metric.py:1-34"
+ },
+ "5817": {
+ "file_id": 469,
+ "content": "This code defines a class `DepthMetric` that inherits from `BaseMetric`. It initializes lists for various metric values and then updates these metrics during each iteration. The code also includes logic to handle distributed computing, using all-reduce operation to average the results across different processes in the same training job.",
+ "type": "comment"
+ },
+ "5818": {
+ "file_id": 469,
+ "content": " sq_rel = paddle.distributed.all_reduce(\n outputs['sq_rel'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n rmse = paddle.distributed.all_reduce(\n outputs['rmse'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n rmse_log = paddle.distributed.all_reduce(\n outputs['rmse_log'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n a1 = paddle.distributed.all_reduce(\n outputs['a1'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n a2 = paddle.distributed.all_reduce(\n outputs['a2'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n a3 = paddle.distributed.all_reduce(\n outputs['a3'],\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n self.abs_rel.append(abs_rel)\n self.sq_rel.append(sq_rel)\n self.rmse.append(rmse)\n self.rmse_log.append(rmse_log)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/depth_metric.py:35-57"
+ },
+ "5819": {
+ "file_id": 469,
+ "content": "This code performs distributed all-reduce operations on several metrics (sq_rel, rmse, rmse\\_log, a1, a2, a3) and calculates their average values by dividing by the world size. These averaged metric values are then appended to corresponding lists (abs_rel, sq_rel, rmse, rmse_log).",
+ "type": "comment"
+ },
+ "5820": {
+ "file_id": 469,
+ "content": " self.a1.append(a1)\n self.a2.append(a2)\n self.a3.append(a3)\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n logger.info(\n '[TEST] finished, abs_rel= {}, sq_rel= {} , rmse= {}, rmse_log= {},'\n 'a1= {}, a2= {}, a3= {}'.format(np.mean(np.array(self.abs_rel)),\n np.mean(np.array(self.sq_rel)),\n np.mean(np.array(self.rmse)),\n np.mean(np.array(self.rmse_log)),\n np.mean(np.array(self.a1)),\n np.mean(np.array(self.a2)),\n np.mean(np.array(self.a3))))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/depth_metric.py:58-77"
+ },
+ "5821": {
+ "file_id": 469,
+ "content": "This code defines a class with methods for processing batches and accumulating metrics. The `process_batch` method appends data to lists, logs progress if the batch ID is divisible by log_interval, and handles the next batch. The `accumulate` method calculates mean values for each metric list and logs them using a logger with the corresponding metric values.",
+ "type": "comment"
+ },
+ "5822": {
+ "file_id": 470,
+ "content": "/paddlevideo/metrics/msrvtt_metric.py",
+ "type": "filepath"
+ },
+ "5823": {
+ "file_id": 470,
+ "content": "The code initializes and defines a class for MS-RNN/VTT model metrics computation, updates metrics using input data, calculates rank metrics (r1, r5, r10, medr, mean), logs these metrics, and signals the end of iterations.",
+ "type": "summary"
+ },
+ "5824": {
+ "file_id": 470,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nimport paddle.nn.functional as F\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass MSRVTTMetric(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.score_matrix = np.zeros((data_size, data_size))\n self.target_matrix = np.zeros((data_size, data_size))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/msrvtt_metric.py:1-31"
+ },
+ "5825": {
+ "file_id": 470,
+ "content": "The code is from the MSRVTTMetric class in paddlevideo's metrics module. It initializes an instance of the class, prepares for metrics computation, and creates score_matrix and target_matrix using numpy with zeroes. These matrices will be used to store results during metric calculations. The class also registers with the base BaseMetric class.",
+ "type": "comment"
+ },
+ "5826": {
+ "file_id": 470,
+ "content": " self.rank_matrix = np.ones((data_size)) * data_size\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n target = data[-1]\n cm_logit = outputs[-1]\n self.score_matrix[batch_id, :] = F.softmax(\n cm_logit, axis=1)[:, 0].reshape([-1]).numpy()\n self.target_matrix[batch_id, :] = target.reshape([-1]).numpy()\n rank = np.where((np.argsort(-self.score_matrix[batch_id]) == np.where(\n self.target_matrix[batch_id] == 1)[0][0]) == 1)[0][0]\n self.rank_matrix[batch_id] = rank\n rank_matrix_tmp = self.rank_matrix[:batch_id + 1]\n r1 = 100.0 * np.sum(rank_matrix_tmp < 1) / len(rank_matrix_tmp)\n r5 = 100.0 * np.sum(rank_matrix_tmp < 5) / len(rank_matrix_tmp)\n r10 = 100.0 * np.sum(rank_matrix_tmp < 10) / len(rank_matrix_tmp)\n medr = np.floor(np.median(rank_matrix_tmp) + 1)\n meanr = np.mean(rank_matrix_tmp) + 1\n logger.info(\n \"[{}] Final r1:{:.3f}, r5:{:.3f}, r10:{:.3f}, mder:{:.3f}, meanr:{:.3f}\"",
+ "type": "code",
+ "location": "/paddlevideo/metrics/msrvtt_metric.py:32-56"
+ },
+ "5827": {
+ "file_id": 470,
+ "content": "This code initializes a rank matrix, updates score and target matrices based on input data, calculates r1, r5, r10 rank metrics and median rank (medr) and mean rank, then logs these metrics.",
+ "type": "comment"
+ },
+ "5828": {
+ "file_id": 470,
+ "content": " .format(batch_id, r1, r5, r10, medr, meanr))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n logger.info(\"Eval Finished!\")",
+ "type": "code",
+ "location": "/paddlevideo/metrics/msrvtt_metric.py:57-62"
+ },
+ "5829": {
+ "file_id": 470,
+ "content": "This code defines a class for accumulating metrics related to the MS-RNN/VTT model. It seems to have methods for updating and finalizing the metric calculations. The update method takes in values r1, r5, r10, medr, and meanr, which are likely performance scores. The accumulate method signals the end of iterations by logging a message saying \"Eval Finished!\".",
+ "type": "comment"
+ },
+ "5830": {
+ "file_id": 471,
+ "content": "/paddlevideo/metrics/multi_crop_metric.py",
+ "type": "filepath"
+ },
+ "5831": {
+ "file_id": 471,
+ "content": "The MultiCropMetric class in PaddleVideo computes top-1/5 accuracy using multi-crop metrics for each video label and logs average values.",
+ "type": "summary"
+ },
+ "5832": {
+ "file_id": 471,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nfrom paddle.hapi.model import _all_gather\nfrom paddlevideo.utils import get_logger\nfrom .registry import METRIC\nfrom .base import BaseMetric\nlogger = get_logger(\"paddlevideo\")\n\"\"\" An example for metrics class.\n MultiCropMetric for slowfast.\n\"\"\"\n@METRIC.register\nclass MultiCropMetric(BaseMetric):\n def __init__(self,\n data_size,\n batch_size,\n num_ensemble_views,\n num_spatial_crops,\n num_classes,\n log_interval=1):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/multi_crop_metric.py:1-35"
+ },
+ "5833": {
+ "file_id": 471,
+ "content": "This code snippet defines a class for MultiCropMetric, which is part of the PaddleVideo library. It initializes instances with various parameters such as data_size, batch_size, num_ensemble_views, num_spatial_crops, and num_classes. The log_interval parameter determines how often to update logs during training. This metric appears to be related to slowfast video processing, as specified in the comment.",
+ "type": "comment"
+ },
+ "5834": {
+ "file_id": 471,
+ "content": " \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.num_ensemble_views = num_ensemble_views\n self.num_spatial_crops = num_spatial_crops\n self.num_classes = num_classes\n self.num_clips = self.num_ensemble_views * self.num_spatial_crops\n num_videos = self.data_size // self.num_clips\n self.video_preds = np.zeros((num_videos, self.num_classes))\n self.video_labels = np.zeros((num_videos, 1), dtype=\"int64\")\n self.clip_count = {}\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n labels = data[2]\n clip_ids = data[3]\n # gather mulit card, results of following process in each card is the same.\n if self.world_size > 1:\n outputs = _all_gather(outputs, self.world_size)\n labels = _all_gather(labels.cuda(), self.world_size)\n clip_ids = _all_gather(clip_ids.cuda(), self.world_size)\n # to numpy",
+ "type": "code",
+ "location": "/paddlevideo/metrics/multi_crop_metric.py:36-61"
+ },
+ "5835": {
+ "file_id": 471,
+ "content": "This code initializes a multi-crop metric class, which takes data size, batch size, log interval, number of ensemble views, and number of spatial crops as parameters. It calculates the number of videos and clips, creates arrays to store video predictions and labels, and initializes a clip_count dictionary. The update method is used to update metrics during each iteration by gathering data across multiple cards if needed, converting outputs to numpy arrays.",
+ "type": "comment"
+ },
+ "5836": {
+ "file_id": 471,
+ "content": " preds = outputs.numpy()\n labels = labels.numpy().astype(\"int64\")\n clip_ids = clip_ids.numpy()\n # preds ensemble\n for ind in range(preds.shape[0]):\n vid_id = int(clip_ids[ind]) // self.num_clips\n ts_idx = int(clip_ids[ind]) % self.num_clips\n if vid_id not in self.clip_count:\n self.clip_count[vid_id] = []\n if ts_idx in self.clip_count[vid_id]:\n logger.info(\n \"[TEST] Passed!! read video {} clip index {} / {} repeatedly.\"\n .format(vid_id, ts_idx, clip_ids[ind]))\n else:\n self.clip_count[vid_id].append(ts_idx)\n self.video_preds[vid_id] += preds[ind] # ensemble method: sum\n if self.video_labels[vid_id].sum() > 0:\n assert self.video_labels[vid_id] == labels[ind]\n self.video_labels[vid_id] = labels[ind]\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/multi_crop_metric.py:62-83"
+ },
+ "5837": {
+ "file_id": 471,
+ "content": "The code loops through each prediction and label for video clips in a batch. It checks if the clip index has been encountered before for a particular video ID, and updates the count if it's a new clip or performs ensemble by summing predictions if it's not. If there are labels for a video, it asserts they match and updates the label accordingly. The code also logs processing information at log intervals.",
+ "type": "comment"
+ },
+ "5838": {
+ "file_id": 471,
+ "content": " batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n # check clip index of each video\n for key in self.clip_count.keys():\n if len(self.clip_count[key]) != self.num_clips or sum(\n self.clip_count[key]) != self.num_clips * (self.num_clips -\n 1) / 2:\n logger.info(\n \"[TEST] Count Error!! video [{}] clip count [{}] not match number clips {}\"\n .format(key, self.clip_count[key], self.num_clips))\n video_preds = paddle.to_tensor(self.video_preds)\n video_labels = paddle.to_tensor(self.video_labels)\n acc_top1 = paddle.metric.accuracy(input=video_preds,\n label=video_labels,\n k=1)\n acc_top5 = paddle.metric.accuracy(input=video_preds,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/multi_crop_metric.py:84-104"
+ },
+ "5839": {
+ "file_id": 471,
+ "content": "This code defines a class that accumulates metrics when all iterations are finished. It checks if the number of clips and their counts match, logging an error if not. Then, it converts video predictions and labels to Paddle tensors and calculates top-1 and top-5 accuracy using Paddle's metric library.",
+ "type": "comment"
+ },
+ "5840": {
+ "file_id": 471,
+ "content": " label=video_labels,\n k=5)\n logger.info('[TEST] finished, avg_acc1= {}, avg_acc5= {} '.format(\n acc_top1.numpy(), acc_top5.numpy()))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/multi_crop_metric.py:105-108"
+ },
+ "5841": {
+ "file_id": 471,
+ "content": "Calculates top-1 and top-5 accuracy using multi-crop metric for each video label, then logs the average accuracy values.",
+ "type": "comment"
+ },
+ "5842": {
+ "file_id": 472,
+ "content": "/paddlevideo/metrics/recall.py",
+ "type": "filepath"
+ },
+ "5843": {
+ "file_id": 472,
+ "content": "This code calculates recall for object detection by iterating through images, selecting proposals based on scores, and calculating IoU between these proposals and ground truth boxes. It defines `recalls()` and `eval_recalls()`, with the latter computing recalls for each image.",
+ "type": "summary"
+ },
+ "5844": {
+ "file_id": 472,
+ "content": "import numpy as np\nimport paddle \ndef _recalls(all_ious, proposal_nums, thrs):\n img_num = all_ious.shape[0]\n total_gt_num = sum([ious.shape[0] for ious in all_ious])\n ious_ = np.zeros((proposal_nums.size, total_gt_num), dtype=np.float32)\n for k, proposal_num in enumerate(proposal_nums):\n tmp_ious = np.zeros(0)\n for i in range(img_num):\n ious = all_ious[i][:, :proposal_num].copy()\n gt_ious = np.zeros(ious.shape[0])\n if ious.size == 0:\n tmp_ious = np.hstack((tmp_ious, gt_ious))\n continue\n for j in range(ious.shape[0]):\n gt_max_overlaps = ious.argmax(axis=1)\n max_ious = ious[np.arange(0, ious.shape[0]), gt_max_overlaps]\n gt_idx = max_ious.argmax()\n gt_ious[j] = max_ious[gt_idx]\n box_idx = gt_max_overlaps[gt_idx]\n ious[gt_idx, :] = -1\n ious[:, box_idx] = -1\n tmp_ious = np.hstack((tmp_ious, gt_ious))\n ious_[k, :] = tmp_ious",
+ "type": "code",
+ "location": "/paddlevideo/metrics/recall.py:1-27"
+ },
+ "5845": {
+ "file_id": 472,
+ "content": "This code calculates recall metric for object detection. It takes in all IoUs (intersection over union), proposal numbers, and thresholds as input. It iterates through images and gt (ground truth) boxes, and then computes the recall scores for each image and stores them in an array. The recall is computed by finding the maximum IOU between ground truth and proposals for each image and storing it in a temporary array, then concatenating these values into the final result.",
+ "type": "comment"
+ },
+ "5846": {
+ "file_id": 472,
+ "content": " ious_ = np.fliplr(np.sort(ious_, axis=1))\n recalls = np.zeros((proposal_nums.size, thrs.size))\n for i, thr in enumerate(thrs):\n recalls[:, i] = (ious_ >= thr).sum(axis=1) / float(total_gt_num)\n return recalls\ndef set_recall_param(proposal_nums, iou_thrs):\n if isinstance(proposal_nums, list):\n proposal_nums_ = np.array(proposal_nums)\n elif isinstance(proposal_nums, int):\n proposal_nums_ = np.array([proposal_nums])\n else:\n proposal_nums_ = proposal_nums\n if iou_thrs is None:\n _iou_thrs = np.array([0.5])\n elif isinstance(iou_thrs, list):\n _iou_thrs = np.array(iou_thrs)\n elif isinstance(iou_thrs, float):\n _iou_thrs = np.array([iou_thrs])\n else:\n _iou_thrs = iou_thrs\n return proposal_nums_, _iou_thrs\ndef eval_recalls(gts, proposals, proposal_nums=None, iou_thrs=None):\n \"\"\"Calculate recalls. \"\"\"\n img_num = len(gts)\n assert img_num == len(proposals)\n proposal_nums, iou_thrs = set_recall_param(proposal_nums, iou_thrs)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/recall.py:29-62"
+ },
+ "5847": {
+ "file_id": 472,
+ "content": "The code defines two functions: `recalls()` and `eval_recalls()`. \n\n`recalls()` calculates the average precision-recall curve by comparing predicted bounding boxes with ground truth ones. It does this by sorting intersection over union (IOU) values, creating a recall matrix based on IOU thresholds, and averaging recall across images.\n\n`eval_recalls()` is a wrapper function that calls `recalls()`. It calculates recalls for each image given ground truths and proposals. It also checks input types and sets default parameters if necessary.",
+ "type": "comment"
+ },
+ "5848": {
+ "file_id": 472,
+ "content": " all_ious = []\n for i in range(img_num):\n if proposals[i].ndim == 2 and proposals[i].shape[1] == 5:\n scores = proposals[i][:, 4]\n sort_idx = np.argsort(scores)[::-1]\n img_proposal = proposals[i][sort_idx, :]\n else:\n img_proposal = proposals[i]\n prop_num = min(img_proposal.shape[0], proposal_nums[-1])\n if gts[i] is None or gts[i].shape[0] == 0:\n ious = np.zeros((0, img_proposal.shape[0]), dtype=np.float32)\n else:\n ious = bbox_overlaps(\n torch.tensor(gts[i]),\n torch.tensor(img_proposal[:prop_num, :4]))\n ious = ious.data.numpy()\n all_ious.append(ious)\n all_ious = np.array(all_ious)\n recalls = _recalls(all_ious, proposal_nums, iou_thrs)\n return recalls",
+ "type": "code",
+ "location": "/paddlevideo/metrics/recall.py:64-84"
+ },
+ "5849": {
+ "file_id": 472,
+ "content": "This code calculates recall for object detection. It iterates through images, sorts and selects proposals based on scores, and then calculates IoU (intersection over union) between these proposals and ground truth boxes. If no ground truth is found or it has zero boxes, all_ious is filled with zeros. The function returns recalls for each image number.",
+ "type": "comment"
+ },
+ "5850": {
+ "file_id": 473,
+ "content": "/paddlevideo/metrics/registry.py",
+ "type": "filepath"
+ },
+ "5851": {
+ "file_id": 473,
+ "content": "The code imports a Registry class from the utils module and initializes the METRIC as an instance of this Registry, designed to store and manage different types of metrics.",
+ "type": "summary"
+ },
+ "5852": {
+ "file_id": 473,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..utils import Registry\nMETRIC = Registry('metric')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/metrics/registry.py:1-17"
+ },
+ "5853": {
+ "file_id": 473,
+ "content": "The code imports a Registry class from the utils module and initializes the METRIC as an instance of this Registry, designed to store and manage different types of metrics.",
+ "type": "comment"
+ },
+ "5854": {
+ "file_id": 474,
+ "content": "/paddlevideo/metrics/segmentation_metric.py",
+ "type": "filepath"
+ },
+ "5855": {
+ "file_id": 474,
+ "content": "The code creates a function for label change detection in video segmentation, computes precision, recall, and F1 score, uses Levenstein distance, and evaluates ground truth and predicted actions.",
+ "type": "summary"
+ },
+ "5856": {
+ "file_id": 474,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport argparse\nimport pandas as pd\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ndef get_labels_scores_start_end_time(input_np,\n frame_wise_labels,\n actions_dict,\n bg_class=[\"background\", \"None\"]):\n labels = []\n starts = []\n ends = []\n scores = []\n boundary_score_ptr = 0\n last_label = frame_wise_labels[0]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:1-35"
+ },
+ "5857": {
+ "file_id": 474,
+ "content": "This code is a part of PaddleVideo library and defines a function get_labels_scores_start_end_time that takes input, frame-wise labels, actions dictionary, and optional background class. It returns labels, starts, ends, and scores based on the input and labels. The function also keeps track of the boundary score pointer and the last label.",
+ "type": "comment"
+ },
+ "5858": {
+ "file_id": 474,
+ "content": " if frame_wise_labels[0] not in bg_class:\n labels.append(frame_wise_labels[0])\n starts.append(0)\n for i in range(len(frame_wise_labels)):\n if frame_wise_labels[i] != last_label:\n if frame_wise_labels[i] not in bg_class:\n labels.append(frame_wise_labels[i])\n starts.append(i)\n if last_label not in bg_class:\n ends.append(i)\n score = np.mean(\n input_np[actions_dict[labels[boundary_score_ptr]], \\\n starts[boundary_score_ptr]:(ends[boundary_score_ptr] + 1)]\n )\n scores.append(score)\n boundary_score_ptr = boundary_score_ptr + 1\n last_label = frame_wise_labels[i]\n if last_label not in bg_class:\n ends.append(i + 1)\n score = np.mean(\n input_np[actions_dict[labels[boundary_score_ptr]], \\\n starts[boundary_score_ptr]:(ends[boundary_score_ptr] + 1)]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:36-57"
+ },
+ "5859": {
+ "file_id": 474,
+ "content": "This code segment is a part of a larger video analysis algorithm. It identifies changes in frame-wise labels and calculates scores for those changes based on input_np data associated with the actions_dict[labels]. The scores are then appended to the 'scores' list, while starts and ends lists keep track of the start and end indices for each identified change. Finally, if the last label is not in bg_class, it adds an ending index to the 'ends' list and calculates a score using input_np data associated with starts and ends indices. The code then updates boundary_score_ptr and proceeds to the next iteration.",
+ "type": "comment"
+ },
+ "5860": {
+ "file_id": 474,
+ "content": " )\n scores.append(score)\n boundary_score_ptr = boundary_score_ptr + 1\n return labels, starts, ends, scores\ndef get_labels_start_end_time(frame_wise_labels,\n bg_class=[\"background\", \"None\"]):\n labels = []\n starts = []\n ends = []\n last_label = frame_wise_labels[0]\n if frame_wise_labels[0] not in bg_class:\n labels.append(frame_wise_labels[0])\n starts.append(0)\n for i in range(len(frame_wise_labels)):\n if frame_wise_labels[i] != last_label:\n if frame_wise_labels[i] not in bg_class:\n labels.append(frame_wise_labels[i])\n starts.append(i)\n if last_label not in bg_class:\n ends.append(i)\n last_label = frame_wise_labels[i]\n if last_label not in bg_class:\n ends.append(i + 1)\n return labels, starts, ends\ndef levenstein(p, y, norm=False):\n m_row = len(p)\n n_col = len(y)\n D = np.zeros([m_row + 1, n_col + 1], np.float)\n for i in range(m_row + 1):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:58-91"
+ },
+ "5861": {
+ "file_id": 474,
+ "content": "The first code chunk is a function that takes a list of frame-wise labels, iterates over the frames, and returns lists for the label names, starting indices, and ending indices. It appends new label names to the labels list and new starting indices to the starts list when a new label appears, and adds corresponding ending indices to the ends list if the previous label was not \"background\" or \"None\". The last label's ending index is added after the loop.\n\nThe second code chunk defines a function that calculates the Levenshtein distance between two strings (p and y) using dynamic programming, which measures the minimum number of operations required to transform one string into another (insertion, deletion, or substitution). The function creates a 2D array D of size (m_row + 1) x (n_col + 1), where m_row is the length of p and n_col is the length of y. It then fills the array using dynamic programming, considering different operations at each step to calculate the minimum distance. This function likely uses the D array for further calculations or returns it as a result.",
+ "type": "comment"
+ },
+ "5862": {
+ "file_id": 474,
+ "content": " D[i, 0] = i\n for i in range(n_col + 1):\n D[0, i] = i\n for j in range(1, n_col + 1):\n for i in range(1, m_row + 1):\n if y[j - 1] == p[i - 1]:\n D[i, j] = D[i - 1, j - 1]\n else:\n D[i, j] = min(D[i - 1, j] + 1, D[i, j - 1] + 1,\n D[i - 1, j - 1] + 1)\n if norm:\n score = (1 - D[-1, -1] / max(m_row, n_col)) * 100\n else:\n score = D[-1, -1]\n return score\ndef edit_score(recognized,\n ground_truth,\n norm=True,\n bg_class=[\"background\", \"None\"]):\n P, _, _ = get_labels_start_end_time(recognized, bg_class)\n Y, _, _ = get_labels_start_end_time(ground_truth, bg_class)\n return levenstein(P, Y, norm)\ndef f_score(recognized, ground_truth, overlap, bg_class=[\"background\", \"None\"]):\n p_label, p_start, p_end = get_labels_start_end_time(recognized, bg_class)\n y_label, y_start, y_end = get_labels_start_end_time(ground_truth, bg_class)\n tp = 0\n fp = 0",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:92-126"
+ },
+ "5863": {
+ "file_id": 474,
+ "content": "The code contains a function called \"levenstein\" that calculates the Levenstein distance between two sequences. The Levenstein distance is a metric used to measure the difference between two strings of characters, such as words or labels. In this case, it's used to compare the recognized and ground truth labels for video segmentation. The function takes in a pair of lists P and Y representing recognized and ground truth labels respectively, and an optional norm parameter which normalizes the score if True. The output is a single numeric value representing the distance between the two lists of labels. This score can be used to evaluate the accuracy of the recognition algorithm in comparison with the ground truth data.",
+ "type": "comment"
+ },
+ "5864": {
+ "file_id": 474,
+ "content": " hits = np.zeros(len(y_label))\n for j in range(len(p_label)):\n intersection = np.minimum(p_end[j], y_end) - np.maximum(\n p_start[j], y_start)\n union = np.maximum(p_end[j], y_end) - np.minimum(p_start[j], y_start)\n IoU = (1.0 * intersection / union) * (\n [p_label[j] == y_label[x] for x in range(len(y_label))])\n # Get the best scoring segment\n idx = np.array(IoU).argmax()\n if IoU[idx] >= overlap and not hits[idx]:\n tp += 1\n hits[idx] = 1\n else:\n fp += 1\n fn = len(y_label) - sum(hits)\n return float(tp), float(fp), float(fn)\ndef boundary_AR(pred_boundary, gt_boundary, overlap_list, max_proposal):\n p_label, p_start, p_end, p_scores = pred_boundary\n y_label, y_start, y_end, _ = gt_boundary\n # sort proposal\n pred_dict = {\n \"label\": p_label,\n \"start\": p_start,\n \"end\": p_end,\n \"scores\": p_scores\n }\n pdf = pd.DataFrame(pred_dict)\n pdf = pdf.sort_values(by=\"scores\", ascending=False)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:128-161"
+ },
+ "5865": {
+ "file_id": 474,
+ "content": "The code calculates the precision, recall, and F1 score for image segmentation by iterating through predicted and ground truth labels. It assigns hits when there is a match between predictions and ground truth, and counts true positives (TP), false positives (FP), and false negatives (FN). The boundary_AR function takes in predicted and ground truth boundaries, sorts them based on scores, and calculates various metrics for image segmentation.",
+ "type": "comment"
+ },
+ "5866": {
+ "file_id": 474,
+ "content": " p_label = list(pdf[\"label\"])\n p_start = list(pdf[\"start\"])\n p_end = list(pdf[\"end\"])\n p_scores = list(pdf[\"scores\"])\n # refine AN\n if len(p_label) < max_proposal and len(p_label) > 0:\n p_label = p_label + [p_label[-1]] * (max_proposal - len(p_label))\n p_start = p_start + [p_start[-1]] * (max_proposal - len(p_start))\n p_start = p_start + p_start[len(p_start) -\n (max_proposal - len(p_start)):]\n p_end = p_end + [p_end[-1]] * (max_proposal - len(p_end))\n p_scores = p_scores + [p_scores[-1]] * (max_proposal - len(p_scores))\n elif len(p_label) > max_proposal:\n p_label[max_proposal:] = []\n p_start[max_proposal:] = []\n p_end[max_proposal:] = []\n p_scores[max_proposal:] = []\n t_AR = np.zeros(len(overlap_list))\n for i in range(len(overlap_list)):\n overlap = overlap_list[i]\n tp = 0\n fp = 0\n hits = np.zeros(len(y_label))\n for j in range(len(p_label)):\n intersection = np.minimum(p_end[j], y_end) - np.maximum(",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:162-191"
+ },
+ "5867": {
+ "file_id": 474,
+ "content": "This code segment handles the refinement of proposals in an object detection model. If the number of proposals is less than the maximum allowed, it repeats the last proposal to meet the requirement. If there are more proposals than the maximum allowed, it discards extra proposals. The code then calculates the average recall (AR) by iterating over the overlap list and counting true positives (tp) and false positives (fp). It also initializes hits for each proposal in the ground truth labels.",
+ "type": "comment"
+ },
+ "5868": {
+ "file_id": 474,
+ "content": " p_start[j], y_start)\n union = np.maximum(p_end[j], y_end) - np.minimum(\n p_start[j], y_start)\n IoU = (1.0 * intersection / union)\n # Get the best scoring segment\n idx = np.array(IoU).argmax()\n if IoU[idx] >= overlap and not hits[idx]:\n tp += 1\n hits[idx] = 1\n else:\n fp += 1\n fn = len(y_label) - sum(hits)\n recall = float(tp) / (float(tp) + float(fn))\n t_AR[i] = recall\n AR = np.mean(t_AR)\n return AR\n@METRIC.register\nclass SegmentationMetric(BaseMetric):\n \"\"\"\n Test for Video Segmentation based model.\n \"\"\"\n def __init__(self,\n data_size,\n batch_size,\n overlap,\n actions_map_file_path,\n log_interval=1,\n tolerance=5,\n boundary_threshold=0.7,\n max_proposal=100):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:192-230"
+ },
+ "5869": {
+ "file_id": 474,
+ "content": "This code calculates the Average Recall (AR) for video segmentation models. It iterates through ground truth and predicted labels, calculates intersection over union (IoU), counts true positives (tp), false positives (fp), and false negatives (fn). Then it computes recall and averages them to obtain AR. The SegmentationMetric class initializes with various parameters for the metric calculation.",
+ "type": "comment"
+ },
+ "5870": {
+ "file_id": 474,
+ "content": " # actions dict generate\n file_ptr = open(actions_map_file_path, 'r')\n actions = file_ptr.read().split('\\n')[:-1]\n file_ptr.close()\n self.actions_dict = dict()\n for a in actions:\n self.actions_dict[a.split()[1]] = int(a.split()[0])\n # cls score\n self.overlap = overlap\n self.overlap_len = len(overlap)\n self.cls_tp = np.zeros(self.overlap_len)\n self.cls_fp = np.zeros(self.overlap_len)\n self.cls_fn = np.zeros(self.overlap_len)\n self.total_correct = 0\n self.total_edit = 0\n self.total_frame = 0\n self.total_video = 0\n # boundary score\n self.max_proposal = max_proposal\n self.AR_at_AN = [[] for _ in range(max_proposal)]\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n groundTruth = data[1]\n predicted = outputs['predict']\n output_np = outputs['output_np']\n outputs_np = predicted.numpy()\n outputs_arr = output_np.numpy()[0, :]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:231-264"
+ },
+ "5871": {
+ "file_id": 474,
+ "content": "This code initializes a SegmentationMetric object, reads an actions map file, and prepares to update metrics during each iteration. It calculates true positives (cls_tp), false positives (cls_fp), and false negatives (cls_fn) for each frame's overlap. The AR_at_AN is also initialized with empty lists for each max_proposal value.",
+ "type": "comment"
+ },
+ "5872": {
+ "file_id": 474,
+ "content": " gt_np = groundTruth.numpy()[0, :]\n recognition = []\n for i in range(outputs_np.shape[0]):\n recognition = np.concatenate((recognition, [\n list(self.actions_dict.keys())[list(\n self.actions_dict.values()).index(outputs_np[i])]\n ]))\n recog_content = list(recognition)\n gt_content = []\n for i in range(gt_np.shape[0]):\n gt_content = np.concatenate((gt_content, [\n list(self.actions_dict.keys())[list(\n self.actions_dict.values()).index(gt_np[i])]\n ]))\n gt_content = list(gt_content)\n pred_boundary = get_labels_scores_start_end_time(\n outputs_arr, recog_content, self.actions_dict)\n gt_boundary = get_labels_scores_start_end_time(\n np.ones(outputs_arr.shape), gt_content, self.actions_dict)\n # cls score\n correct = 0\n total = 0\n edit = 0\n for i in range(len(gt_content)):\n total += 1\n #accumulate",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:265-295"
+ },
+ "5873": {
+ "file_id": 474,
+ "content": "This code segment compares ground truth and predicted actions for a video. It converts the numpy arrays to lists, generates predicted and ground truth boundaries using the `get_labels_scores_start_end_time` function, and then initializes variables for accuracy calculation.",
+ "type": "comment"
+ },
+ "5874": {
+ "file_id": 474,
+ "content": " self.total_frame += 1\n if gt_content[i] == recog_content[i]:\n correct += 1\n #accumulate\n self.total_correct += 1\n edit_num = edit_score(recog_content, gt_content)\n edit += edit_num\n self.total_edit += edit_num\n for s in range(self.overlap_len):\n tp1, fp1, fn1 = f_score(recog_content, gt_content, self.overlap[s])\n # accumulate\n self.cls_tp[s] += tp1\n self.cls_fp[s] += fp1\n self.cls_fn[s] += fn1\n # accumulate\n self.total_video += 1\n # proposal score\n for AN in range(self.max_proposal):\n AR = boundary_AR(pred_boundary,\n gt_boundary,\n self.overlap,\n max_proposal=(AN + 1))\n self.AR_at_AN[AN].append(AR)\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n # cls metric\n Acc = 100 * float(self.total_correct) / self.total_frame",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:296-330"
+ },
+ "5875": {
+ "file_id": 474,
+ "content": "This code calculates segmentation metrics, including accuracy, false positives, and false negatives for video frames. It also keeps track of total correct predictions, edit distances, proposal scores, and accumulates these metrics per video. The `accumulate` function is used to calculate classification accuracy when all iterations are finished.",
+ "type": "comment"
+ },
+ "5876": {
+ "file_id": 474,
+ "content": " Edit = (1.0 * self.total_edit) / self.total_video\n Fscore = dict()\n for s in range(self.overlap_len):\n precision = self.cls_tp[s] / float(self.cls_tp[s] + self.cls_fp[s])\n recall = self.cls_tp[s] / float(self.cls_tp[s] + self.cls_fn[s])\n f1 = 2.0 * (precision * recall) / (precision + recall)\n f1 = np.nan_to_num(f1) * 100\n Fscore[self.overlap[s]] = f1\n # proposal metric\n proposal_AUC = np.array(self.AR_at_AN) * 100\n AUC = np.mean(proposal_AUC)\n AR_at_AN1 = np.mean(proposal_AUC[0, :])\n AR_at_AN5 = np.mean(proposal_AUC[4, :])\n AR_at_AN15 = np.mean(proposal_AUC[14, :])\n # log metric\n log_mertic_info = \"dataset model performence: \"\n # preds ensemble\n log_mertic_info += \"Acc: {:.4f}, \".format(Acc)\n log_mertic_info += 'Edit: {:.4f}, '.format(Edit)\n for s in range(len(self.overlap)):\n log_mertic_info += 'F1@{:0.2f}: {:.4f}, '.format(\n self.overlap[s], Fscore[self.overlap[s]])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:331-356"
+ },
+ "5877": {
+ "file_id": 474,
+ "content": "This code calculates segmentation metrics, including Edit distance, F1 score at different overlap levels, and proposal area under the curve (AUC). It then stores these values in a dictionary and computes average AUCs for different overlap thresholds. The code also calculates an ensemble metric based on accuracy (Acc) and Edit distance. Finally, it logs this information as a string.",
+ "type": "comment"
+ },
+ "5878": {
+ "file_id": 474,
+ "content": " # boundary metric\n log_mertic_info += \"Auc: {:.4f}, \".format(AUC)\n log_mertic_info += \"AR@AN1: {:.4f}, \".format(AR_at_AN1)\n log_mertic_info += \"AR@AN5: {:.4f}, \".format(AR_at_AN5)\n log_mertic_info += \"AR@AN15: {:.4f}, \".format(AR_at_AN15)\n logger.info(log_mertic_info)\n # log metric\n metric_dict = dict()\n metric_dict['Acc'] = Acc\n metric_dict['Edit'] = Edit\n for s in range(len(self.overlap)):\n metric_dict['F1@{:0.2f}'.format(\n self.overlap[s])] = Fscore[self.overlap[s]]\n metric_dict['Auc'] = AUC\n metric_dict['AR@AN1'] = AR_at_AN1\n metric_dict['AR@AN5'] = AR_at_AN5\n metric_dict['AR@AN15'] = AR_at_AN15\n # clear for next epoch\n # cls\n self.cls_tp = np.zeros(self.overlap_len)\n self.cls_fp = np.zeros(self.overlap_len)\n self.cls_fn = np.zeros(self.overlap_len)\n self.total_correct = 0\n self.total_edit = 0\n self.total_frame = 0\n self.total_video = 0",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:358-385"
+ },
+ "5879": {
+ "file_id": 474,
+ "content": "This code calculates and logs various segmentation metrics, including AUC, AR@AN1, AR@AN5, and AR@AN15. It also updates the metric dictionary with F1 scores for different overlap thresholds and clears the classifier statistics for the next epoch.",
+ "type": "comment"
+ },
+ "5880": {
+ "file_id": 474,
+ "content": " # proposal\n self.AR_at_AN = [[] for _ in range(self.max_proposal)]\n return metric_dict",
+ "type": "code",
+ "location": "/paddlevideo/metrics/segmentation_metric.py:386-389"
+ },
+ "5881": {
+ "file_id": 474,
+ "content": "This code initializes the attribute \"AR_at_AN\" as a list of empty lists, with the length equal to the maximum number of proposals. This is done within the context of a class method and the list will likely be used for storing evaluation metrics related to these proposals. The method then returns a dictionary (metric_dict) containing other potentially calculated metrics.",
+ "type": "comment"
+ },
+ "5882": {
+ "file_id": 475,
+ "content": "/paddlevideo/metrics/skeleton_metric.py",
+ "type": "filepath"
+ },
+ "5883": {
+ "file_id": 475,
+ "content": "The SkeletonMetric class in PaddleVideo measures skeleton-based model performance metrics, supports batch size 1 and single card testing, and calculates top1 and top5 accuracy for batches with labels. It logs processing info, updates progress, and accumulates metrics while saving results to 'submission.csv'.",
+ "type": "summary"
+ },
+ "5884": {
+ "file_id": 475,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nimport csv\nimport paddle.nn.functional as F\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass SkeletonMetric(BaseMetric):\n \"\"\"\n Test for Skeleton based model.\n note: only support batch size = 1, single card test.\n Args:\n out_file: str, file to save test results.\n \"\"\"\n def __init__(self,\n data_size,\n batch_size,\n out_file='submission.csv',",
+ "type": "code",
+ "location": "/paddlevideo/metrics/skeleton_metric.py:1-38"
+ },
+ "5885": {
+ "file_id": 475,
+ "content": "This code is for the SkeletonMetric class in PaddleVideo, a machine learning framework. The class measures performance metrics for skeleton-based models. It supports batch size 1 and single card testing. Results can be saved to a file named 'submission.csv'.",
+ "type": "comment"
+ },
+ "5886": {
+ "file_id": 475,
+ "content": " log_interval=1,\n top_k=5):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.top1 = []\n self.top5 = []\n self.values = []\n self.out_file = out_file\n self.k = top_k\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n if data[0].shape[0] != outputs.shape[0]:\n num_segs = data[0].shape[1]\n batch_size = outputs.shape[0]\n outputs = outputs.reshape(\n [batch_size // num_segs, num_segs, outputs.shape[-1]])\n outputs = outputs.mean(axis=1)\n if len(data) == 2: # data with label\n labels = data[1]\n top1 = paddle.metric.accuracy(input=outputs, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=outputs, label=labels, k=self.k)\n if self.world_size > 1:\n top1 = paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM) / self.world_size",
+ "type": "code",
+ "location": "/paddlevideo/metrics/skeleton_metric.py:39-65"
+ },
+ "5887": {
+ "file_id": 475,
+ "content": "This code initializes a metrics class for tracking accuracy metrics during training. The `__init__` function sets up the top1, top5, and values lists to store metric results, as well as an output file path and the desired top k value. The `update` method processes data from each batch iteration, updating the metrics based on whether the input data contains labels or not. It also handles distributed training by averaging across multiple workers.",
+ "type": "comment"
+ },
+ "5888": {
+ "file_id": 475,
+ "content": " top5 = paddle.distributed.all_reduce(\n top5, op=paddle.distributed.ReduceOp.SUM) / self.world_size\n self.top1.append(top1.numpy())\n self.top5.append(top5.numpy())\n else: # data without label, only support batch_size=1. Used for fsd-10.\n prob = F.softmax(outputs)\n clas = paddle.argmax(prob, axis=1).numpy()[0]\n self.values.append((batch_id, clas))\n # preds ensemble\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n if self.top1: # data with label\n logger.info('[TEST] finished, avg_acc1= {}, avg_acc5= {}'.format(\n np.mean(np.array(self.top1)), np.mean(np.array(self.top5))))\n else:\n headers = ['sample_index', 'predict_category']",
+ "type": "code",
+ "location": "/paddlevideo/metrics/skeleton_metric.py:66-88"
+ },
+ "5889": {
+ "file_id": 475,
+ "content": "This code segment is part of a class that handles metrics for a testing process. It calculates top1 and top5 accuracy for batches with labels and stores them. For batches without labels, it performs softmax on outputs and gets the class with highest probability. It logs processing information and updates progress. Finally, it accumulates metrics when all iterations are done.",
+ "type": "comment"
+ },
+ "5890": {
+ "file_id": 475,
+ "content": " with open(\n self.out_file,\n 'w',\n ) as fp:\n writer = csv.writer(fp)\n writer.writerow(headers)\n writer.writerows(self.values)\n logger.info(\"Results saved in {} !\".format(self.out_file))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/skeleton_metric.py:89-96"
+ },
+ "5891": {
+ "file_id": 475,
+ "content": "Writes headers and values from self.values to file, saves results in out_file and logs success.",
+ "type": "comment"
+ },
+ "5892": {
+ "file_id": 476,
+ "content": "/paddlevideo/metrics/transnetv2_metric.py",
+ "type": "filepath"
+ },
+ "5893": {
+ "file_id": 476,
+ "content": "This code calculates precision, recall, and F1-score for a TransNetV2 metric machine learning model by handling scene location transformations and errors. It iterates through different thresholds before logging the results.",
+ "type": "summary"
+ },
+ "5894": {
+ "file_id": 476,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ndef predictions_to_scenes(predictions):\n scenes = []\n t, t_prev, start = -1, 0, 0\n for i, t in enumerate(predictions):\n if t_prev == 1 and t == 0:\n start = i\n if t_prev == 0 and t == 1 and i != 0:\n scenes.append([start, i])\n t_prev = t\n if t == 0:\n scenes.append([start, i])\n # just fix if all predictions are 1",
+ "type": "code",
+ "location": "/paddlevideo/metrics/transnetv2_metric.py:1-34"
+ },
+ "5895": {
+ "file_id": 476,
+ "content": "This function, named \"predictions_to_scenes\", takes in a list of predictions and outputs a list of scene locations. The scenes are determined by identifying changes from 0 to 1 and vice versa. If all predictions are 1, the function adds a final scene ending at the last index. The code also includes error checking for cases where all predictions are 1 or when there is a disruption in prediction data flow.",
+ "type": "comment"
+ },
+ "5896": {
+ "file_id": 476,
+ "content": " if len(scenes) == 0:\n return np.array([[0, len(predictions) - 1]], dtype=np.int32)\n return np.array(scenes, dtype=np.int32)\ndef evaluate_scenes(gt_scenes, pred_scenes, n_frames_miss_tolerance=2):\n \"\"\"\n Adapted from: https://github.com/gyglim/shot-detection-evaluation\n The original based on: http://imagelab.ing.unimore.it/imagelab/researchActivity.asp?idActivity=19\n n_frames_miss_tolerance:\n Number of frames it is possible to miss ground truth by, and still being counted as a correct detection.\n Examples of computation with different tolerance margin:\n n_frames_miss_tolerance = 0\n pred_scenes: [[0, 5], [6, 9]] -> pred_trans: [[5.5, 5.5]]\n gt_scenes: [[0, 5], [6, 9]] -> gt_trans: [[5.5, 5.5]] -> HIT\n gt_scenes: [[0, 4], [5, 9]] -> gt_trans: [[4.5, 4.5]] -> MISS\n n_frames_miss_tolerance = 1\n pred_scenes: [[0, 5], [6, 9]] -> pred_trans: [[5.0, 6.0]]\n gt_scenes: [[0, 5], [6, 9]] -> gt_trans: [[5.0, 6.0]] -> HIT\n gt_scenes: [[0, 4], [5, 9]] -> gt_trans: [[4.0, 5.0]] -> HIT",
+ "type": "code",
+ "location": "/paddlevideo/metrics/transnetv2_metric.py:35-57"
+ },
+ "5897": {
+ "file_id": 476,
+ "content": "This function converts scene lists to transition lists. If there are no scenes, it returns a transition list with one element. The function is based on an external source and adapted for specific use cases. It can handle different tolerance margins, which affects how the pred_scenes and gt_scenes are transformed into prediction transitions (pred_trans) and ground truth transitions (gt_trans), respectively. A \"HIT\" or \"MISS\" status is determined based on these converted lists.",
+ "type": "comment"
+ },
+ "5898": {
+ "file_id": 476,
+ "content": " gt_scenes: [[0, 3], [4, 9]] -> gt_trans: [[3.0, 4.0]] -> MISS\n n_frames_miss_tolerance = 2\n pred_scenes: [[0, 5], [6, 9]] -> pred_trans: [[4.5, 6.5]]\n gt_scenes: [[0, 5], [6, 9]] -> gt_trans: [[4.5, 6.5]] -> HIT\n gt_scenes: [[0, 4], [5, 9]] -> gt_trans: [[3.5, 5.5]] -> HIT\n gt_scenes: [[0, 3], [4, 9]] -> gt_trans: [[2.5, 4.5]] -> HIT\n gt_scenes: [[0, 2], [3, 9]] -> gt_trans: [[1.5, 3.5]] -> MISS\n Users should be careful about adopting these functions in any commercial matters.\n \"\"\"\n shift = n_frames_miss_tolerance / 2\n gt_scenes = gt_scenes.astype(np.float32) + np.array([[-0.5 + shift, 0.5 - shift]])\n pred_scenes = pred_scenes.astype(np.float32) + np.array([[-0.5 + shift, 0.5 - shift]])\n gt_trans = np.stack([gt_scenes[:-1, 1], gt_scenes[1:, 0]], 1)\n pred_trans = np.stack([pred_scenes[:-1, 1], pred_scenes[1:, 0]], 1)\n i, j = 0, 0\n tp, fp, fn = 0, 0, 0\n while i < len(gt_trans) or j < len(pred_trans):\n if j == len(pred_trans) or pred_trans[j, 0] > gt_trans[i, 1]:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/transnetv2_metric.py:58-80"
+ },
+ "5899": {
+ "file_id": 476,
+ "content": "This code adjusts and transforms input frame scene and transition data, and then iterates through both to calculate true positives (TP), false positives (FP), and false negatives (FN) for evaluation.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/59.json b/docs/data/59.json
new file mode 100644
index 000000000..99f444294
--- /dev/null
+++ b/docs/data/59.json
@@ -0,0 +1,549 @@
+{
+ "5900": {
+ "file_id": 476,
+ "content": " fn += 1\n i += 1\n elif i == len(gt_trans) or pred_trans[j, 1] < gt_trans[i, 0]:\n fp += 1\n j += 1\n else:\n i += 1\n j += 1\n tp += 1\n if tp + fp != 0:\n p = tp / (tp + fp)\n else:\n p = 0\n if tp + fn != 0:\n r = tp / (tp + fn)\n else:\n r = 0\n if p + r != 0:\n f1 = (p * r * 2) / (p + r)\n else:\n f1 = 0\n assert tp + fn == len(gt_trans)\n assert tp + fp == len(pred_trans)\n return p, r, f1, (tp, fp, fn)\ndef create_scene_based_summaries(one_hot_pred, one_hot_gt):\n thresholds = np.array([\n 0.02, 0.06, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\n ])\n precision, recall, f1, tp, fp, fn = np.zeros_like(thresholds), np.zeros_like(thresholds),\\\n np.zeros_like(thresholds), np.zeros_like(thresholds),\\\n np.zeros_like(thresholds), np.zeros_like(thresholds)\n gt_scenes = predictions_to_scenes(one_hot_gt)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/transnetv2_metric.py:81-120"
+ },
+ "5901": {
+ "file_id": 476,
+ "content": "This function calculates precision, recall, and F1-score for transnetv2 metric given ground truth (gt) and predicted (pred) transcript sequences. It iterates through the sequences to count true positives (tp), false negatives (fn), and false positives (fp). Afterwards, it computes precision, recall, and F1-score based on these counts. The function also asserts that the total number of true positives matches the length of gt_trans and the total number of false positives matches the length of pred_trans. It then returns the calculated metrics and the count of tp, fp, and fn. The create_scene_based_summaries function generates precision, recall, and F1-score for different thresholds using a numpy array. It initializes these metrics as well as the counts of true positives, false negatives, false positives, and false negatives to zero, then iterates over the thresholds to calculate the metric values for each threshold.",
+ "type": "comment"
+ },
+ "5902": {
+ "file_id": 476,
+ "content": " for i in range(len(thresholds)):\n pred_scenes = predictions_to_scenes(\n (one_hot_pred > thresholds[i]).astype(np.uint8)\n )\n precision[i], recall[i], f1[i], (tp[i], fp[i], fn[i]) = evaluate_scenes(gt_scenes, pred_scenes)\n best_idx = np.argmax(f1)\n return f1[best_idx]\n@METRIC.register\nclass TransNetV2Metric(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.predictions = []\n self.total_stats = {\"tp\": 0, \"fp\": 0, \"fn\": 0}\n def update(self, batch_id, data, one_hot):\n \"\"\"update metrics during each iter\n \"\"\"\n if isinstance(one_hot, tuple):\n one_hot = one_hot[0]\n one_hot = paddle.nn.functional.sigmoid(one_hot)[0]\n self.predictions.append(one_hot.numpy()[25:75])\n gt_scenes = data[1]\n is_new_file = data[2]\n if is_new_file:\n self.compute(gt_scenes)\n # preds ensemble",
+ "type": "code",
+ "location": "/paddlevideo/metrics/transnetv2_metric.py:121-152"
+ },
+ "5903": {
+ "file_id": 476,
+ "content": "This code is from the TransNetV2Metric class, which calculates metrics for a model's predictions. It iterates through different thresholds to compute precision, recall, F1 score, and true positive, false positive, and false negative counts. The update method appends predictions and computes metrics when a new file is encountered.",
+ "type": "comment"
+ },
+ "5904": {
+ "file_id": 476,
+ "content": " if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def compute(self, gt_scenes):\n predictions = np.concatenate(self.predictions, 0)[:len(frames)]\n _, _, _, (tp, fp, fn), fp_mistakes, fn_mistakes = evaluate_scenes(\n gt_scenes, predictions_to_scenes((predictions >= args.thr).astype(np.uint8)))\n self.total_stats[\"tp\"] += tp\n self.total_stats[\"fp\"] += fp\n self.total_stats[\"fn\"] += fn\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n p = self.total_stats[\"tp\"] / (self.total_stats[\"tp\"] + self.total_stats[\"fp\"])\n r = self.total_stats[\"tp\"] / (self.total_stats[\"tp\"] + self.total_stats[\"fn\"])\n f1 = (p * r * 2) / (p + r)\n logger.info('[TEST] finished, Precision= {:5.2f}, Recall= {:5.2f} , F1 Score= {:5.2f} '.format(\n p * 100, r * 100, f1 * 100))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/transnetv2_metric.py:153-174"
+ },
+ "5905": {
+ "file_id": 476,
+ "content": "The code calculates precision, recall, and F1 score for a machine learning model. It accumulates the metrics after processing all batches and logs the results using logger. It also displays the Precision, Recall, and F1 Score at the end of computation.",
+ "type": "comment"
+ },
+ "5906": {
+ "file_id": 477,
+ "content": "/paddlevideo/metrics/ucf24_utils.py",
+ "type": "filepath"
+ },
+ "5907": {
+ "file_id": 477,
+ "content": "The PaddleVideo library's UCF101 dataset offers utility functions and the Ucf24Metrics class for metric manipulation, bounding box handling, and precision/recall calculations. It computes mAP for image classification tasks and stores results per class using utility methods to read bounding box text files.",
+ "type": "summary"
+ },
+ "5908": {
+ "file_id": 477,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# Forked from: https://github.com/rafaelpadilla/Object-Detection-Metrics\n# Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br)\nimport glob\nimport os\nimport shutil\nimport sys\nfrom collections import Counter\nimport numpy as np\nfrom enum import Enum\nimport cv2\nclass MethodAveragePrecision(Enum):\n \"\"\"\n Class representing if the coordinates are relative to the\n image size or are absolute values.\n Developed by: Rafael Padilla",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:1-33"
+ },
+ "5909": {
+ "file_id": 477,
+ "content": "This code snippet is from the UCF101 dataset utility functions in the PaddleVideo library. It contains an enum class representing average precision metrics and a copyright notice with license information, original source link, and developer contact details.",
+ "type": "comment"
+ },
+ "5910": {
+ "file_id": 477,
+ "content": " Last modification: Apr 28 2018\n \"\"\"\n EveryPointInterpolation = 1\n ElevenPointInterpolation = 2\nclass CoordinatesType(Enum):\n \"\"\"\n Class representing if the coordinates are relative to the\n image size or are absolute values.\n Developed by: Rafael Padilla\n Last modification: Apr 28 2018\n \"\"\"\n Relative = 1\n Absolute = 2\nclass BBType(Enum):\n \"\"\"\n Class representing if the bounding box is groundtruth or not.\n Developed by: Rafael Padilla\n Last modification: May 24 2018\n \"\"\"\n GroundTruth = 1\n Detected = 2\nclass BBFormat(Enum):\n \"\"\"\n Class representing the format of a bounding box.\n It can be (X,Y,width,height) => XYWH\n or (X1,Y1,X2,Y2) => XYX2Y2\n Developed by: Rafael Padilla\n Last modification: May 24 2018\n \"\"\"\n XYWH = 1\n XYX2Y2 = 2\ndef convertToRelativeValues(size, box):\n dw = 1. / (size[0])\n dh = 1. / (size[1])\n cx = (box[1] + box[0]) / 2.0\n cy = (box[3] + box[2]) / 2.0\n w = box[1] - box[0]\n h = box[3] - box[2]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:34-81"
+ },
+ "5911": {
+ "file_id": 477,
+ "content": "This code defines three enumerations (CoordinatesType, BBType, and BBFormat) to represent different types of coordinates and bounding boxes. It also includes a function convertToRelativeValues that takes a size and a box as input and returns the box in relative values. The code was developed by Rafael Padilla with last modifications on April 28th for CoordinatesType, May 24th for BBType and format, and the function convertToRelativeValues is defined as well.",
+ "type": "comment"
+ },
+ "5912": {
+ "file_id": 477,
+ "content": " x = cx * dw\n y = cy * dh\n w = w * dw\n h = h * dh\n return x, y, w, h\ndef convertToAbsoluteValues(size, box):\n xIn = round(((2 * float(box[0]) - float(box[2])) * size[0] / 2))\n yIn = round(((2 * float(box[1]) - float(box[3])) * size[1] / 2))\n xEnd = xIn + round(float(box[2]) * size[0])\n yEnd = yIn + round(float(box[3]) * size[1])\n if xIn < 0:\n xIn = 0\n if yIn < 0:\n yIn = 0\n if xEnd >= size[0]:\n xEnd = size[0] - 1\n if yEnd >= size[1]:\n yEnd = size[1] - 1\n return xIn, yIn, xEnd, yEnd\ndef add_bb_into_image(image, bb, color=(255, 0, 0), thickness=2, label=None):\n r = int(color[0])\n g = int(color[1])\n b = int(color[2])\n font = cv2.FONT_HERSHEY_SIMPLEX\n fontScale = 0.5\n fontThickness = 1\n x1, y1, x2, y2 = bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2)\n x1 = int(x1)\n y1 = int(y1)\n x2 = int(x2)\n y2 = int(y2)\n cv2.rectangle(image, (x1, y1), (x2, y2), (b, g, r), thickness)\n # Add label\n if label is not None:\n # Get size of the text box",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:82-122"
+ },
+ "5913": {
+ "file_id": 477,
+ "content": "Function `ucf24_utils.py:81-121` defines a function `convertToAbsoluteValues` which takes in the size and bounding box coordinates (x, y, w, h) as input and returns absolute values for xIn, yIn, xEnd, yEnd considering the image size. If any of these values fall outside the image boundaries, they are adjusted to the last valid pixel within the image.\nThis code also includes a function `add_bb_into_image` which adds a bounding box with given coordinates (x1, y1, x2, y2) and label on the image using OpenCV's rectangle() function and font() function for adding labels to the bounding boxes.",
+ "type": "comment"
+ },
+ "5914": {
+ "file_id": 477,
+ "content": " (tw, th) = cv2.getTextSize(label, font, fontScale, fontThickness)[0]\n # Top-left coord of the textbox\n (xin_bb, yin_bb) = (x1 + thickness, y1 - th + int(12.5 * fontScale))\n # Checking position of the text top-left (outside or inside the bb)\n if yin_bb - th <= 0: # if outside the image\n yin_bb = y1 + th # put it inside the bb\n r_Xin = x1 - int(thickness / 2)\n r_Yin = y1 - th - int(thickness / 2)\n # Draw filled rectangle to put the text in it\n cv2.rectangle(image, (r_Xin, r_Yin - thickness),\n (r_Xin + tw + thickness * 3, r_Yin + th + int(12.5 * fontScale)), (b, g, r),\n -1)\n cv2.putText(image, label, (xin_bb, yin_bb), font, fontScale, (0, 0, 0), fontThickness,\n cv2.LINE_AA)\n return image\nclass BoundingBox:\n def __init__(self,\n imageName,\n classId,\n x,\n y,\n w,\n h,\n typeCoordinates=None,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:123-148"
+ },
+ "5915": {
+ "file_id": 477,
+ "content": "This function calculates the text box coordinates and draws a rectangle around it, then adds text within the rectangle. The text position is adjusted if it's outside the image area. It also initializes a class for bounding boxes with properties like image name, class ID, and coordinates.",
+ "type": "comment"
+ },
+ "5916": {
+ "file_id": 477,
+ "content": " imgSize=None,\n bbType=None,\n classConfidence=None,\n format=None):\n \"\"\"Constructor.\n Args:\n imageName: String representing the image name.\n classId: String value representing class id.\n x: Float value representing the X upper-left coordinate of the bounding box.\n y: Float value representing the Y upper-left coordinate of the bounding box.\n w: Float value representing the width bounding box.\n h: Float value representing the height bounding box.\n typeCoordinates: (optional) Enum (Relative or Absolute) represents if the bounding box\n coordinates (x,y,w,h) are absolute or relative to size of the image. Default:'Absolute'.\n imgSize: (optional) 2D vector (width, height)=>(int, int) represents the size of the\n image of the bounding box. If typeCoordinates is 'Relative', imgSize is required.\n bbType: (optional) Enum (Groundtruth or Detection) identifies if the bounding box",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:149-165"
+ },
+ "5917": {
+ "file_id": 477,
+ "content": "This code snippet defines a constructor for the class Ucf24Metrics, which takes parameters like image name, class id, bounding box coordinates (x, y, w, h), and type of bounding box coordinates. It also accepts optional arguments such as imgSize, bbType, classConfidence, and format. If typeCoordinates is 'Relative', then imgSize is required. The constructor initializes an object representing a metric for UCF101 dataset.",
+ "type": "comment"
+ },
+ "5918": {
+ "file_id": 477,
+ "content": " represents a ground truth or a detection. If it is a detection, the classConfidence has\n to be informed.\n classConfidence: (optional) Float value representing the confidence of the detected\n class. If detectionType is Detection, classConfidence needs to be informed.\n format: (optional) Enum (BBFormat.XYWH or BBFormat.XYX2Y2) indicating the format of the\n coordinates of the bounding boxes. BBFormat.XYWH: \n BBFormat.XYX2Y2: .\n \"\"\"\n self._imageName = imageName\n self._typeCoordinates = typeCoordinates\n if typeCoordinates == CoordinatesType.Relative and imgSize is None:\n raise IOError(\n 'Parameter \\'imgSize\\' is required. It is necessary to inform the image size.')\n if bbType == BBType.Detected and classConfidence is None:\n raise IOError(\n 'For bbType=\\'Detection\\', it is necessary to inform the classConfidence value.')",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:166-181"
+ },
+ "5919": {
+ "file_id": 477,
+ "content": "This code defines a class with properties: imageName, typeCoordinates (Relative or Absolute), imgSize (image size required if typeCoordinates is Relative), bbType (Ground Truth or Detection), and classConfidence (optional for Detection). It also includes error checks for mandatory parameters (imgSize for Relative typeCoordinates and classConfidence for Detection bbType).",
+ "type": "comment"
+ },
+ "5920": {
+ "file_id": 477,
+ "content": " self._classConfidence = classConfidence\n self._bbType = bbType\n self._classId = classId\n self._format = format\n # If relative coordinates, convert to absolute values\n # For relative coords: (x,y,w,h)=(X_center/img_width , Y_center/img_height)\n if typeCoordinates == CoordinatesType.Relative:\n (self._x, self._y, self._w, self._h) = convertToAbsoluteValues(imgSize, (x, y, w, h))\n self._width_img = imgSize[0]\n self._height_img = imgSize[1]\n if format == BBFormat.XYWH:\n self._x2 = self._w\n self._y2 = self._h\n self._w = self._x2 - self._x\n self._h = self._y2 - self._y\n else:\n raise IOError(\n 'For relative coordinates, the format must be XYWH (x,y,width,height)')\n # For absolute coords: (x,y,w,h)=real bb coords\n else:\n self._x = x\n self._y = y\n if format == BBFormat.XYWH:\n self._w = w",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:183-207"
+ },
+ "5921": {
+ "file_id": 477,
+ "content": "This function converts relative bounding box coordinates to absolute values and assigns them to the object. If the given format is XYWH, it adjusts the width and height accordingly. For absolute coordinates, it directly assigns the provided values. If the format does not match XYWH for relative coordinates, an IOError is raised.",
+ "type": "comment"
+ },
+ "5922": {
+ "file_id": 477,
+ "content": " self._h = h\n self._x2 = self._x + self._w\n self._y2 = self._y + self._h\n else: # format == BBFormat.XYX2Y2: .\n self._x2 = w\n self._y2 = h\n self._w = self._x2 - self._x\n self._h = self._y2 - self._y\n if imgSize is None:\n self._width_img = None\n self._height_img = None\n else:\n self._width_img = imgSize[0]\n self._height_img = imgSize[1]\n def getAbsoluteBoundingBox(self, format=None):\n if format == BBFormat.XYWH:\n return self._x, self._y, self._w, self._h\n elif format == BBFormat.XYX2Y2:\n return self._x, self._y, self._x2, self._y2\n def getRelativeBoundingBox(self, imgSize=None):\n if imgSize is None and self._width_img is None and self._height_img is None:\n raise IOError(\n 'Parameter \\'imgSize\\' is required. It is necessary to inform the image size.')",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:208-232"
+ },
+ "5923": {
+ "file_id": 477,
+ "content": "The code defines a class with methods to handle bounding box formats. It supports two formats: XYWH and XYX2Y2. The constructor initializes the bounding box dimensions and image size if provided. The getAbsoluteBoundingBox method returns the bounding box coordinates based on the format specified. If no image size is available, getRelativeBoundingBox requires the imgSize parameter to determine the absolute position of the bounding box.",
+ "type": "comment"
+ },
+ "5924": {
+ "file_id": 477,
+ "content": " if imgSize is None:\n return convertToRelativeValues((imgSize[0], imgSize[1]),\n (self._x, self._y, self._w, self._h))\n else:\n return convertToRelativeValues((self._width_img, self._height_img),\n (self._x, self._y, self._w, self._h))\n def getImageName(self):\n return self._imageName\n def getConfidence(self):\n return self._classConfidence\n def getFormat(self):\n return self._format\n def getClassId(self):\n return self._classId\n def getImageSize(self):\n return self._width_img, self._height_img\n def getCoordinatesType(self):\n return self._typeCoordinates\n def getBBType(self):\n return self._bbType\n @staticmethod\n def compare(det1, det2):\n det1BB = det1.getAbsoluteBoundingBox(format=BBFormat.XYWH)\n det1ImgSize = det1.getImageSize()\n det2BB = det2.getAbsoluteBoundingBox(format=BBFormat.XYWH)\n det2ImgSize = det2.getImageSize()",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:233-266"
+ },
+ "5925": {
+ "file_id": 477,
+ "content": "This code defines a class with various getter methods to access different attributes of the detection result. The class also contains a static method compare() that takes two detections as input and compares them using absolute bounding boxes and image sizes.",
+ "type": "comment"
+ },
+ "5926": {
+ "file_id": 477,
+ "content": " if det1.getClassId() == det2.getClassId() and \\\n det1.classConfidence == det2.classConfidenc() and \\\n det1BB[0] == det2BB[0] and \\\n det1BB[1] == det2BB[1] and \\\n det1BB[2] == det2BB[2] and \\\n det1BB[3] == det2BB[3] and \\\n det1ImgSize[0] == det1ImgSize[0] and \\\n det2ImgSize[1] == det2ImgSize[1]:\n return True\n return False\n @staticmethod\n def clone(boundingBox):\n absBB = boundingBox.getAbsoluteBoundingBox(format=BBFormat.XYWH)\n newBoundingBox = BoundingBox(\n boundingBox.getImageName(),\n boundingBox.getClassId(),\n absBB[0],\n absBB[1],\n absBB[2],\n absBB[3],\n typeCoordinates=boundingBox.getCoordinatesType(),\n imgSize=boundingBox.getImageSize(),\n bbType=boundingBox.getBBType(),\n classConfidence=boundingBox.getConfidence(),\n format=BBFormat.XYWH)\n return newBoundingBox",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:268-294"
+ },
+ "5927": {
+ "file_id": 477,
+ "content": "The code snippet compares two bounding boxes to check if they match by comparing their class IDs, coordinates, and image sizes. If the conditions are met, it returns True; otherwise, False. The static method `clone` creates a new bounding box with the same properties as an existing one, allowing for easy cloning of bounding boxes.",
+ "type": "comment"
+ },
+ "5928": {
+ "file_id": 477,
+ "content": "class BoundingBoxes:\n def __init__(self):\n self._boundingBoxes = []\n def addBoundingBox(self, bb):\n self._boundingBoxes.append(bb)\n def removeBoundingBox(self, _boundingBox):\n for d in self._boundingBoxes:\n if BoundingBox.compare(d, _boundingBox):\n del self._boundingBoxes[d]\n return\n def removeAllBoundingBoxes(self):\n self._boundingBoxes = []\n def getBoundingBoxes(self):\n return self._boundingBoxes\n def getBoundingBoxByClass(self, classId):\n boundingBoxes = []\n for d in self._boundingBoxes:\n if d.getClassId() == classId: # get only specified bounding box type\n boundingBoxes.append(d)\n return boundingBoxes\n def getClasses(self):\n classes = []\n for d in self._boundingBoxes:\n c = d.getClassId()\n if c not in classes:\n classes.append(c)\n return classes\n def getBoundingBoxesByType(self, bbType):\n # get only specified bb type",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:297-332"
+ },
+ "5929": {
+ "file_id": 477,
+ "content": "This class represents a collection of bounding boxes with methods to add, remove, and retrieve bounding boxes based on their type or class. It also provides functionality to retrieve all classes present in the bounding boxes.",
+ "type": "comment"
+ },
+ "5930": {
+ "file_id": 477,
+ "content": " return [d for d in self._boundingBoxes if d.getBBType() == bbType]\n def getBoundingBoxesByImageName(self, imageName):\n # get only specified bb type\n return [d for d in self._boundingBoxes if d.getImageName() == imageName]\n def count(self, bbType=None):\n if bbType is None: # Return all bounding boxes\n return len(self._boundingBoxes)\n count = 0\n for d in self._boundingBoxes:\n if d.getBBType() == bbType: # get only specified bb type\n count += 1\n return count\n def clone(self):\n newBoundingBoxes = BoundingBoxes()\n for d in self._boundingBoxes:\n det = BoundingBox.clone(d)\n newBoundingBoxes.addBoundingBox(det)\n return newBoundingBoxes\n def drawAllBoundingBoxes(self, image, imageName):\n bbxes = self.getBoundingBoxesByImageName(imageName)\n for bb in bbxes:\n if bb.getBBType() == BBType.GroundTruth: # if ground truth\n image = add_bb_into_image(image, bb, color=(0, 255, 0)) # green",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:333-359"
+ },
+ "5931": {
+ "file_id": 477,
+ "content": "Function `getBoundingBoxesByBBType` returns a list of bounding boxes with the specified BB type.\nFunction `getBoundingBoxesByImageName` returns a list of bounding boxes for the given image name.\nMethod `count` counts and returns the number of bounding boxes with the specified BB type, or all bounding boxes if no type is provided.\nMethod `clone` creates a new instance of BoundingBoxes and adds clones of each bounding box from the original instance.\nFunction `drawAllBoundingBoxes` draws all bounding boxes for the given image name on the specified image, only ground truth bounding boxes are drawn in green color.",
+ "type": "comment"
+ },
+ "5932": {
+ "file_id": 477,
+ "content": " else: # if detection\n image = add_bb_into_image(image, bb, color=(255, 0, 0)) # red\n return image\nclass Evaluator:\n def GetPascalVOCMetrics(self,\n boundingboxes,\n IOUThreshold=0.5,\n method=None):\n \"\"\"Get the metrics used by the VOC Pascal 2012 challenge.\n Get\n Args:\n boundingboxes: Object of the class BoundingBoxes representing ground truth and detected\n bounding boxes;\n IOUThreshold: IOU threshold indicating which detections will be considered TP or FP\n (default value = 0.5);\n method (default = EveryPointInterpolation): It can be calculated as the implementation\n in the official PASCAL VOC toolkit (EveryPointInterpolation), or applying the 11-point\n interpolatio as described in the paper \"The PASCAL Visual Object Classes(VOC) Challenge\"\n or EveryPointInterpolation\" (ElevenPointInterpolation);",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:360-380"
+ },
+ "5933": {
+ "file_id": 477,
+ "content": "The code defines a function `GetPascalVOCMetrics` within the `Evaluator` class to calculate metrics for Pascal VOC Challenge. It takes `boundingboxes`, `IOUThreshold`, and `method` as input parameters. The method can be set as `EveryPointInterpolation` or `ElevenPointInterpolation`. This function calculates precision, recall, F1 score, and AP metric using the provided parameters for Pascal VOC Challenge evaluation.",
+ "type": "comment"
+ },
+ "5934": {
+ "file_id": 477,
+ "content": " Returns:\n A list of dictionaries. Each dictionary contains information and metrics of each class.\n The keys of each dictionary are:\n dict['class']: class representing the current dictionary;\n dict['precision']: array with the precision values;\n dict['recall']: array with the recall values;\n dict['AP']: average precision;\n dict['interpolated precision']: interpolated precision values;\n dict['interpolated recall']: interpolated recall values;\n dict['total positives']: total number of ground truth positives;\n dict['total TP']: total number of True Positive detections;\n dict['total FP']: total number of False Negative detections;\n \"\"\"\n ret = [] # list containing metrics (precision, recall, average precision) of each class\n # List with all ground truths (Ex: [imageName,class,confidence=1, (bb coordinates XYX2Y2)])\n groundTruths = []\n # List with all detections (Ex: [imageName,class,confidence,(bb coordinates XYX2Y2)])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:381-397"
+ },
+ "5935": {
+ "file_id": 477,
+ "content": "The function returns a list of dictionaries, each containing information and metrics of each class. The keys include class representation, precision values, recall values, average precision, interpolated precision, interpolated recall, total positives, total true positives, and total false positives. It initializes an empty list \"ret\" to store the metrics for each class, as well as groundTruths and detection lists.",
+ "type": "comment"
+ },
+ "5936": {
+ "file_id": 477,
+ "content": " detections = []\n # Get all classes\n classes = []\n # Loop through all bounding boxes and separate them into GTs and detections\n for bb in boundingboxes.getBoundingBoxes():\n # [imageName, class, confidence, (bb coordinates XYX2Y2)]\n if bb.getBBType() == BBType.GroundTruth:\n groundTruths.append([\n bb.getImageName(),\n bb.getClassId(), 1,\n bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2)\n ])\n else:\n detections.append([\n bb.getImageName(),\n bb.getClassId(),\n bb.getConfidence(),\n bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2)\n ])\n # get class\n if bb.getClassId() not in classes:\n classes.append(bb.getClassId())\n classes = sorted(classes)\n # Precision x Recall is obtained individually by each class\n # Loop through by classes",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:398-422"
+ },
+ "5937": {
+ "file_id": 477,
+ "content": "The code initializes empty lists for detections and classes, then iterates through all bounding boxes. It separates ground truth (GT) bounding boxes from detections, appending them to their respective lists with additional information such as image name, class ID, confidence, and bounding box coordinates. It also keeps track of unique classes and sorts them. The code will then use these lists and sorted classes for precision-recall calculations by individual classes.",
+ "type": "comment"
+ },
+ "5938": {
+ "file_id": 477,
+ "content": " for c in classes:\n # Get only detection of class c\n dects = []\n [dects.append(d) for d in detections if d[1] == c]\n # Get only ground truths of class c\n gts = []\n [gts.append(g) for g in groundTruths if g[1] == c]\n npos = len(gts)\n # sort detections by decreasing confidence\n dects = sorted(dects, key=lambda conf: conf[2], reverse=True)\n TP = np.zeros(len(dects))\n FP = np.zeros(len(dects))\n # create dictionary with amount of gts for each image\n det = Counter([cc[0] for cc in gts])\n for key, val in det.items():\n det[key] = np.zeros(val)\n # Loop through detections\n for d in range(len(dects)):\n # Find ground truth image\n gt = [gt for gt in gts if gt[0] == dects[d][0]]\n iouMax = sys.float_info.min\n for j in range(len(gt)):\n iou = Evaluator.iou(dects[d][3], gt[j][3])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:423-445"
+ },
+ "5939": {
+ "file_id": 477,
+ "content": "Iterating through classes, the code collects detections and ground truths for each class. It then calculates the number of positive ground truths (npos), sorts detections by confidence level, and initializes True Positive (TP) and False Positive (FP) arrays. The code creates a dictionary to store the amount of ground truths per image, and iterates through detections to find corresponding ground truth images, calculating Intersection over Union (IoU) between detection and ground truth bounding boxes.",
+ "type": "comment"
+ },
+ "5940": {
+ "file_id": 477,
+ "content": " if iou > iouMax:\n iouMax = iou\n jmax = j\n # Assign detection as true positive/don't care/false positive\n if iouMax >= IOUThreshold:\n if det[dects[d][0]][jmax] == 0:\n TP[d] = 1 # count as true positive\n det[dects[d][0]][jmax] = 1 # flag as already 'seen'\n else:\n FP[d] = 1 # count as false positive\n # - A detected \"cat\" is overlaped with a GT \"cat\" with IOU >= IOUThreshold.\n else:\n FP[d] = 1 # count as false positive\n # compute precision, recall and average precision\n acc_FP = np.cumsum(FP)\n acc_TP = np.cumsum(TP)\n rec = acc_TP / npos\n prec = np.divide(acc_TP, (acc_FP + acc_TP))\n # Depending on the method, call the right implementation\n if method == MethodAveragePrecision.EveryPointInterpolation:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:446-465"
+ },
+ "5941": {
+ "file_id": 477,
+ "content": "This code calculates true positives, false positives, and computes precision, recall, and average precision. It checks if a detected object overlaps with ground truth objects using IOU threshold. If the overlap is within the threshold, it counts as a true positive or false positive depending on whether the object has already been marked 'seen'. Finally, based on the method chosen (EveryPointInterpolation in this case), it calls the appropriate average precision calculation function.",
+ "type": "comment"
+ },
+ "5942": {
+ "file_id": 477,
+ "content": " [ap, mpre, mrec, ii] = Evaluator.CalculateAveragePrecision(rec, prec)\n else:\n [ap, mpre, mrec, _] = Evaluator.ElevenPointInterpolatedAP(rec, prec)\n # add class result in the dictionary to be returned\n r = {\n 'class': c,\n 'precision': prec,\n 'recall': rec,\n 'AP': ap,\n 'interpolated precision': mpre,\n 'interpolated recall': mrec,\n 'total positives': npos,\n 'total TP': np.sum(TP),\n 'total FP': np.sum(FP)\n }\n ret.append(r)\n return ret\n @staticmethod\n def CalculateAveragePrecision(rec, prec):\n mrec = [0]\n [mrec.append(e) for e in rec]\n mrec.append(1)\n mpre = [0]\n [mpre.append(e) for e in prec]\n mpre.append(0)\n for i in range(len(mpre) - 1, 0, -1):\n mpre[i - 1] = max(mpre[i - 1], mpre[i])\n ii = []\n for i in range(len(mrec) - 1):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:466-495"
+ },
+ "5943": {
+ "file_id": 477,
+ "content": "Calculates average precision for each class using CalculateAveragePrecision or ElevenPointInterpolatedAP depending on the input. Appends the results to a dictionary, then adds the dictionary to a list and returns it.",
+ "type": "comment"
+ },
+ "5944": {
+ "file_id": 477,
+ "content": " if mrec[1:][i] != mrec[0:-1][i]:\n ii.append(i + 1)\n ap = 0\n for i in ii:\n ap = ap + np.sum((mrec[i] - mrec[i - 1]) * mpre[i])\n return [ap, mpre[0:len(mpre) - 1], mrec[0:len(mpre) - 1], ii]\n @staticmethod\n # 11-point interpolated average precision\n def ElevenPointInterpolatedAP(rec, prec):\n mrec = []\n [mrec.append(e) for e in rec]\n mpre = []\n [mpre.append(e) for e in prec]\n recallValues = np.linspace(0, 1, 11)\n recallValues = list(recallValues[::-1])\n rhoInterp = []\n recallValid = []\n for r in recallValues:\n # Obtain all recall values higher or equal than r\n argGreaterRecalls = np.argwhere(mrec[:] >= r)\n pmax = 0\n # If there are recalls above r\n if argGreaterRecalls.size != 0:\n pmax = max(mpre[argGreaterRecalls.min():])\n recallValid.append(r)\n rhoInterp.append(pmax)\n # By definition AP = sum(max(precision whose recall is above r))/11",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:496-523"
+ },
+ "5945": {
+ "file_id": 477,
+ "content": "The code calculates the 11-point interpolated average precision (AP) between recall and precision values. It first appends recall and precision lists in reverse order, then creates a list of recall values from 0 to 1 in reverse order. Next, it iterates over these recall values, finding all recall values greater than or equal to the current value and selecting the maximum precision at that index. Finally, it returns the interpolated AP by summing the maximum precisions for each recall value and dividing by 11. The resulting AP values are stored in a list along with the original recall and precision lists, as well as an indicator list of indices where the recall values were greater than or equal to the current recall value.",
+ "type": "comment"
+ },
+ "5946": {
+ "file_id": 477,
+ "content": " ap = sum(rhoInterp) / 11\n # Generating values for the plot\n rvals = [recallValid[0]]\n [rvals.append(e) for e in recallValid]\n rvals.append(0)\n pvals = [0]\n [pvals.append(e) for e in rhoInterp]\n pvals.append(0)\n # rhoInterp = rhoInterp[::-1]\n cc = []\n for i in range(len(rvals)):\n p = (rvals[i], pvals[i - 1])\n if p not in cc:\n cc.append(p)\n p = (rvals[i], pvals[i])\n if p not in cc:\n cc.append(p)\n recallValues = [i[0] for i in cc]\n rhoInterp = [i[1] for i in cc]\n return [ap, rhoInterp, recallValues, None]\n # For each detections, calculate IOU with reference\n @staticmethod\n def _getAllIOUs(reference, detections):\n ret = []\n bbReference = reference.getAbsoluteBoundingBox(BBFormat.XYX2Y2)\n # img = np.zeros((200,200,3), np.uint8)\n for d in detections:\n bb = d.getAbsoluteBoundingBox(BBFormat.XYX2Y2)\n iou = Evaluator.iou(bbReference, bb)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:524-553"
+ },
+ "5947": {
+ "file_id": 477,
+ "content": "The code calculates average precision (AP) and Area Under Curve (AUC), then generates recall and precision values for a plot. It also defines a method to calculate the Intersection over Union (IoU) between reference and detection bounding boxes.",
+ "type": "comment"
+ },
+ "5948": {
+ "file_id": 477,
+ "content": " ret.append((iou, reference, d)) # iou, reference, detection\n return sorted(ret, key=lambda i: i[0], reverse=True) # sort by iou (from highest to lowest)\n @staticmethod\n def iou(boxA, boxB):\n # if boxes dont intersect\n if Evaluator._boxesIntersect(boxA, boxB) is False:\n return 0\n interArea = Evaluator._getIntersectionArea(boxA, boxB)\n union = Evaluator._getUnionAreas(boxA, boxB, interArea=interArea)\n # intersection over union\n iou = interArea / union\n assert iou >= 0\n return iou\n @staticmethod\n def _boxesIntersect(boxA, boxB):\n if boxA[0] > boxB[2]:\n return False # boxA is right of boxB\n if boxB[0] > boxA[2]:\n return False # boxA is left of boxB\n if boxA[3] < boxB[1]:\n return False # boxA is above boxB\n if boxA[1] > boxB[3]:\n return False # boxA is below boxB\n return True\n @staticmethod\n def _getIntersectionArea(boxA, boxB):\n xA = max(boxA[0], boxB[0])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:554-583"
+ },
+ "5949": {
+ "file_id": 477,
+ "content": "The code calculates the IoU (intersection over union) between two bounding boxes, and returns a list of detection results sorted by IoU in descending order. It also includes utility methods to check if two boxes intersect and calculate the intersection area.",
+ "type": "comment"
+ },
+ "5950": {
+ "file_id": 477,
+ "content": " yA = max(boxA[1], boxB[1])\n xB = min(boxA[2], boxB[2])\n yB = min(boxA[3], boxB[3])\n # intersection area\n return (xB - xA + 1) * (yB - yA + 1)\n @staticmethod\n def _getUnionAreas(boxA, boxB, interArea=None):\n area_A = Evaluator._getArea(boxA)\n area_B = Evaluator._getArea(boxB)\n if interArea is None:\n interArea = Evaluator._getIntersectionArea(boxA, boxB)\n return float(area_A + area_B - interArea)\n @staticmethod\n def _getArea(box):\n return (box[2] - box[0] + 1) * (box[3] - box[1] + 1)\n# Validate formats\ndef ValidateFormats(argFormat, argName, errors):\n if argFormat == 'xywh':\n return BBFormat.XYWH\n elif argFormat == 'xyrb':\n return BBFormat.XYX2Y2\n elif argFormat is None:\n return BBFormat.XYWH # default when nothing is passed\n else:\n errors.append(\n 'argument %s: invalid value. It must be either \\'xywh\\' or \\'xyrb\\'' % argName)\n# Validate mandatory args\ndef ValidateMandatoryArgs(arg, argName, errors):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:584-617"
+ },
+ "5951": {
+ "file_id": 477,
+ "content": "This code contains functions to calculate intersection and union areas of two bounding boxes, and two validation functions for argument formats and mandatory arguments. The ValidateFormats function checks if the format is 'xywh', 'xyrb' or None (default) and returns a corresponding BBFormat type. The ValidateMandatoryArgs function checks if an argument exists and appends an error message to 'errors' if it doesn't meet the requirements.",
+ "type": "comment"
+ },
+ "5952": {
+ "file_id": 477,
+ "content": " if arg is None:\n errors.append('argument %s: required argument' % argName)\n else:\n return True\ndef ValidateImageSize(arg, argName, argInformed, errors):\n errorMsg = 'argument %s: required argument if %s is relative' % (argName, argInformed)\n ret = None\n if arg is None:\n errors.append(errorMsg)\n else:\n arg = arg.replace('(', '').replace(')', '')\n args = arg.split(',')\n if len(args) != 2:\n errors.append(\n '%s. It must be in the format \\'width,height\\' (e.g. \\'600,400\\')' % errorMsg)\n else:\n if not args[0].isdigit() or not args[1].isdigit():\n errors.append(\n '%s. It must be in INdiaTEGER the format \\'width,height\\' (e.g. \\'600,400\\')' %\n errorMsg)\n else:\n ret = (int(args[0]), int(args[1]))\n return ret\n# Validate coordinate types\ndef ValidateCoordinatesTypes(arg, argName, errors):\n if arg == 'abs':\n return CoordinatesType.Absolute",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:618-648"
+ },
+ "5953": {
+ "file_id": 477,
+ "content": "This code defines a function ValidateImageSize that checks if the image size argument is valid. It appends error messages to the errors list if the argument is missing or not in the correct format 'width,height'. The function also handles the case where the argument is relative and requires both width and height to be integers. Finally, it returns a tuple of (width, height) if valid. Additionally, there's a ValidateCoordinatesTypes function that checks if the coordinate type argument is valid and returns the CoordinatesType.Absolute if 'abs'.",
+ "type": "comment"
+ },
+ "5954": {
+ "file_id": 477,
+ "content": " elif arg == 'rel':\n return CoordinatesType.Relative\n elif arg is None:\n return CoordinatesType.Absolute # default when nothing is passed\n errors.append('argument %s: invalid value. It must be either \\'rel\\' or \\'abs\\'' % argName)\ndef getBoundingBoxes(directory,\n isGT,\n bbFormat,\n coordType,\n allBoundingBoxes=None,\n allClasses=None,\n imgSize=(0, 0)):\n \"\"\"Read txt files containing bounding boxes (ground truth and detections).\"\"\"\n print(directory)\n if allBoundingBoxes is None:\n allBoundingBoxes = BoundingBoxes()\n if allClasses is None:\n allClasses = []\n # Read ground truths\n os.chdir(directory)\n files = glob.glob(\"*.txt\")\n files.sort()\n for f in files:\n nameOfImage = f.replace(\".txt\", \"\")\n fh1 = open(f, \"r\")\n for line in fh1:\n line = line.replace(\"\\n\", \"\")\n if line.replace(' ', '') == '':\n continue",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:649-680"
+ },
+ "5955": {
+ "file_id": 477,
+ "content": "This code reads text files containing bounding boxes (ground truth and detections). It handles 'relative' or 'absolute' coordinates, and checks for invalid arguments. The function takes a directory, image format, and coordinate type as inputs, and returns bounding boxes and classes. If allBoundingBoxes or allClasses are None, it initializes them. It changes the working directory to the specified directory and reads all files in alphabetical order.",
+ "type": "comment"
+ },
+ "5956": {
+ "file_id": 477,
+ "content": " splitLine = line.split(\" \")\n if isGT:\n idClass = (splitLine[0]) # class\n x = float(splitLine[1])\n y = float(splitLine[2])\n w = float(splitLine[3])\n h = float(splitLine[4])\n bb = BoundingBox(\n nameOfImage,\n idClass,\n x,\n y,\n w,\n h,\n coordType,\n imgSize,\n BBType.GroundTruth,\n format=bbFormat)\n else:\n idClass = (splitLine[0]) # class\n confidence = float(splitLine[1])\n x = float(splitLine[2])\n y = float(splitLine[3])\n w = float(splitLine[4])\n h = float(splitLine[5])\n bb = BoundingBox(\n nameOfImage,\n idClass,\n x,\n y,\n w,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:681-711"
+ },
+ "5957": {
+ "file_id": 477,
+ "content": "This code reads a line of text and determines whether it represents ground truth or predicted bounding boxes. It then initializes BoundingBox objects with the appropriate attributes based on the type (ground truth or prediction) and stores them accordingly.",
+ "type": "comment"
+ },
+ "5958": {
+ "file_id": 477,
+ "content": " h,\n coordType,\n imgSize,\n BBType.Detected,\n confidence,\n format=bbFormat)\n allBoundingBoxes.addBoundingBox(bb)\n if idClass not in allClasses:\n allClasses.append(idClass)\n fh1.close()\n return allBoundingBoxes, allClasses\ndef get_mAP(gtFolder, detFolder, threshold=0.5, savePath=None):\n gtFormat = 'xyrb'\n detFormat = 'xyrb'\n gtCoordinates = 'abs'\n detCoordinates = 'abs'\n gtFolder = os.path.join(os.path.abspath('.'), gtFolder)\n detFolder = os.path.join(os.path.abspath('.'), detFolder)\n iouThreshold = threshold\n # Arguments validation\n errors = []\n # Validate formats\n gtFormat = ValidateFormats(gtFormat, 'gtFormat', errors)\n detFormat = ValidateFormats(detFormat, '-detformat', errors)\n # Coordinates types\n gtCoordType = ValidateCoordinatesTypes(gtCoordinates, '-gtCoordinates', errors)\n detCoordType = ValidateCoordinatesTypes(detCoordinates, '-detCoordinates', errors)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:712-743"
+ },
+ "5959": {
+ "file_id": 477,
+ "content": "The code defines a function to calculate the mean average precision (mAP) between ground truth and detected objects in image classification tasks. It takes input folders containing ground truth and detection results, adjustable threshold for determining true positives, and an optional save path for the output file. The code performs argument validation to ensure correct formats and coordinate types.",
+ "type": "comment"
+ },
+ "5960": {
+ "file_id": 477,
+ "content": " imgSize = (0, 0)\n # Create directory to save results\n shutil.rmtree(savePath, ignore_errors=True) # Clear folder\n if savePath is not None:\n os.makedirs(savePath)\n # Get groundtruth boxes\n allBoundingBoxes, allClasses = getBoundingBoxes(\n gtFolder, True, gtFormat, gtCoordType, imgSize=imgSize)\n # Get detected boxes\n allBoundingBoxes, allClasses = getBoundingBoxes(\n detFolder, False, detFormat, detCoordType, allBoundingBoxes, allClasses, imgSize=imgSize)\n allClasses.sort()\n evaluator = Evaluator()\n acc_AP = 0\n validClasses = 0\n # Plot Precision x Recall curve\n detections = evaluator.GetPascalVOCMetrics(allBoundingBoxes, iouThreshold,\n method=MethodAveragePrecision.EveryPointInterpolation)\n # each detection is a class and store AP and mAP results in AP_res list\n AP_res = []\n for metricsPerClass in detections:\n # Get metric values per each class\n cl = metricsPerClass['class']\n ap = metricsPerClass['AP']",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:744-772"
+ },
+ "5961": {
+ "file_id": 477,
+ "content": "This code is creating a directory to save results, clearing any previous content in the folder. It then retrieves ground truth and detected bounding boxes, sorts classes, initializes an evaluator object, and calculates average precision (AP) for each class, storing the AP and mean AP results in the AP_res list.",
+ "type": "comment"
+ },
+ "5962": {
+ "file_id": 477,
+ "content": " totalPositives = metricsPerClass['total positives']\n if totalPositives > 0:\n validClasses = validClasses + 1\n acc_AP = acc_AP + ap\n ap_str = \"{0:.2f}%\".format(ap * 100)\n AP_res.append('AP: %s (%s)' % (ap_str, cl))\n mAP = acc_AP / validClasses\n mAP_str = \"{0:.2f}%\".format(mAP * 100)\n AP_res.append('mAP: %s' % mAP_str)\n return AP_res",
+ "type": "code",
+ "location": "/paddlevideo/metrics/ucf24_utils.py:773-783"
+ },
+ "5963": {
+ "file_id": 477,
+ "content": "This code calculates mean Average Precision (mAP) for each class and returns it as a list. It iterates through valid classes, calculates Average Precision (AP) for each class if there are positive samples, updates mAP by averaging APs of all valid classes, and appends AP and class labels to the result list. The final mAP value is also formatted and added to the result list before returning it.",
+ "type": "comment"
+ },
+ "5964": {
+ "file_id": 478,
+ "content": "/paddlevideo/metrics/vos_metric.py",
+ "type": "filepath"
+ },
+ "5965": {
+ "file_id": 478,
+ "content": "The PaddleVideo framework's VOSMetric class is responsible for video object segmentation tasks, data processing, and model preparation. It includes methods 'flip_tensor', 'save_mask', and manages various operations such as handling failures and logging data.",
+ "type": "summary"
+ },
+ "5966": {
+ "file_id": 478,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport os\nimport paddle\nimport zipfile\nimport time\nfrom PIL import Image\nfrom paddle.io import DataLoader\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass VOSMetric(BaseMetric):\n def __init__(self,\n data_size,\n batch_size,\n result_root,\n zip_dir,\n log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:1-38"
+ },
+ "5967": {
+ "file_id": 478,
+ "content": "This code is part of the PaddleVideo framework, implementing the VOSMetric class. It registers a metric for video object segmentation tasks using the PaddlePaddle library. The class takes parameters such as data size, batch size, result root directory, and zip directory for results storage.",
+ "type": "comment"
+ },
+ "5968": {
+ "file_id": 478,
+ "content": " self.video_num = 0\n self.total_time = 0\n self.total_frame = 0\n self.total_sfps = 0\n self.total_video_num = data_size\n self.count = 0\n self.result_root = result_root\n self.zip_dir = zip_dir\n def update(self, batch_id, data, model):\n \"\"\"update metrics during each iter\n \"\"\"\n self.video_num += 1\n seq_dataset = data\n seq_name = seq_dataset.seq_name\n logger.info('Prcessing Seq {} [{}/{}]:'.format(seq_name, self.video_num,\n self.total_video_num))\n seq_dataloader = DataLoader(seq_dataset,\n return_list=True,\n batch_size=1,\n shuffle=False,\n num_workers=0)\n seq_total_time = 0\n seq_total_frame = 0\n ref_embeddings = []\n ref_masks = []\n prev_embedding = []\n prev_mask = []\n with paddle.no_grad():",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:39-68"
+ },
+ "5969": {
+ "file_id": 478,
+ "content": "This code initializes a VOS metric class with parameters such as total_video_num, result_root and zip_dir. The update method processes each video in the dataset, updating metrics like seq_total_time and seq_total_frame. It also prepares variables for reference embeddings and masks for the Video Object Segmentation task using PaddlePaddle framework.",
+ "type": "comment"
+ },
+ "5970": {
+ "file_id": 478,
+ "content": " for frame_idx, samples in enumerate(seq_dataloader):\n time_start = time.time()\n all_preds = []\n join_label = None\n for aug_idx in range(len(samples)):\n if len(ref_embeddings) <= aug_idx:\n ref_embeddings.append([])\n ref_masks.append([])\n prev_embedding.append(None)\n prev_mask.append(None)\n sample = samples[aug_idx]\n ref_emb = ref_embeddings[aug_idx]\n ref_m = ref_masks[aug_idx]\n prev_emb = prev_embedding[aug_idx]\n prev_m = prev_mask[aug_idx]\n current_img = sample['current_img']\n if 'current_label' in sample.keys():\n current_label = sample['current_label']\n current_label = paddle.to_tensor(current_label)\n else:\n current_label = None",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:69-91"
+ },
+ "5971": {
+ "file_id": 478,
+ "content": "This code appears to be part of a data loading and processing loop for a video object detection model. It loads samples from a sequential dataloader, processes each augmented image, and appends their embeddings and masks to the corresponding lists. The labels are also loaded if available. This process is repeated for all augmented images in the sequence.",
+ "type": "comment"
+ },
+ "5972": {
+ "file_id": 478,
+ "content": " obj_num = sample['meta']['obj_num']\n imgname = sample['meta']['current_name']\n ori_height = sample['meta']['height']\n ori_width = sample['meta']['width']\n current_img = current_img\n obj_num = obj_num\n bs, _, h, w = current_img.shape\n data_batch = [\n ref_emb, ref_m, prev_emb, prev_m, current_img,\n [ori_height, ori_width], obj_num\n ]\n all_pred, current_embedding = model(data_batch, mode='test')\n if frame_idx == 0:\n if current_label is None:\n logger.info(\n \"No first frame label in Seq {}.\".format(\n seq_name))\n ref_embeddings[aug_idx].append(current_embedding)\n ref_masks[aug_idx].append(current_label)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:93-113"
+ },
+ "5973": {
+ "file_id": 478,
+ "content": "This code prepares data for a video object detection model. It extracts necessary information from the sample such as obj_num, imgname, ori_height and ori_width. The current image shape is also obtained. A list of data is created including reference embedding, reference mask, previous embedding, previous mask, the current image, image dimensions, and object number. The model is then used to generate all predictions and current embedding. If it's the first frame, if no label exists, an info message is logged. Reference embeddings and masks are appended accordingly.",
+ "type": "comment"
+ },
+ "5974": {
+ "file_id": 478,
+ "content": " prev_embedding[aug_idx] = current_embedding\n prev_mask[aug_idx] = current_label\n else:\n if sample['meta']['flip']: #False\n all_pred = self.flip_tensor(all_pred, 3)\n # In YouTube-VOS, not all the objects appear in the first frame for the first time. Thus, we\n # have to introduce new labels for new objects, if necessary.\n if not sample['meta']['flip'] and not (\n current_label is None) and join_label is None:\n join_label = paddle.cast(current_label,\n dtype='int64')\n all_preds.append(all_pred)\n if current_label is not None:\n ref_embeddings[aug_idx].append(current_embedding)\n prev_embedding[aug_idx] = current_embedding",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:115-129"
+ },
+ "5975": {
+ "file_id": 478,
+ "content": "In this code, it checks if the sample has a 'meta' field with 'flip' set to True. If not, it checks if there are new labels for new objects. If necessary, it introduces a new label and adds the current prediction and embedding to their respective lists. The prev_embedding is also updated.",
+ "type": "comment"
+ },
+ "5976": {
+ "file_id": 478,
+ "content": " if frame_idx > 0:\n all_preds = paddle.concat(all_preds, axis=0)\n all_preds = paddle.mean(\n all_preds, axis=0) #average results if augmentation\n pred_label = paddle.argmax(all_preds, axis=0)\n if join_label is not None:\n join_label = paddle.squeeze(paddle.squeeze(join_label,\n axis=0),\n axis=0)\n keep = paddle.cast((join_label == 0), dtype=\"int64\")\n pred_label = pred_label * keep + join_label * (1 - keep)\n pred_label = pred_label\n current_label = paddle.reshape(\n pred_label, shape=[1, 1, ori_height, ori_width])\n flip_pred_label = self.flip_tensor(pred_label, 1)\n flip_current_label = paddle.reshape(\n flip_pred_label, shape=[1, 1, ori_height, ori_width])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:131-147"
+ },
+ "5977": {
+ "file_id": 478,
+ "content": "This code calculates the mean of previous predictions, then finds the maximum value from these averaged results. It handles joining labels if present and reshapes the final prediction to match the original image dimensions.",
+ "type": "comment"
+ },
+ "5978": {
+ "file_id": 478,
+ "content": " for aug_idx in range(len(samples)):\n if join_label is not None:\n if samples[aug_idx]['meta']['flip']:\n ref_masks[aug_idx].append(flip_current_label)\n else:\n ref_masks[aug_idx].append(current_label)\n if samples[aug_idx]['meta']['flip']:\n prev_mask[aug_idx] = flip_current_label\n else:\n prev_mask[\n aug_idx] = current_label #update prev_mask\n one_frametime = time.time() - time_start\n seq_total_time += one_frametime\n seq_total_frame += 1\n obj_num = float(obj_num)\n logger.info('Frame: {}, Obj Num: {}, Time: {}'.format(\n imgname[0], obj_num, one_frametime))\n self.save_mask(\n pred_label,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:149-168"
+ },
+ "5979": {
+ "file_id": 478,
+ "content": "The code iterates over a list of samples, updating reference and previous masks based on whether the sample is flipped or not. It then calculates the time taken for one frame, adds it to total sequence time, increments the total frame count, logs frame information including object number, and saves the predicted label mask.",
+ "type": "comment"
+ },
+ "5980": {
+ "file_id": 478,
+ "content": " os.path.join(self.result_root, seq_name,\n imgname[0].split('.')[0] + '.png'))\n else:\n one_frametime = time.time() - time_start\n seq_total_time += one_frametime\n logger.info('Ref Frame: {}, Time: {}'.format(\n imgname[0], one_frametime))\n del (ref_embeddings)\n del (ref_masks)\n del (prev_embedding)\n del (prev_mask)\n del (seq_dataset)\n del (seq_dataloader)\n seq_avg_time_per_frame = seq_total_time / seq_total_frame\n self.total_time += seq_total_time\n self.total_frame += seq_total_frame\n total_avg_time_per_frame = self.total_time / self.total_frame\n self.total_sfps += seq_avg_time_per_frame\n avg_sfps = self.total_sfps / (batch_id + 1)\n logger.info(\"Seq {} FPS: {}, Total FPS: {}, FPS per Seq: {}\".format(\n seq_name, 1. / seq_avg_time_per_frame,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:169-191"
+ },
+ "5981": {
+ "file_id": 478,
+ "content": "This code calculates the average time per frame for a video sequence and reports it. It also keeps track of total time, total frames, average frames per second (FPS) for each sequence, and overall FPS. It logs this information for debugging or analysis purposes. The code handles both cases where all frames are successfully processed and when some frames fail processing. It then deletes unnecessary variables to free up memory.",
+ "type": "comment"
+ },
+ "5982": {
+ "file_id": 478,
+ "content": " 1. / total_avg_time_per_frame, 1. / avg_sfps))\n def flip_tensor(self, tensor, dim=0):\n inv_idx = paddle.cast(paddle.arange(tensor.shape[dim] - 1, -1, -1),\n dtype=\"int64\")\n tensor = paddle.index_select(x=tensor, index=inv_idx, axis=dim)\n return tensor\n def save_mask(self, mask_tensor, path):\n _palette = [\n 0, 0, 0, 128, 0, 0, 0, 128, 0, 128, 128, 0, 0, 0, 128, 128, 0, 128,\n 0, 128, 128, 128, 128, 128, 64, 0, 0, 191, 0, 0, 64, 128, 0, 191,\n 128, 0, 64, 0, 128, 191, 0, 128, 64, 128, 128, 191, 128, 128, 0, 64,\n 0, 128, 64, 0, 0, 191, 0, 128, 191, 0, 0, 64, 128, 128, 64, 128, 22,\n 22, 22, 23, 23, 23, 24, 24, 24, 25, 25, 25, 26, 26, 26, 27, 27, 27,\n 28, 28, 28, 29, 29, 29, 30, 30, 30, 31, 31, 31, 32, 32, 32, 33, 33,\n 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 37, 37, 37, 38, 38, 38, 39,\n 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 43, 43, 43, 44, 44, 44,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:192-209"
+ },
+ "5983": {
+ "file_id": 478,
+ "content": "This code defines a class with two methods: 'flip_tensor' and 'save_mask'. The 'flip_tensor' method flips the tensor along a specified dimension by inverting the indices. The 'save_mask' method saves a mask tensor to a specified file path using a provided palette.",
+ "type": "comment"
+ },
+ "5984": {
+ "file_id": 478,
+ "content": " 45, 45, 45, 46, 46, 46, 47, 47, 47, 48, 48, 48, 49, 49, 49, 50, 50,\n 50, 51, 51, 51, 52, 52, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56,\n 56, 56, 57, 57, 57, 58, 58, 58, 59, 59, 59, 60, 60, 60, 61, 61, 61,\n 62, 62, 62, 63, 63, 63, 64, 64, 64, 65, 65, 65, 66, 66, 66, 67, 67,\n 67, 68, 68, 68, 69, 69, 69, 70, 70, 70, 71, 71, 71, 72, 72, 72, 73,\n 73, 73, 74, 74, 74, 75, 75, 75, 76, 76, 76, 77, 77, 77, 78, 78, 78,\n 79, 79, 79, 80, 80, 80, 81, 81, 81, 82, 82, 82, 83, 83, 83, 84, 84,\n 84, 85, 85, 85, 86, 86, 86, 87, 87, 87, 88, 88, 88, 89, 89, 89, 90,\n 90, 90, 91, 91, 91, 92, 92, 92, 93, 93, 93, 94, 94, 94, 95, 95, 95,\n 96, 96, 96, 97, 97, 97, 98, 98, 98, 99, 99, 99, 100, 100, 100, 101,\n 101, 101, 102, 102, 102, 103, 103, 103, 104, 104, 104, 105, 105,\n 105, 106, 106, 106, 107, 107, 107, 108, 108, 108, 109, 109, 109,\n 110, 110, 110, 111, 111, 111, 112, 112, 112, 113, 113, 113, 114,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:210-222"
+ },
+ "5985": {
+ "file_id": 478,
+ "content": "This code appears to be a list of numbers, each representing a potential value for an unknown variable. It spans from 45 to 114 and includes each number exactly once. Without context or additional information, it's difficult to determine the purpose or meaning behind these values.",
+ "type": "comment"
+ },
+ "5986": {
+ "file_id": 478,
+ "content": " 114, 114, 115, 115, 115, 116, 116, 116, 117, 117, 117, 118, 118,\n 118, 119, 119, 119, 120, 120, 120, 121, 121, 121, 122, 122, 122,\n 123, 123, 123, 124, 124, 124, 125, 125, 125, 126, 126, 126, 127,\n 127, 127, 128, 128, 128, 129, 129, 129, 130, 130, 130, 131, 131,\n 131, 132, 132, 132, 133, 133, 133, 134, 134, 134, 135, 135, 135,\n 136, 136, 136, 137, 137, 137, 138, 138, 138, 139, 139, 139, 140,\n 140, 140, 141, 141, 141, 142, 142, 142, 143, 143, 143, 144, 144,\n 144, 145, 145, 145, 146, 146, 146, 147, 147, 147, 148, 148, 148,\n 149, 149, 149, 150, 150, 150, 151, 151, 151, 152, 152, 152, 153,\n 153, 153, 154, 154, 154, 155, 155, 155, 156, 156, 156, 157, 157,\n 157, 158, 158, 158, 159, 159, 159, 160, 160, 160, 161, 161, 161,\n 162, 162, 162, 163, 163, 163, 164, 164, 164, 165, 165, 165, 166,\n 166, 166, 167, 167, 167, 168, 168, 168, 169, 169, 169, 170, 170,\n 170, 171, 171, 171, 172, 172, 172, 173, 173, 173, 174, 174, 174,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:223-236"
+ },
+ "5987": {
+ "file_id": 478,
+ "content": "This code snippet contains a series of consecutive integers from 114 to 174.",
+ "type": "comment"
+ },
+ "5988": {
+ "file_id": 478,
+ "content": " 175, 175, 175, 176, 176, 176, 177, 177, 177, 178, 178, 178, 179,\n 179, 179, 180, 180, 180, 181, 181, 181, 182, 182, 182, 183, 183,\n 183, 184, 184, 184, 185, 185, 185, 186, 186, 186, 187, 187, 187,\n 188, 188, 188, 189, 189, 189, 190, 190, 190, 191, 191, 191, 192,\n 192, 192, 193, 193, 193, 194, 194, 194, 195, 195, 195, 196, 196,\n 196, 197, 197, 197, 198, 198, 198, 199, 199, 199, 200, 200, 200,\n 201, 201, 201, 202, 202, 202, 203, 203, 203, 204, 204, 204, 205,\n 205, 205, 206, 206, 206, 207, 207, 207, 208, 208, 208, 209, 209,\n 209, 210, 210, 210, 211, 211, 211, 212, 212, 212, 213, 213, 213,\n 214, 214, 214, 215, 215, 215, 216, 216, 216, 217, 217, 217, 218,\n 218, 218, 219, 219, 219, 220, 220, 220, 221, 221, 221, 222, 222,\n 222, 223, 223, 223, 224, 224, 224, 225, 225, 225, 226, 226, 226,\n 227, 227, 227, 228, 228, 228, 229, 229, 229, 230, 230, 230, 231,\n 231, 231, 232, 232, 232, 233, 233, 233, 234, 234, 234, 235, 235,",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:237-250"
+ },
+ "5989": {
+ "file_id": 478,
+ "content": "This code snippet is likely representing a list of numbers, potentially related to frame or timestamp values in the video processing context.",
+ "type": "comment"
+ },
+ "5990": {
+ "file_id": 478,
+ "content": " 235, 236, 236, 236, 237, 237, 237, 238, 238, 238, 239, 239, 239,\n 240, 240, 240, 241, 241, 241, 242, 242, 242, 243, 243, 243, 244,\n 244, 244, 245, 245, 245, 246, 246, 246, 247, 247, 247, 248, 248,\n 248, 249, 249, 249, 250, 250, 250, 251, 251, 251, 252, 252, 252,\n 253, 253, 253, 254, 254, 254, 255, 255, 255\n ]\n mask = mask_tensor.cpu().numpy().astype('uint8')\n mask = Image.fromarray(mask).convert('P')\n mask.putpalette(_palette)\n mask.save(path)\n def zip_folder(self, source_folder, zip_dir):\n f = zipfile.ZipFile(zip_dir, 'w', zipfile.ZIP_DEFLATED)\n pre_len = len(os.path.dirname(source_folder))\n for dirpath, dirnames, filenames in os.walk(source_folder):\n for filename in filenames:\n pathfile = os.path.join(dirpath, filename)\n arcname = pathfile[pre_len:].strip(os.path.sep)\n f.write(pathfile, arcname)\n f.close()\n def accumulate(self):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:251-272"
+ },
+ "5991": {
+ "file_id": 478,
+ "content": "Code snippet creates a mask from tensor data, converts it to an image and saves it with specified palette.\nThe 'zip_folder' function compresses the contents of a source folder into a zip file, preserving directory structure.\nThe 'accumulate' function is not defined in this code chunk.",
+ "type": "comment"
+ },
+ "5992": {
+ "file_id": 478,
+ "content": " \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n self.zip_folder(self.result_root, self.zip_dir)\n logger.info('Save result to {}.'.format(self.zip_dir))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/vos_metric.py:273-276"
+ },
+ "5993": {
+ "file_id": 478,
+ "content": "This code snippet is part of a class that handles metrics calculation. It accumulates metrics once all iterations are complete, then zips the results and saves them to a specified directory (zip_dir) using self.zip_folder method from the parent class. The logger.info statement displays an informational message confirming the save location and name of the zip file in the zip_dir.",
+ "type": "comment"
+ },
+ "5994": {
+ "file_id": 479,
+ "content": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py",
+ "type": "filepath"
+ },
+ "5995": {
+ "file_id": 479,
+ "content": "The AveragePrecisionCalculator calculates average precision in video object detection tasks, using a priority queue and providing methods for non-interpolated average precision. It also includes sorting, recall & precision computation, data shuffling, and prediction normalization.",
+ "type": "summary"
+ },
+ "5996": {
+ "file_id": 479,
+ "content": "# Copyright 2020 Google Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Calculate or keep track of the interpolated average precision.\nIt provides an interface for calculating interpolated average precision for an\nentire list or the top-n ranked items. For the definition of the\n(non-)interpolated average precision:\nhttp://trec.nist.gov/pubs/trec15/appendices/CE.MEASURES06.pdf\nExample usages:\n1) Use it as a static function call to directly calculate average precision for\na short ranked list in the memory.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:1-23"
+ },
+ "5997": {
+ "file_id": 479,
+ "content": "This code calculates the interpolated average precision for an entire list or top-n ranked items, following the definition provided in the given reference. It can be used as a static function call to calculate average precision for short ranked lists in memory.",
+ "type": "comment"
+ },
+ "5998": {
+ "file_id": 479,
+ "content": "```\nimport random\np = np.array([random.random() for _ in xrange(10)])\na = np.array([random.choice([0, 1]) for _ in xrange(10)])\nap = average_precision_calculator.AveragePrecisionCalculator.ap(p, a)\n```\n2) Use it as an object for long ranked list that cannot be stored in memory or\nthe case where partial predictions can be observed at a time (Tensorflow\npredictions). In this case, we first call the function accumulate many times\nto process parts of the ranked list. After processing all the parts, we call\npeek_interpolated_ap_at_n.\n```\np1 = np.array([random.random() for _ in xrange(5)])\na1 = np.array([random.choice([0, 1]) for _ in xrange(5)])\np2 = np.array([random.random() for _ in xrange(5)])\na2 = np.array([random.choice([0, 1]) for _ in xrange(5)])\n# interpolated average precision at 10 using 1000 break points\ncalculator = average_precision_calculator.AveragePrecisionCalculator(10)\ncalculator.accumulate(p1, a1)\ncalculator.accumulate(p2, a2)\nap3 = calculator.peek_ap_at_n()\n```\n\"\"\"\nimport heapq\nimport random\nimport numbers",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:25-55"
+ },
+ "5999": {
+ "file_id": 479,
+ "content": "The code defines an AveragePrecisionCalculator class that calculates average precision based on a ranked list. The calculator can handle long lists that cannot fit in memory or partial predictions observed over time (such as from Tensorflow). It uses the accumulate method to process parts of the ranked list and peek_interpolated_ap_at_n to calculate the interpolated average precision at a specific recall level.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/6.json b/docs/data/6.json
new file mode 100644
index 000000000..24027edfa
--- /dev/null
+++ b/docs/data/6.json
@@ -0,0 +1,541 @@
+{
+ "600": {
+ "file_id": 55,
+ "content": " seq_ref_frame_embedding = paddle.transpose(seq_ref_frame_embedding,\n [1, 2, 0])\n seq_ref_scribble_label = paddle.transpose(\n scale_ref_scribble_label[n], [1, 2, 0])\n nn_features_n = local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding=seq_ref_frame_embedding,\n query_embedding=seq_ref_frame_embedding,\n prev_frame_labels=seq_ref_scribble_label,\n gt_ids=gt_id,\n max_distance=self.cfg['model_max_local_distance'])\n #######\n ######################Global map update\n if seq_names[n] not in global_map_tmp_dic:\n global_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).tile([1000, 1, 1, 1, 1])\n nn_features_n_ = paddle.where(\n nn_features_n <=\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0),",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:814-832"
+ },
+ "601": {
+ "file_id": 55,
+ "content": "Updating the global map with the nearest neighbor features for each sequence, only if it's not already in the global_map_tmp_dic.",
+ "type": "comment"
+ },
+ "602": {
+ "file_id": 55,
+ "content": " nn_features_n,\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0))\n ###\n ###\n # print('detach 3')\n # nn_features_n_ = nn_features_n_.detach()\n global_map_tmp_dic[seq_names[n]][\n frame_num[n]] = nn_features_n_.detach()[0]\n ##################Local map update\n if local_map_dics is not None:\n local_map_tmp_dic, local_map_dist_dic = local_map_dics\n if seq_names[n] not in local_map_dist_dic:\n local_map_dist_dic[seq_names[n]] = paddle.zeros([1000, 9])\n if seq_names[n] not in local_map_tmp_dic:\n local_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).unsqueeze(0).tile([1000, 9, 1, 1, 1, 1])\n local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num\n - 1] = 0\n local_map_dics = (local_map_tmp_dic, local_map_dist_dic)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:833-854"
+ },
+ "603": {
+ "file_id": 55,
+ "content": "This code segment appears to be updating the global and local map dictionaries in a video processing model. The global_map_tmp_dic is being updated with nn_features_n_.detach()[0] at the current frame. Additionally, if the sequence name exists in the local_map_dist_dic or local_map_tmp_dic it is being modified accordingly.",
+ "type": "comment"
+ },
+ "604": {
+ "file_id": 55,
+ "content": " ##################\n to_cat_current_frame_embedding = ref_frame_embedding[n].unsqueeze(\n 0).tile((gt_id.shape[0], 1, 1, 1))\n to_cat_nn_feature_n = nn_features_n.squeeze(0).transpose(\n [2, 3, 0, 1])\n to_cat_scribble_mask_to_cat = (\n float_(seq_ref_scribble_label) == float_(gt_id)\n ) # float comparision?\n to_cat_scribble_mask_to_cat = float_(\n to_cat_scribble_mask_to_cat.unsqueeze(-1).transpose(\n [2, 3, 0, 1]))\n if not first_inter:\n seq_prev_round_label = scale_prev_round_label[n].transpose(\n [1, 2, 0])\n to_cat_prev_round_to_cat = (\n float_(seq_prev_round_label) == float_(gt_id)\n ) # float comparision?\n to_cat_prev_round_to_cat = float_(\n to_cat_prev_round_to_cat.unsqueeze(-1).transpose(\n [2, 3, 0, 1]))\n else:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:856-878"
+ },
+ "605": {
+ "file_id": 55,
+ "content": "This code calculates the current frame embedding and nn_feature_n for each object instance in the scene. It then creates a scribble mask for each object and, if not the first iteration, also creates a previous round mask. The code uses transpose and unsqueeze functions for tensor manipulation and float comparisons to create binary masks.",
+ "type": "comment"
+ },
+ "606": {
+ "file_id": 55,
+ "content": " to_cat_prev_round_to_cat = paddle.zeros_like(\n to_cat_scribble_mask_to_cat)\n to_cat_prev_round_to_cat[0] = 1.\n to_cat = paddle.concat(\n (to_cat_current_frame_embedding, to_cat_scribble_mask_to_cat,\n to_cat_prev_round_to_cat), 1)\n pred_ = self.inter_seghead(to_cat)\n pred_ = pred_.transpose([1, 0, 2, 3])\n dic_tmp[seq_names[n]] = pred_\n if local_map_dics is None:\n return dic_tmp\n else:\n return dic_tmp, local_map_dics",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py:879-893"
+ },
+ "607": {
+ "file_id": 55,
+ "content": "In this code, a concatenation of current frame embedding, scribble mask, and previous round information is passed to inter_seghead for segmentation prediction. The predictions are then transposed before being added to dic_tmp for further processing. If local_map_dics is None, the function returns dic_tmp; otherwise, it returns both dic_tmp and local_map_dics.",
+ "type": "comment"
+ },
+ "608": {
+ "file_id": 56,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/__init__.py",
+ "type": "filepath"
+ },
+ "609": {
+ "file_id": 56,
+ "content": "Copyright notice and license information for the module. Imports IntVOS from the same directory and adds it to __all__.",
+ "type": "summary"
+ },
+ "610": {
+ "file_id": 56,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .IntVOS import IntVOS\n__all__ = ['IntVOS'\n]",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/__init__.py:1-17"
+ },
+ "611": {
+ "file_id": 56,
+ "content": "Copyright notice and license information for the module. Imports IntVOS from the same directory and adds it to __all__.",
+ "type": "comment"
+ },
+ "612": {
+ "file_id": 57,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/modeling/registry.py",
+ "type": "filepath"
+ },
+ "613": {
+ "file_id": 57,
+ "content": "The \"registry.py\" file in PaddleVideo's EIVideo application defines Registry classes for components of the video processing pipeline, and organizes them into four registries for different functionalities: bbox_coder, estimator, multimodal, and segment.",
+ "type": "summary"
+ },
+ "614": {
+ "file_id": 57,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..utils import Registry\nBACKBONES = Registry('backbone')\nHEADS = Registry('head')\nRECOGNIZERS = Registry('recognizer')\nLOCALIZERS = Registry('localizer')\nPARTITIONERS = Registry('partitioner')\nSEGMENT = Registry('segmentation')\nLOSSES = Registry('loss')\nROI_EXTRACTORS = Registry('roi_extractor')\nDETECTORS = Registry('detectors')\nBBOX_ASSIGNERS = Registry('bbox_assigner')\nBBOX_SAMPLERS = Registry('bbox_sampler')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/registry.py:1-27"
+ },
+ "615": {
+ "file_id": 57,
+ "content": "The code snippet is from the \"registry.py\" file in PaddleVideo's EIVideo application. It defines several Registry classes, each representing a different component of the video processing pipeline: BACKBONES, HEADS, RECOGNIZERS, LOCALIZERS, PARTITIONERS, SEGMENT, LOSSES, ROI_EXTRACTORS, DETECTORS, BBOX_ASSIGNERS, and BBOX_SAMPLERS. These Registry classes will be used to register and manage different instances of these components in the video processing pipeline.",
+ "type": "comment"
+ },
+ "616": {
+ "file_id": 57,
+ "content": "BBOX_CODERS = Registry('bbox_coder')\nESTIMATORS = Registry('estimator')\nMULTIMODAL = Registry('multimodal')\nSEGMENT = Registry('segment')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/registry.py:28-31"
+ },
+ "617": {
+ "file_id": 57,
+ "content": "This code defines four registries for different functionalities: bbox_coder, estimator, multimodal, and segment. These registries will be used to organize and manage different types of models or algorithms in the codebase.",
+ "type": "comment"
+ },
+ "618": {
+ "file_id": 58,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py",
+ "type": "filepath"
+ },
+ "619": {
+ "file_id": 58,
+ "content": "This code initializes weights for a PaddlePaddle layer with options for customization and truncated normal distribution, offering proper initialization for deep learning models using normal distribution and PaddlePaddle's Normal initializer.",
+ "type": "summary"
+ },
+ "620": {
+ "file_id": 58,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn.initializer as init\nimport numpy as np\nfrom scipy import special\ndef weight_init_(layer,\n func,\n weight_name=None,\n bias_name=None,\n bias_value=0.0,\n **kwargs):\n \"\"\"\n In-place params init function.\n Usage:\n .. code-block:: python\n import paddle\n import numpy as np\n data = np.ones([3, 4], dtype='float32')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py:1-36"
+ },
+ "621": {
+ "file_id": 58,
+ "content": "The code is a function for initializing the weights of a given layer using different functions. It supports in-place parameter initialization and can be used with PaddlePaddle framework. The function accepts various arguments to customize the weight initialization process.",
+ "type": "comment"
+ },
+ "622": {
+ "file_id": 58,
+ "content": " linear = paddle.nn.Linear(4, 4)\n input = paddle.to_tensor(data)\n print(linear.weight)\n linear(input)\n weight_init_(linear, 'Normal', 'fc_w0', 'fc_b0', std=0.01, mean=0.1)\n print(linear.weight)\n \"\"\"\n if hasattr(layer, 'weight') and layer.weight is not None:\n getattr(init, func)(**kwargs)(layer.weight)\n if weight_name is not None:\n # override weight name\n layer.weight.name = weight_name\n if hasattr(layer, 'bias') and layer.bias is not None:\n init.Constant(bias_value)(layer.bias)\n if bias_name is not None:\n # override bias name\n layer.bias.name = bias_name\ndef _no_grad_trunc_normal_(tensor, mean, std, a, b):\n def norm_cdf(x):\n # Computes standard normal cumulative distribution function\n return (1. + math.erf(x / math.sqrt(2.))) / 2.\n if (mean < a - 2 * std) or (mean > b + 2 * std):\n print(\"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. \"\n \"The distribution of values may be incorrect.\")",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py:37-66"
+ },
+ "623": {
+ "file_id": 58,
+ "content": "Code initializes a linear layer, prints its weight, applies weight initialization with normal distribution, and prints the weight again. The _no_grad_trunc_normal_ function sets tensor values to be truncated normal with specified mean, std, a, and b parameters.",
+ "type": "comment"
+ },
+ "624": {
+ "file_id": 58,
+ "content": " with paddle.no_grad():\n # Values are generated by using a truncated uniform distribution and\n # then using the inverse CDF for the normal distribution.\n # Get upper and lower cdf values\n l = norm_cdf((a - mean) / std)\n u = norm_cdf((b - mean) / std)\n # Uniformly fill tensor with values from [l, u], then translate to [2l-1, 2u-1].\n tmp = np.random.uniform(2 * l - 1, 2 * u - 1,\n size=list(tensor.shape)).astype(np.float32)\n # Use inverse cdf transform for normal distribution to get truncated\n # standard normal\n tmp = special.erfinv(tmp)\n # Transform to proper mean, std\n tmp *= (std * math.sqrt(2.0))\n tmp += mean\n # Clamp to ensure it's in the proper range\n tmp = np.clip(tmp, a, b)\n tensor.set_value(paddle.to_tensor(tmp))\n return tensor\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = tensor.dim()\n if dimensions < 2:\n raise ValueError(\n \"Fan in and fan out can not be computed for tensor with fewer than 2 dimensions\"",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py:68-98"
+ },
+ "625": {
+ "file_id": 58,
+ "content": "This code initializes the weights of a tensor by generating values from a truncated normal distribution, with the lower and upper bounds defined by 'a' and 'b'. It then transforms these values to ensure they are within the desired range, mean, and standard deviation. The resulting tensor is set as the new value for the original tensor. This process ensures proper initialization for deep learning models.",
+ "type": "comment"
+ },
+ "626": {
+ "file_id": 58,
+ "content": " )\n num_input_fmaps = tensor.shape[1]\n num_output_fmaps = tensor.shape[0]\n receptive_field_size = 1\n if tensor.dim() > 2:\n receptive_field_size = tensor[0][0].numel()\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef trunc_normal_(tensor, mean=0., std=1., a=-2., b=2.):\n return _no_grad_trunc_normal_(tensor, mean, std, a, b)\ndef kaiming_normal_(tensor, a=0., mode='fan_in', nonlinearity='leaky_relu'):\n def _calculate_correct_fan(tensor, mode):\n mode = mode.lower()\n valid_modes = ['fan_in', 'fan_out']\n if mode not in valid_modes:\n raise ValueError(\n \"Mode {} not supported, please use one of {}\".format(\n mode, valid_modes))\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n return fan_in if mode == 'fan_in' else fan_out\n def calculate_gain(nonlinearity, param=None):\n linear_fns = [\n 'linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d',",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py:99-130"
+ },
+ "627": {
+ "file_id": 58,
+ "content": "The code defines a function for initializing the weight of a neural network. It first calculates the fan-in and fan-out based on the shape and dimensions of the tensor. Then, it provides options to initialize weights with truncated normal or Kaiming normal distributions. The trunc_normal_ and kaiming_normal_ functions are also defined to handle different initialization methods with optional parameters for mean, std, a, b, mode, and nonlinearity.",
+ "type": "comment"
+ },
+ "628": {
+ "file_id": 58,
+ "content": " 'conv_transpose2d', 'conv_transpose3d'\n ]\n if nonlinearity in linear_fns or nonlinearity == 'sigmoid':\n return 1\n elif nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if param is None:\n negative_slope = 0.01\n elif not isinstance(param, bool) and isinstance(\n param, int) or isinstance(param, float):\n negative_slope = param\n else:\n raise ValueError(\n \"negative_slope {} not a valid number\".format(param))\n return math.sqrt(2.0 / (1 + negative_slope**2))\n else:\n raise ValueError(\n \"Unsupported nonlinearity {}\".format(nonlinearity))\n fan = _calculate_correct_fan(tensor, mode)\n gain = calculate_gain(nonlinearity, a)\n std = gain / math.sqrt(fan)\n with paddle.no_grad():\n paddle.nn.initializer.Normal(0, std)(tensor)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py:131-157"
+ },
+ "629": {
+ "file_id": 58,
+ "content": "This function initializes the weights of a tensor with a normal distribution. It checks the type of nonlinearity function and returns an appropriate gain factor, then calculates the standard deviation for weight initialization using fan inversion formula. The final step is to initialize the tensor with Normal initializer from PaddlePaddle library.",
+ "type": "comment"
+ },
+ "630": {
+ "file_id": 58,
+ "content": " return tensor",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/modeling/weight_init.py:158-158"
+ },
+ "631": {
+ "file_id": 58,
+ "content": "Initializing weights for a neural network model.\nThis function returns the initialized tensor with random values.",
+ "type": "comment"
+ },
+ "632": {
+ "file_id": 59,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/tasks/__init__.py",
+ "type": "filepath"
+ },
+ "633": {
+ "file_id": 59,
+ "content": "This code snippet is importing the \"test_model\" function from the \"test.py\" module in the same directory and adding it to the __all__ list. The text at the beginning of the file contains license information and copyright notice.",
+ "type": "summary"
+ },
+ "634": {
+ "file_id": 59,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .test import test_model\n__all__ = [\n 'test_model',\n]",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/tasks/__init__.py:1-19"
+ },
+ "635": {
+ "file_id": 59,
+ "content": "This code snippet is importing the \"test_model\" function from the \"test.py\" module in the same directory and adding it to the __all__ list. The text at the beginning of the file contains license information and copyright notice.",
+ "type": "comment"
+ },
+ "636": {
+ "file_id": 60,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/tasks/test.py",
+ "type": "filepath"
+ },
+ "637": {
+ "file_id": 60,
+ "content": "This code imports libraries and defines a test_model function for testing a model without gradient calculation. It sets configuration, updates model's test_step function with updated parameters, and performs multi-card testing.",
+ "type": "summary"
+ },
+ "638": {
+ "file_id": 60,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom EIVideo.paddlevideo.utils import get_logger, load\nfrom ..loader.builder import build_dataloader, build_dataset\nfrom ..metrics import build_metric\nfrom ..modeling.builder import build_model\nfrom ..modeling.framework import Manet\nlogger = get_logger(\"paddlevideo\")\n@paddle.no_grad()\ndef test_model(cfg, weights, parallel=True):\n \"\"\"Test model entry\n Args:\n cfg (dict): configuration.\n weights (str): weights path to load.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/tasks/test.py:1-31"
+ },
+ "639": {
+ "file_id": 60,
+ "content": "This code imports necessary libraries and functions for testing a model. It defines a function called test_model that takes configuration (cfg), weights path (weights), and parallel flag as arguments. The function performs model testing without gradient calculation (paddle.no_grad()) to save computation resources.",
+ "type": "comment"
+ },
+ "640": {
+ "file_id": 60,
+ "content": " parallel (bool): Whether to do multi-cards testing. Default: True.\n \"\"\"\n if cfg.MODEL.framework == \"Manet\":\n cfg_helper = {\"knns\": 1, \"is_save_image\": True}\n cfg.update(cfg_helper)\n final = Manet().test_step(**cfg, weights=weights, parallel=False)\n return final",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/tasks/test.py:32-39"
+ },
+ "641": {
+ "file_id": 60,
+ "content": "This code sets the configuration for multi-card testing and then calls the Manet model's test_step function with updated configuration, weights, and parallel set to False.",
+ "type": "comment"
+ },
+ "642": {
+ "file_id": 61,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/__init__.py",
+ "type": "filepath"
+ },
+ "643": {
+ "file_id": 61,
+ "content": "This code imports various functions and classes from different modules within the PaddleVideo library. It also sets up logger and profiler functionality, provides a build function for creating objects, and handles saving and loading data.",
+ "type": "summary"
+ },
+ "644": {
+ "file_id": 61,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import Registry\nfrom .build_utils import build\nfrom .config import *\nfrom .logger import setup_logger, coloring, get_logger\nfrom .record import AverageMeter, build_record, log_batch, log_epoch\nfrom .dist_utils import get_dist_info, main_only\nfrom .save_load import save, load, load_ckpt, mkdir\nfrom .precise_bn import do_preciseBN\nfrom .profiler import add_profiler_step\n__all__ = ['Registry', 'build']",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/__init__.py:1-24"
+ },
+ "645": {
+ "file_id": 61,
+ "content": "This code imports various functions and classes from different modules within the PaddleVideo library. It also sets up logger and profiler functionality, provides a build function for creating objects, and handles saving and loading data.",
+ "type": "comment"
+ },
+ "646": {
+ "file_id": 62,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/build_utils.py",
+ "type": "filepath"
+ },
+ "647": {
+ "file_id": 62,
+ "content": "The \"build\" function takes a config dictionary and registry, constructs an object from the configuration, checks for required keys, retrieves class from the registry, and returns the instance.",
+ "type": "summary"
+ },
+ "648": {
+ "file_id": 62,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\ndef build(cfg, registry, key='name'):\n \"\"\"Build a module from config dict.\n Args:\n cfg (dict): Config dict. It should at least contain the key.\n registry (XXX): The registry to search the type from.\n key (str): the key.\n Returns:\n obj: The constructed object.\n \"\"\"\n assert isinstance(cfg, dict) and key in cfg\n cfg_copy = cfg.copy()\n obj_type = cfg_copy.pop(key)\n obj_cls = registry.get(obj_type)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/build_utils.py:1-31"
+ },
+ "649": {
+ "file_id": 62,
+ "content": "This code defines a function named \"build\" that takes a config dictionary and a registry, builds an object from the given configuration dictionary, and returns it. The function asserts that the input is a valid dictionary and checks if the required key exists. It then retrieves the object type from the dictionary and gets the corresponding class from the registry before returning the constructed object.",
+ "type": "comment"
+ },
+ "650": {
+ "file_id": 62,
+ "content": " if obj_cls is None:\n raise KeyError('{} is not in the {} registry'.format(\n obj_type, registry.name))\n return obj_cls(**cfg_copy)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/build_utils.py:32-35"
+ },
+ "651": {
+ "file_id": 62,
+ "content": "Checks if an object class is provided, raises a KeyError if not found in the registry, and returns an instance of the found class with provided configuration.",
+ "type": "comment"
+ },
+ "652": {
+ "file_id": 63,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py",
+ "type": "filepath"
+ },
+ "653": {
+ "file_id": 63,
+ "content": "The code imports libraries, sets up logger, and defines AttrDict class for config files. It includes functions to parse, print, visualize, check, and replace configurations using 'override' function. The code parses a config file, applies overrides, checks if the input option is a string, separates key-value pairs, splits keys by dots, calls `print_config()` and `check_config()`, and returns the updated config object.",
+ "type": "summary"
+ },
+ "654": {
+ "file_id": 63,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport yaml\nfrom EIVideo.paddlevideo.utils.logger import coloring, setup_logger\n__all__ = ['get_config']\nlogger = setup_logger(\"./\", name=\"paddlevideo\", level=\"INFO\")\nclass AttrDict(dict):\n def __getattr__(self, key):\n return self[key]\n def __setattr__(self, key, value):\n if key in self.__dict__:\n self.__dict__[key] = value\n else:\n self[key] = value\ndef create_attr_dict(yaml_config):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py:1-34"
+ },
+ "655": {
+ "file_id": 63,
+ "content": "This code block is importing necessary libraries, setting up logger, and defining an AttrDict class for handling config files. It also defines a function create_attr_dict that takes in a yaml configuration file.",
+ "type": "comment"
+ },
+ "656": {
+ "file_id": 63,
+ "content": " from ast import literal_eval\n for key, value in yaml_config.items():\n if type(value) is dict:\n yaml_config[key] = value = AttrDict(value)\n if isinstance(value, str):\n try:\n value = literal_eval(value)\n except BaseException:\n pass\n if isinstance(value, AttrDict):\n create_attr_dict(yaml_config[key])\n else:\n yaml_config[key] = value\ndef parse_config(cfg_file):\n \"\"\"Load a config file into AttrDict\"\"\"\n with open(cfg_file, 'r') as fopen:\n yaml_config = AttrDict(yaml.load(fopen, Loader=yaml.SafeLoader))\n create_attr_dict(yaml_config)\n return yaml_config\ndef print_dict(d, delimiter=0):\n \"\"\"\n Recursively visualize a dict and\n indenting acrrording by the relationship of keys.\n \"\"\"\n placeholder = \"-\" * 60\n for k, v in sorted(d.items()):\n if isinstance(v, dict):\n logger.info(\"{}{} : \".format(delimiter * \" \", coloring(k,\n \"HEADER\")))",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py:35-67"
+ },
+ "657": {
+ "file_id": 63,
+ "content": "This code defines functions for parsing and printing config files. The `parse_config` function loads a config file into an AttrDict object, handling nested dictionaries and string values. The `print_dict` function recursively visualizes a dictionary, indented based on the relationship of keys.",
+ "type": "comment"
+ },
+ "658": {
+ "file_id": 63,
+ "content": " print_dict(v, delimiter + 4)\n elif isinstance(v, list) and len(v) >= 1 and isinstance(v[0], dict):\n logger.info(\"{}{} : \".format(delimiter * \" \",\n coloring(str(k), \"HEADER\")))\n for value in v:\n print_dict(value, delimiter + 4)\n else:\n logger.info(\"{}{} : {}\".format(delimiter * \" \",\n coloring(k, \"HEADER\"),\n coloring(v, \"OKGREEN\")))\n if k.isupper():\n logger.info(placeholder)\ndef print_config(config):\n \"\"\"\n visualize configs\n Arguments:\n config: configs\n \"\"\"\n print_dict(config)\ndef check_config(config):\n \"\"\"\n Check config\n \"\"\"\n pass\ndef override(dl, ks, v):\n \"\"\"\n Recursively replace dict of list\n Args:\n dl(dict or list): dict or list to be replaced\n ks(list): list of keys\n v(str): value to be replaced\n \"\"\"\n def str2num(v):\n try:\n return eval(v)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py:68-109"
+ },
+ "659": {
+ "file_id": 63,
+ "content": "This code defines functions for visualizing and checking configurations, as well as recursively replacing dictionary or list values. It includes functions to print a configuration, check the configuration (currently empty), and override values in a dictionary or list. The print function formats output with coloring and delimiters, and the override function handles both dictionaries and lists for value replacement.",
+ "type": "comment"
+ },
+ "660": {
+ "file_id": 63,
+ "content": " except Exception:\n return v\n assert isinstance(dl, (list, dict)), (\"{} should be a list or a dict\")\n assert len(ks) > 0, ('lenght of keys should larger than 0')\n if isinstance(dl, list):\n k = str2num(ks[0])\n if len(ks) == 1:\n assert k < len(dl), ('index({}) out of range({})'.format(k, dl))\n dl[k] = str2num(v)\n else:\n override(dl[k], ks[1:], v)\n else:\n if len(ks) == 1:\n #assert ks[0] in dl, ('{} is not exist in {}'.format(ks[0], dl))\n if not ks[0] in dl:\n logger.warning('A new filed ({}) detected!'.format(ks[0], dl))\n dl[ks[0]] = str2num(v)\n else:\n assert ks[0] in dl, (\n '({}) doesn\\'t exist in {}, a new dict field is invalid'.format(\n ks[0], dl))\n override(dl[ks[0]], ks[1:], v)\ndef override_config(config, options=None):\n \"\"\"\n Recursively override the config\n Args:\n config(dict): dict to be replaced",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py:110-139"
+ },
+ "661": {
+ "file_id": 63,
+ "content": "This function overrides the config, recursively replacing values in the dictionary or list. It requires the configuration and optional options, both as dictionaries. If the key exists, it updates the value, otherwise a warning is issued for new fields, and a new field is created if the key is present in the options. If the key does not exist, an error is thrown.\n\nIn other words, this function allows you to update your configuration by replacing values with new ones. It also helps to identify and handle newly-appearing fields.",
+ "type": "comment"
+ },
+ "662": {
+ "file_id": 63,
+ "content": " options(list): list of pairs(key0.key1.idx.key2=value)\n such as: [\n epochs=20',\n 'PIPELINE.train.transform.1.ResizeImage.resize_short=300'\n ]\n Returns:\n config(dict): replaced config\n \"\"\"\n if options is not None:\n for opt in options:\n assert isinstance(opt,\n str), (\"option({}) should be a str\".format(opt))\n assert \"=\" in opt, (\n \"option({}) should contain a =\"\n \"to distinguish between key and value\".format(opt))\n pair = opt.split('=')\n assert len(pair) == 2, (\"there can be only a = in the option\")\n key, value = pair\n keys = key.split('.')\n override(config, keys, value)\n return config\ndef get_config(fname, overrides=None, show=True):\n \"\"\"\n Read config from file\n \"\"\"\n assert os.path.exists(fname), ('config file({}) is not exist'.format(fname))\n config = parse_config(fname)\n override_config(config, overrides)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py:140-170"
+ },
+ "663": {
+ "file_id": 63,
+ "content": "The code parses a config file and applies overrides. It checks if the input option is a string and if it contains an equal sign to separate key-value pairs. It then splits the option into key and value, further splitting keys by dots. The function overrides the configuration file with these options, returning the updated configuration.",
+ "type": "comment"
+ },
+ "664": {
+ "file_id": 63,
+ "content": " if show:\n print_config(config)\n check_config(config)\n return config",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/config.py:171-174"
+ },
+ "665": {
+ "file_id": 63,
+ "content": "This code checks if `show` is True, and if so, it calls the function `print_config(config)`. It then always calls another function `check_config(config)`, before finally returning the `config` object. This implies that `print_config()` prints out configuration details, while `check_config()` checks for correctness or validity of the configuration. The config is returned regardless to ensure it's available to the rest of the codebase.",
+ "type": "comment"
+ },
+ "666": {
+ "file_id": 64,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/dist_utils.py",
+ "type": "filepath"
+ },
+ "667": {
+ "file_id": 64,
+ "content": "This code is from PaddleVideo's EIVideo module and includes util functions for distributed computing. It defines a function get_dist_info() to retrieve the current rank and world size, and main_only() is a decorator that only runs the wrapped function if the rank is 0 (used in distributed environments).",
+ "type": "summary"
+ },
+ "668": {
+ "file_id": 64,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport functools\nimport paddle\nimport paddle.distributed as dist\ndef get_dist_info():\n world_size = dist.get_world_size()\n rank = dist.get_rank()\n return rank, world_size\ndef main_only(func):\n @functools.wraps(func)\n def wrapper(*args, **kwargs):\n rank, _ = get_dist_info()\n if rank == 0:\n return func(*args, **kwargs)\n return wrapper",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/dist_utils.py:1-30"
+ },
+ "669": {
+ "file_id": 64,
+ "content": "This code is from PaddleVideo's EIVideo module and includes util functions for distributed computing. It defines a function get_dist_info() to retrieve the current rank and world size, and main_only() is a decorator that only runs the wrapped function if the rank is 0 (used in distributed environments).",
+ "type": "comment"
+ },
+ "670": {
+ "file_id": 65,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py",
+ "type": "filepath"
+ },
+ "671": {
+ "file_id": 65,
+ "content": "This code sets up a colorful logging function for PaddleVideo, initializes logger with verbosity levels, and ensures non-propagation of logs. It configures logger for Python's logging module using different formats and handlers based on local rank.",
+ "type": "summary"
+ },
+ "672": {
+ "file_id": 65,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport logging\nimport os\nimport sys\nimport datetime\nfrom paddle.distributed import ParallelEnv\nColor = {\n 'RED': '\\033[31m',\n 'HEADER': '\\033[35m', # deep purple\n 'PURPLE': '\\033[95m', # purple\n 'OKBLUE': '\\033[94m',\n 'OKGREEN': '\\033[92m',\n 'WARNING': '\\033[93m',\n 'FAIL': '\\033[91m',\n 'ENDC': '\\033[0m'\n}\ndef coloring(message, color=\"OKGREEN\"):\n assert color in Color.keys()\n if os.environ.get('COLORING', True):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:1-38"
+ },
+ "673": {
+ "file_id": 65,
+ "content": "This code is from the \"logger.py\" file in the PaddleVideo project, and it sets up a coloring function for logging messages with optional colors using ANSI escape sequences. The function takes a message and an optional color parameter, which should be one of the defined colors in the Color dictionary. It asserts that the provided color is indeed a key in the dictionary, and then returns the message with the specified color applied. The function also checks the environment variable \"COLORING\" to determine whether coloring should be enabled or not (default is True).",
+ "type": "comment"
+ },
+ "674": {
+ "file_id": 65,
+ "content": " return Color[color] + str(message) + Color[\"ENDC\"]\n else:\n return message\nlogger_initialized = []\ndef setup_logger(output=None, name=\"paddlevideo\", level=\"INFO\"):\n \"\"\"\n Initialize the paddlevideo logger and set its verbosity level to \"INFO\".\n Args:\n output (str): a file name or a directory to save log. If None, will not save log file.\n If ends with \".txt\" or \".log\", assumed to be a file name.\n Otherwise, logs will be saved to `output/log.txt`.\n name (str): the root module name of this logger\n Returns:\n logging.Logger: a logger\n \"\"\"\n def time_zone(sec, fmt):\n real_time = datetime.datetime.now()\n return real_time.timetuple()\n logging.Formatter.converter = time_zone\n logger = logging.getLogger(name)\n if level == \"INFO\":\n logger.setLevel(logging.INFO)\n elif level==\"DEBUG\":\n logger.setLevel(logging.DEBUG)\n logger.propagate = False\n if level == \"DEBUG\":\n plain_formatter = logging.Formatter(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:39-71"
+ },
+ "675": {
+ "file_id": 65,
+ "content": "This code initializes the PaddleVideo logger and sets its verbosity level to \"INFO\" or \"DEBUG\", depending on the input argument. It also defines a custom time zone converter for logging, and ensures that the logger does not propagate logs to its parent loggers.",
+ "type": "comment"
+ },
+ "676": {
+ "file_id": 65,
+ "content": " \"[%(asctime)s] %(name)s %(levelname)s: %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n else:\n plain_formatter = logging.Formatter(\n \"[%(asctime)s] %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n # stdout logging: master only\n local_rank = ParallelEnv().local_rank\n if local_rank == 0:\n ch = logging.StreamHandler(stream=sys.stdout)\n ch.setLevel(logging.DEBUG)\n formatter = plain_formatter\n ch.setFormatter(formatter)\n logger.addHandler(ch)\n # file logging: all workers\n if output is not None:\n if output.endswith(\".txt\") or output.endswith(\".log\"):\n filename = output\n else:\n filename = os.path.join(output, \".log.txt\")\n if local_rank > 0:\n filename = filename + \".rank{}\".format(local_rank)\n # PathManager.mkdirs(os.path.dirname(filename))\n os.makedirs(os.path.dirname(filename), exist_ok=True)\n # fh = logging.StreamHandler(_cached_log_stream(filename)\n fh = logging.FileHandler(filename, mode='a')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:72-100"
+ },
+ "677": {
+ "file_id": 65,
+ "content": "This code configures a logger for Python's logging module. It uses different formats and handlers (stdout, file) based on the local rank of the process, creating separate log files for each worker ranked greater than 0. If the output is a .txt or .log file, it will be used as-is; otherwise, a .log.txt file with optional rank appended will be created. The code also ensures that missing directories for the log file are created beforehand.",
+ "type": "comment"
+ },
+ "678": {
+ "file_id": 65,
+ "content": " fh.setLevel(logging.DEBUG)\n fh.setFormatter(plain_formatter)\n logger.addHandler(fh)\n logger_initialized.append(name)\n return logger\ndef get_logger(name, output=None):\n logger = logging.getLogger(name)\n if name in logger_initialized:\n return logger\n return setup_logger(name=name, output=name)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:101-113"
+ },
+ "679": {
+ "file_id": 65,
+ "content": "This code initializes a logger object and sets its level to DEBUG, adds a file handler with a plain formatter, and appends the logger's name to an initialized list. The function returns the logger if it has been previously initialized for the given name; otherwise, it sets up the logger using the provided name and optional output.",
+ "type": "comment"
+ },
+ "680": {
+ "file_id": 66,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py",
+ "type": "filepath"
+ },
+ "681": {
+ "file_id": 66,
+ "content": "This PyTorch code uses OpenCV for image processing, offers conversion functions and error handling with PaddleVideo. It initializes tensors using Xavier/Glorot or Kaiming normal distribution, favoring Torch.nn.init methods over older ones.",
+ "type": "summary"
+ },
+ "682": {
+ "file_id": 66,
+ "content": "from __future__ import absolute_import\nimport json\nimport math\nimport os\nimport pickle\nimport warnings\nimport numpy\nimport numpy as np\nfrom numpy import inf\nfrom paddle import Tensor, concat, reshape, nn\nimport paddle\nfrom typing import Union, Iterable\n# from reprod_log.compare import compute_diff\n# from reprod_log.utils import check_print_diff, np2torch, np2paddle, torch2np, paddle2np\n_tensor_or_tensors = Union[paddle.Tensor, Iterable[paddle.Tensor]]\n_palette = [\n 0, 0, 0, 128, 0, 0, 0, 128, 0, 128, 128, 0, 0, 0, 128, 128, 0, 128, 0, 128,\n 128, 128, 128, 128, 64, 0, 0, 191, 0, 0, 64, 128, 0, 191, 128, 0, 64, 0,\n 128, 191, 0, 128, 64, 128, 128, 191, 128, 128, 0, 64, 0, 128, 64, 0, 0, 191,\n 0, 128, 191, 0, 0, 64, 128, 128, 64, 128, 22, 22, 22, 23, 23, 23, 24, 24,\n 24, 25, 25, 25, 26, 26, 26, 27, 27, 27, 28, 28, 28, 29, 29, 29, 30, 30, 30,\n 31, 31, 31, 32, 32, 32, 33, 33, 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 37,\n 37, 37, 38, 38, 38, 39, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 43, 43,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1-28"
+ },
+ "683": {
+ "file_id": 66,
+ "content": "Imports various modules and defines a type hint for paddle tensor or iterable of tensors.",
+ "type": "comment"
+ },
+ "684": {
+ "file_id": 66,
+ "content": " 43, 44, 44, 44, 45, 45, 45, 46, 46, 46, 47, 47, 47, 48, 48, 48, 49, 49, 49,\n 50, 50, 50, 51, 51, 51, 52, 52, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56,\n 56, 56, 57, 57, 57, 58, 58, 58, 59, 59, 59, 60, 60, 60, 61, 61, 61, 62, 62,\n 62, 63, 63, 63, 64, 64, 64, 65, 65, 65, 66, 66, 66, 67, 67, 67, 68, 68, 68,\n 69, 69, 69, 70, 70, 70, 71, 71, 71, 72, 72, 72, 73, 73, 73, 74, 74, 74, 75,\n 75, 75, 76, 76, 76, 77, 77, 77, 78, 78, 78, 79, 79, 79, 80, 80, 80, 81, 81,\n 81, 82, 82, 82, 83, 83, 83, 84, 84, 84, 85, 85, 85, 86, 86, 86, 87, 87, 87,\n 88, 88, 88, 89, 89, 89, 90, 90, 90, 91, 91, 91, 92, 92, 92, 93, 93, 93, 94,\n 94, 94, 95, 95, 95, 96, 96, 96, 97, 97, 97, 98, 98, 98, 99, 99, 99, 100,\n 100, 100, 101, 101, 101, 102, 102, 102, 103, 103, 103, 104, 104, 104, 105,\n 105, 105, 106, 106, 106, 107, 107, 107, 108, 108, 108, 109, 109, 109, 110,\n 110, 110, 111, 111, 111, 112, 112, 112, 113, 113, 113, 114, 114, 114, 115,\n 115, 115, 116, 116, 116, 117, 117, 117, 118, 118, 118, 119, 119, 119, 120,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:29-41"
+ },
+ "685": {
+ "file_id": 66,
+ "content": "This code consists of a long sequence of integers with no apparent functionality or structure. It may represent an array, list, or range of values used in various parts of the codebase, but without further context, it is impossible to determine the specific purpose or usage for these numbers.",
+ "type": "comment"
+ },
+ "686": {
+ "file_id": 66,
+ "content": " 120, 120, 121, 121, 121, 122, 122, 122, 123, 123, 123, 124, 124, 124, 125,\n 125, 125, 126, 126, 126, 127, 127, 127, 128, 128, 128, 129, 129, 129, 130,\n 130, 130, 131, 131, 131, 132, 132, 132, 133, 133, 133, 134, 134, 134, 135,\n 135, 135, 136, 136, 136, 137, 137, 137, 138, 138, 138, 139, 139, 139, 140,\n 140, 140, 141, 141, 141, 142, 142, 142, 143, 143, 143, 144, 144, 144, 145,\n 145, 145, 146, 146, 146, 147, 147, 147, 148, 148, 148, 149, 149, 149, 150,\n 150, 150, 151, 151, 151, 152, 152, 152, 153, 153, 153, 154, 154, 154, 155,\n 155, 155, 156, 156, 156, 157, 157, 157, 158, 158, 158, 159, 159, 159, 160,\n 160, 160, 161, 161, 161, 162, 162, 162, 163, 163, 163, 164, 164, 164, 165,\n 165, 165, 166, 166, 166, 167, 167, 167, 168, 168, 168, 169, 169, 169, 170,\n 170, 170, 171, 171, 171, 172, 172, 172, 173, 173, 173, 174, 174, 174, 175,\n 175, 175, 176, 176, 176, 177, 177, 177, 178, 178, 178, 179, 179, 179, 180,\n 180, 180, 181, 181, 181, 182, 182, 182, 183, 183, 183, 184, 184, 184, 185,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:42-54"
+ },
+ "687": {
+ "file_id": 66,
+ "content": "This code likely contains a list of integer values, potentially representing coordinates or other numerical data.",
+ "type": "comment"
+ },
+ "688": {
+ "file_id": 66,
+ "content": " 185, 185, 186, 186, 186, 187, 187, 187, 188, 188, 188, 189, 189, 189, 190,\n 190, 190, 191, 191, 191, 192, 192, 192, 193, 193, 193, 194, 194, 194, 195,\n 195, 195, 196, 196, 196, 197, 197, 197, 198, 198, 198, 199, 199, 199, 200,\n 200, 200, 201, 201, 201, 202, 202, 202, 203, 203, 203, 204, 204, 204, 205,\n 205, 205, 206, 206, 206, 207, 207, 207, 208, 208, 208, 209, 209, 209, 210,\n 210, 210, 211, 211, 211, 212, 212, 212, 213, 213, 213, 214, 214, 214, 215,\n 215, 215, 216, 216, 216, 217, 217, 217, 218, 218, 218, 219, 219, 219, 220,\n 220, 220, 221, 221, 221, 222, 222, 222, 223, 223, 223, 224, 224, 224, 225,\n 225, 225, 226, 226, 226, 227, 227, 227, 228, 228, 228, 229, 229, 229, 230,\n 230, 230, 231, 231, 231, 232, 232, 232, 233, 233, 233, 234, 234, 234, 235,\n 235, 235, 236, 236, 236, 237, 237, 237, 238, 238, 238, 239, 239, 239, 240,\n 240, 240, 241, 241, 241, 242, 242, 242, 243, 243, 243, 244, 244, 244, 245,\n 245, 245, 246, 246, 246, 247, 247, 247, 248, 248, 248, 249, 249, 249, 250,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:55-67"
+ },
+ "689": {
+ "file_id": 66,
+ "content": "This code appears to be a list of integers. It is difficult to provide a brief comment as there seems to be no clear context or purpose for these numbers in this specific location.",
+ "type": "comment"
+ },
+ "690": {
+ "file_id": 66,
+ "content": " 250, 250, 251, 251, 251, 252, 252, 252, 253, 253, 253, 254, 254, 254, 255,\n 255, 255\n]\n# paddle.set_device('gpu') if paddle.is_compiled_with_cuda() else paddle.set_device('cpu')\nimport paddle\nimport PIL\nimport numbers\nimport numpy as np\nfrom PIL import Image\nfrom paddle.vision.transforms import BaseTransform\nfrom paddle.vision.transforms import functional as F\nimport numpy as np\nfrom scipy.ndimage import interpolation, binary_dilation\ntry:\n from skimage import morphology, transform\nexcept ImportError as e:\n print(\n f\"{e}, [scikit-image] package and it's dependencies is required for EIVideo.\"\n )\nimport paddle\nimport cv2\nimport random\n####\ndef mask_damager(labels=None, p_black=0.2):\n scales = (0.8, 1.0, 1.2)\n kernel_size = random.randint(10, 15)\n kernel = np.ones((kernel_size, kernel_size), np.uint8)\n if random.random() < p_black:\n final_label = paddle.zeros_like(labels)\n final_label = final_label.squeeze().numpy()\n else:\n prot = random.randint(5, 15)\n nrot = random.randint(-15, -5)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:68-105"
+ },
+ "691": {
+ "file_id": 66,
+ "content": "This code defines a function called \"mask\\_damager\" which takes in labels and a probability of blacking out as input. It randomly scales the image using a scale range of (0.8, 1.0, 1.2), generates a random kernel size between 10 to 15, and applies random rotation to the image. If a random number is less than the given probability, it sets the final label as black; otherwise, it performs random rotations and scaling on the input labels.",
+ "type": "comment"
+ },
+ "692": {
+ "file_id": 66,
+ "content": " rots = [prot, nrot, 0]\n rot = rots[random.randint(0, 2)]\n sc = scales[random.randint(0, 2)]\n _, _, h, w = labels.shape\n tmp = labels.squeeze()\n tmp = tmp.unsqueeze(-1)\n tmp = tmp.numpy().astype(np.uint8)\n morph_p = random.random()\n if morph_p < 0.5:\n tmp = cv2.morphologyEx(tmp, cv2.MORPH_OPEN, kernel)\n else:\n tmp = cv2.morphologyEx(tmp, cv2.MORPH_CLOSE, kernel)\n tmp = tmp.astype(np.uint8)\n center = (w / 2, h / 2)\n M = cv2.getRotationMatrix2D(center, rot, sc)\n final_label = cv2.warpAffine(tmp, M, (w, h), cv2.INTER_NEAREST)\n return final_label\ncolor_map = [\n [0, 0, 0],\n [255, 127, 0],\n [30, 144, 255],\n [186, 85, 211],\n [255, 105, 180],\n [192, 255, 62],\n [255, 105, 180],\n [50, 255, 255],\n]\ncolor_map_np = np.array(color_map)\ndef overlay_davis(image, mask, alpha=0.5):\n \"\"\" Overlay segmentation on top of RGB image. from davis official\"\"\"\n im_overlay = image.copy()\n mask = mask.astype('uint8')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:106-146"
+ },
+ "693": {
+ "file_id": 66,
+ "content": "The code performs morphological operations on an image using OpenCV and then applies a rotation transformation to overlay the segmentation mask onto the RGB image. It uses different colors for different classes in the segmentation mask.",
+ "type": "comment"
+ },
+ "694": {
+ "file_id": 66,
+ "content": " colored_mask = color_map_np[mask]\n foreground = image * alpha + (1 - alpha) * colored_mask\n binary_mask = (mask > 0)\n # Compose image\n im_overlay[binary_mask] = foreground[binary_mask]\n countours = binary_dilation(binary_mask) ^ binary_mask\n im_overlay[countours, :] = 0\n return im_overlay.astype(image.dtype)\n# TODO\ndef submit_masks(masks, images, inter_file_path):\n overlays = []\n save_result_path = os.path.join(inter_file_path, 'result')\n os.makedirs(save_result_path, exist_ok=True)\n for imgname, (mask, image) in enumerate(zip(masks, images)):\n overlay = overlay_davis(image, mask)\n overlays.append(overlay.tolist())\n overlay = Image.fromarray(overlay)\n imgname = str(imgname)\n while len(imgname) < 5:\n imgname = '0' + imgname\n overlay.save(os.path.join(save_result_path, imgname + '.png'))\n result = {'overlays': overlays}\n # result = {'masks': masks.tolist()}\n with open(os.path.join(save_result_path, 'masks.json'), 'w') as f:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:147-172"
+ },
+ "695": {
+ "file_id": 66,
+ "content": "This function takes a list of masks and images, and for each pair, it applies an overlay function to generate an overlay image. It saves these overlay images in the specified directory with filenames corresponding to their original image names. Additionally, it stores the list of overlays as JSON in a file named \"masks.json\". The comments suggest that there might be another function to store masks as a list instead of overlays.",
+ "type": "comment"
+ },
+ "696": {
+ "file_id": 66,
+ "content": " json.dump(result, f)\ndef load_video(path, min_side=None):\n frame_list = []\n cap = cv2.VideoCapture(path)\n while (cap.isOpened()):\n _, frame = cap.read()\n if frame is None:\n break\n frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n if min_side:\n h, w = frame.shape[:2]\n new_w = (w * min_side // min(w, h))\n new_h = (h * min_side // min(w, h))\n frame = cv2.resize(frame, (new_w, new_h),\n interpolation=cv2.INTER_CUBIC)\n # .transpose([2, 0, 1])\n frame_list.append(frame)\n frames = np.stack(frame_list, axis=0)\n return frames\ndef get_scribbles():\n for i in range(8):\n with open(f'/home/lc/paddlevideo/data/bike-packing/lable/{i + 1}.json'\n ) as f:\n scribbles = json.load(f)\n first_scribble = not i\n yield scribbles, first_scribble\ndef get_images(sequence='bike-packing'):\n img_path = os.path.join('/home/lc/paddlevideo/data', sequence.strip(),",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:173-206"
+ },
+ "697": {
+ "file_id": 66,
+ "content": "load_video function reads frames from a video file and optionally resizes the frame to match a minimum side length, appending each frame to a list. The function then stacks the frames in the list into a single numpy array and returns it. get_scribbles generates scribble data for 8 labels by iterating through corresponding JSON files and yields the data along with a flag indicating if it is the first label or not. get_images retrieves video images from a specified sequence directory.",
+ "type": "comment"
+ },
+ "698": {
+ "file_id": 66,
+ "content": " 'frame')\n img_files = os.listdir(img_path)\n img_files.sort()\n files = []\n for img in img_files:\n img_file = np.array(Image.open(os.path.join(img_path, img)))\n files.append(img_file)\n return np.array(files)\ndef rough_ROI(ref_scribble_labels):\n #### b*1*h*w\n dist = 20\n b, _, h, w = ref_scribble_labels.shape\n filter_ = paddle.zeros_like(ref_scribble_labels)\n to_fill = paddle.zeros_like(ref_scribble_labels)\n for i in range(b):\n no_background = (ref_scribble_labels[i] != -1)\n no_background = no_background.squeeze(0)\n no_b = no_background.nonzero()\n (h_min, w_min) = paddle.min(no_b, 0)\n (h_max, w_max) = paddle.max(no_b, 0)\n filter_[i, 0,\n max(h_min - dist, 0):min(h_max + dist, h - 1),\n max(w_min - dist, 0):min(w_max + dist, w - 1)] = 1\n final_scribble_labels = paddle.where(byte_(filter_), ref_scribble_labels,\n to_fill)\n return final_scribble_labels",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:207-236"
+ },
+ "699": {
+ "file_id": 66,
+ "content": "The code defines two functions: \"load_image\" and \"rough_ROI\". The \"load_image\" function loads an image from a specified directory, sorts the images by file name, reads each image using PIL library, and returns the images as a numpy array. The \"rough_ROI\" function receives scribble labels as input, determines the bounding box around each scribble in the batch, applies this bounding box to another mask, and returns the final scribble labels after filtering.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/60.json b/docs/data/60.json
new file mode 100644
index 000000000..883ca2800
--- /dev/null
+++ b/docs/data/60.json
@@ -0,0 +1,544 @@
+{
+ "6000": {
+ "file_id": 479,
+ "content": "import numpy\nclass AveragePrecisionCalculator(object):\n \"\"\"Calculate the average precision and average precision at n.\"\"\"\n def __init__(self, top_n=None):\n \"\"\"Construct an AveragePrecisionCalculator to calculate average precision.\n This class is used to calculate the average precision for a single label.\n Args:\n top_n: A positive Integer specifying the average precision at n, or\n None to use all provided data points.\n Raises:\n ValueError: An error occurred when the top_n is not a positive integer.\n \"\"\"\n if not ((isinstance(top_n, int) and top_n >= 0) or top_n is None):\n raise ValueError(\"top_n must be a positive integer or None.\")\n self._top_n = top_n # average precision at n\n self._total_positives = 0 # total number of positives have seen\n self._heap = [] # max heap of (prediction, actual)\n @property\n def heap_size(self):\n \"\"\"Gets the heap size maintained in the class.\"\"\"\n return len(self._heap)\n @property",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:57-86"
+ },
+ "6001": {
+ "file_id": 479,
+ "content": "AveragePrecisionCalculator is a class used for calculating the average precision and average precision at n. It constructs an object to calculate average precision for single label, with optional top_n parameter for average precision at n. If top_n is not positive integer or None, a ValueError is raised. The class maintains heap of (prediction, actual) pairs and total positives seen. Heap size can be queried using the heap_size property.",
+ "type": "comment"
+ },
+ "6002": {
+ "file_id": 479,
+ "content": " def num_accumulated_positives(self):\n \"\"\"Gets the number of positive samples that have been accumulated.\"\"\"\n return self._total_positives\n def accumulate(self, predictions, actuals, num_positives=None):\n \"\"\"Accumulate the predictions and their ground truth labels.\n After the function call, we may call peek_ap_at_n to actually calculate\n the average precision.\n Note predictions and actuals must have the same shape.\n Args:\n predictions: a list storing the prediction scores.\n actuals: a list storing the ground truth labels. Any value\n larger than 0 will be treated as positives, otherwise as negatives.\n num_positives = If the 'predictions' and 'actuals' inputs aren't complete,\n then it's possible some true positives were missed in them. In that case,\n you can provide 'num_positives' in order to accurately track recall.\n Raises:\n ValueError: An error occurred when the format of the input is not the\n numpy 1-D array or the shape of predictions and actuals does not match.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:87-108"
+ },
+ "6003": {
+ "file_id": 479,
+ "content": "This code defines a class that calculates the average precision in video object detection tasks. It provides methods to accumulate positive samples and return the total number of positives. The accumulate method takes prediction scores, ground truth labels, and optional num_positives parameter for accurate tracking when inputs are incomplete.",
+ "type": "comment"
+ },
+ "6004": {
+ "file_id": 479,
+ "content": " \"\"\"\n if len(predictions) != len(actuals):\n raise ValueError(\n \"the shape of predictions and actuals does not match.\")\n if not num_positives is None:\n if not isinstance(num_positives,\n numbers.Number) or num_positives < 0:\n raise ValueError(\n \"'num_positives' was provided but it wan't a nonzero number.\"\n )\n if not num_positives is None:\n self._total_positives += num_positives\n else:\n self._total_positives += numpy.size(numpy.where(actuals > 0))\n topk = self._top_n\n heap = self._heap\n for i in range(numpy.size(predictions)):\n if topk is None or len(heap) < topk:\n heapq.heappush(heap, (predictions[i], actuals[i]))\n else:\n if predictions[i] > heap[0][0]: # heap[0] is the smallest\n heapq.heappop(heap)\n heapq.heappush(heap, (predictions[i], actuals[i]))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:109-134"
+ },
+ "6005": {
+ "file_id": 479,
+ "content": "This code checks if the length of predictions and actuals match. It also ensures that num_positives is a nonzero number, then adds positives to total_positives. The code uses heapq to push and pop elements in a priority queue based on top_n, ensuring correctness and efficiency.",
+ "type": "comment"
+ },
+ "6006": {
+ "file_id": 479,
+ "content": " def clear(self):\n \"\"\"Clear the accumulated predictions.\"\"\"\n self._heap = []\n self._total_positives = 0\n def peek_ap_at_n(self):\n \"\"\"Peek the non-interpolated average precision at n.\n Returns:\n The non-interpolated average precision at n (default 0).\n If n is larger than the length of the ranked list,\n the average precision will be returned.\n \"\"\"\n if self.heap_size <= 0:\n return 0\n predlists = numpy.array(list(zip(*self._heap)))\n ap = self.ap_at_n(predlists[0],\n predlists[1],\n n=self._top_n,\n total_num_positives=self._total_positives)\n return ap\n @staticmethod\n def ap(predictions, actuals):\n \"\"\"Calculate the non-interpolated average precision.\n Args:\n predictions: a numpy 1-D array storing the sparse prediction scores.\n actuals: a numpy 1-D array storing the ground truth labels. Any value\n larger than 0 will be treated as positives, otherwise as negatives.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:136-166"
+ },
+ "6007": {
+ "file_id": 479,
+ "content": "This code defines a class that calculates non-interpolated average precision. It has methods to clear accumulated predictions, peek the non-interpolated average precision at n, and calculate non-interpolated average precision from prediction and actual scores. The class uses numpy arrays and requires positive labels to be greater than 0 and negative labels as 0.",
+ "type": "comment"
+ },
+ "6008": {
+ "file_id": 479,
+ "content": " Returns:\n The non-interpolated average precision at n.\n If n is larger than the length of the ranked list,\n the average precision will be returned.\n Raises:\n ValueError: An error occurred when the format of the input is not the\n numpy 1-D array or the shape of predictions and actuals does not match.\n \"\"\"\n return AveragePrecisionCalculator.ap_at_n(predictions, actuals, n=None)\n @staticmethod\n def ap_at_n(predictions, actuals, n=20, total_num_positives=None):\n \"\"\"Calculate the non-interpolated average precision.\n Args:\n predictions: a numpy 1-D array storing the sparse prediction scores.\n actuals: a numpy 1-D array storing the ground truth labels. Any value\n larger than 0 will be treated as positives, otherwise as negatives.\n n: the top n items to be considered in ap@n.\n total_num_positives : (optionally) you can specify the number of total\n positive\n in the list. If specified, it will be used in calculation.\n Returns:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:168-192"
+ },
+ "6009": {
+ "file_id": 479,
+ "content": "This code calculates the non-interpolated average precision at a specified number 'n' from given predictions and actuals. It raises a ValueError if the input format is not a numpy 1D array, or the shape of predictions and actuals does not match.",
+ "type": "comment"
+ },
+ "6010": {
+ "file_id": 479,
+ "content": " The non-interpolated average precision at n.\n If n is larger than the length of the ranked list,\n the average precision will be returned.\n Raises:\n ValueError: An error occurred when\n 1) the format of the input is not the numpy 1-D array;\n 2) the shape of predictions and actuals does not match;\n 3) the input n is not a positive integer.\n \"\"\"\n if len(predictions) != len(actuals):\n raise ValueError(\n \"the shape of predictions and actuals does not match.\")\n if n is not None:\n if not isinstance(n, int) or n <= 0:\n raise ValueError(\"n must be 'None' or a positive integer.\"\n \" It was '%s'.\" % n)\n ap = 0.0\n predictions = numpy.array(predictions)\n actuals = numpy.array(actuals)\n # add a shuffler to avoid overestimating the ap\n predictions, actuals = AveragePrecisionCalculator._shuffle(\n predictions, actuals)\n sortidx = sorted(range(len(predictions)),",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:193-220"
+ },
+ "6011": {
+ "file_id": 479,
+ "content": "This function calculates the non-interpolated average precision at a given rank 'n'. It checks if the lengths of predictions and actuals match, ensures 'n' is an integer greater than zero, and shuffles the lists to avoid overestimation. If any errors occur, it raises a ValueError.",
+ "type": "comment"
+ },
+ "6012": {
+ "file_id": 479,
+ "content": " key=lambda k: predictions[k],\n reverse=True)\n if total_num_positives is None:\n numpos = numpy.size(numpy.where(actuals > 0))\n else:\n numpos = total_num_positives\n if numpos == 0:\n return 0\n if n is not None:\n numpos = min(numpos, n)\n delta_recall = 1.0 / numpos\n poscount = 0.0\n # calculate the ap\n r = len(sortidx)\n if n is not None:\n r = min(r, n)\n for i in range(r):\n if actuals[sortidx[i]] > 0:\n poscount += 1\n ap += poscount / (i + 1) * delta_recall\n return ap\n @staticmethod\n def _shuffle(predictions, actuals):\n random.seed(0)\n suffidx = random.sample(range(len(predictions)), len(predictions))\n predictions = predictions[suffidx]\n actuals = actuals[suffidx]\n return predictions, actuals\n @staticmethod\n def _zero_one_normalize(predictions, epsilon=1e-7):",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:221-256"
+ },
+ "6013": {
+ "file_id": 479,
+ "content": "This function calculates the average precision (AP) of a set of predictions and actuals. It first sorts the predictions by value, then calculates recall and precision to compute AP. If a total number of positives is provided, it uses that instead of counting non-zero actuals. The function also includes helper methods for shuffling the data and normalizing predictions with an epsilon value.",
+ "type": "comment"
+ },
+ "6014": {
+ "file_id": 479,
+ "content": " \"\"\"Normalize the predictions to the range between 0.0 and 1.0.\n For some predictions like SVM predictions, we need to normalize them before\n calculate the interpolated average precision. The normalization will not\n change the rank in the original list and thus won't change the average\n precision.\n Args:\n predictions: a numpy 1-D array storing the sparse prediction scores.\n epsilon: a small constant to avoid denominator being zero.\n Returns:\n The normalized prediction.\n \"\"\"\n denominator = numpy.max(predictions) - numpy.min(predictions)\n ret = (predictions - numpy.min(predictions)) / numpy.max(\n denominator, epsilon)\n return ret",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/average_precision_calculator.py:257-274"
+ },
+ "6015": {
+ "file_id": 479,
+ "content": "This function normalizes the predictions to a range of 0.0-1.0, ensuring that the rank in the original list remains unchanged and does not affect the average precision calculation. It prevents division by zero using a small epsilon value.",
+ "type": "comment"
+ },
+ "6016": {
+ "file_id": 480,
+ "content": "/paddlevideo/metrics/youtube8m/eval_util.py",
+ "type": "filepath"
+ },
+ "6017": {
+ "file_id": 480,
+ "content": "This code imports libraries, defines a merging function and uses Paddlevideo classes for evaluation. It calculates Hit@1, measures video-level precision, averages results to assess model performance. The function computes top-k triplet predictions, raises ValueError if k is not a positive integer, and initializes HitOneMetric class for evaluation metrics in Youtube8m's PaddleVideo module.",
+ "type": "summary"
+ },
+ "6018": {
+ "file_id": 480,
+ "content": "# Copyright 2016 Google Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Provides functions to help with evaluating models.\"\"\"\nimport numpy as np\nimport paddle\nfrom paddlevideo.utils import get_logger\nfrom ..base import BaseMetric\nfrom ..registry import METRIC\nfrom . import average_precision_calculator as ap_calculator\nfrom . import mean_average_precision_calculator as map_calculator\nlogger = get_logger(\"paddlevideo\")\ndef flatten(l):\n \"\"\" Merges a list of lists into a single list. \"\"\"\n return [item for sublist in l for item in sublist]",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:1-29"
+ },
+ "6019": {
+ "file_id": 480,
+ "content": "The code provides functions for evaluating models. It imports necessary libraries, defines a function to merge multiple lists into one, and includes classes for Average Precision Calculator and Mean Average Precision Calculator from the paddlevideo module.",
+ "type": "comment"
+ },
+ "6020": {
+ "file_id": 480,
+ "content": "def calculate_hit_at_one(predictions, actuals):\n \"\"\"\n Hit@k: indicates the fraction of test samples that contain at least\n one of the ground truth labels in the top k predictions,\n i.e topk.\n Args:\n predictions: Matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n actuals: Matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n Returns:\n float: The average hit at one across the entire batch.\n \"\"\"\n top_prediction = np.argmax(predictions, 1)\n hits = actuals[np.arange(actuals.shape[0]), top_prediction]\n return np.mean(hits)\ndef calculate_precision_at_equal_recall_rate(predictions, actuals):\n \"\"\"\n PERR: measures the video-level annotation precision when we retrieve the same number\n of entities per video as there are in the ground-truth.\n More details please refer to: https://arxiv.org/abs/1609.08675\n Args:\n predictions: Matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:32-60"
+ },
+ "6021": {
+ "file_id": 480,
+ "content": "Calculates Hit@1, the fraction of samples with at least one ground truth label in top predictions.\nMeasures video-level annotation precision when retrieving the same number of entities as ground truth.",
+ "type": "comment"
+ },
+ "6022": {
+ "file_id": 480,
+ "content": " actuals: Matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n Returns:\n float: The average precision at equal recall rate across the entire batch.\n \"\"\"\n aggregated_precision = 0.0\n num_videos = actuals.shape[0]\n for row in np.arange(num_videos):\n num_labels = int(np.sum(actuals[row]))\n top_indices = np.argpartition(predictions[row],\n -num_labels)[-num_labels:]\n item_precision = 0.0\n for label_index in top_indices:\n if predictions[row][label_index] > 0:\n item_precision += actuals[row][label_index]\n item_precision /= top_indices.size\n aggregated_precision += item_precision\n aggregated_precision /= num_videos\n return aggregated_precision\ndef calculate_gap(predictions, actuals, top_k=20):\n \"\"\"\n GAP: the global average precision.\n Only the top_k predictions are taken for each of the videos.\n Args:\n predictions: Matrix containing the outputs of the model.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:61-90"
+ },
+ "6023": {
+ "file_id": 480,
+ "content": "The code calculates the average precision at equal recall rate and global average precision for a batch of videos. It iterates over each video, determines the number of labels, finds the top indices based on predictions, calculates item-wise precision, aggregates these precisions for all videos, and returns the averaged precision as well as the gap score.",
+ "type": "comment"
+ },
+ "6024": {
+ "file_id": 480,
+ "content": " Dimensions are 'batch' x 'num_classes'.\n actuals: Matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n top_k: How many predictions to use per video.\n Returns:\n float: The global average precision.\n \"\"\"\n gap_calculator = ap_calculator.AveragePrecisionCalculator()\n sparse_predictions, sparse_labels, num_positives = top_k_by_class(\n predictions, actuals, top_k)\n gap_calculator.accumulate(flatten(sparse_predictions),\n flatten(sparse_labels), sum(num_positives))\n return gap_calculator.peek_ap_at_n()\ndef top_k_by_class(predictions, labels, k=20):\n \"\"\"Extracts the top k predictions for each video, sorted by class.\n Args:\n predictions: A numpy matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n k: the top k non-zero entries to preserve in each prediction.\n Returns:\n A tuple (predictions,labels, true_positives). 'predictions' and 'labels'",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:91-116"
+ },
+ "6025": {
+ "file_id": 480,
+ "content": "This code calculates the global average precision by first extracting the top k predictions for each video, sorted by class. It then accumulates these results using an AveragePrecisionCalculator and returns the global average precision.",
+ "type": "comment"
+ },
+ "6026": {
+ "file_id": 480,
+ "content": " are lists of lists of floats. 'true_positives' is a list of scalars. The\n length of the lists are equal to the number of classes. The entries in the\n predictions variable are probability predictions, and\n the corresponding entries in the labels variable are the ground truth for\n those predictions. The entries in 'true_positives' are the number of true\n positives for each class in the ground truth.\n Raises:\n ValueError: An error occurred when the k is not a positive integer.\n \"\"\"\n if k <= 0:\n raise ValueError(\"k must be a positive integer.\")\n k = min(k, predictions.shape[1])\n num_classes = predictions.shape[1]\n prediction_triplets = []\n for video_index in range(predictions.shape[0]):\n prediction_triplets.extend(\n top_k_triplets(predictions[video_index], labels[video_index], k))\n out_predictions = [[] for v in range(num_classes)]\n out_labels = [[] for v in range(num_classes)]\n for triplet in prediction_triplets:",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:117-137"
+ },
+ "6027": {
+ "file_id": 480,
+ "content": "This function takes in a list of lists containing probability predictions and ground truth labels for multiple classes, and calculates top-k triplet predictions based on the given k value. It raises a ValueError if k is not a positive integer. The function then creates empty lists to store output predictions and labels for each class.",
+ "type": "comment"
+ },
+ "6028": {
+ "file_id": 480,
+ "content": " out_predictions[triplet[0]].append(triplet[1])\n out_labels[triplet[0]].append(triplet[2])\n out_true_positives = [np.sum(labels[:, i]) for i in range(num_classes)]\n return out_predictions, out_labels, out_true_positives\ndef top_k_triplets(predictions, labels, k=20):\n \"\"\"Get the top_k for a 1-d numpy array. Returns a sparse list of tuples in\n (prediction, class) format\"\"\"\n m = len(predictions)\n k = min(k, m)\n indices = np.argpartition(predictions, -k)[-k:]\n return [(index, predictions[index], labels[index]) for index in indices]\n@METRIC.register\nclass HitOneMetric(BaseMetric):\n \"\"\"A class to store the evaluation metrics.\"\"\"\n def __init__(self,\n num_class,\n top_k,\n data_size,\n batch_size,\n log_interval=20):\n \"\"\"Construct an HitOneMetric object to store the evaluation metrics.\"\"\"\n self.hit_at_one = []\n self.perr = []\n self.gap = []\n super().__init__(data_size, batch_size, log_interval)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:138-167"
+ },
+ "6029": {
+ "file_id": 480,
+ "content": "This code calculates top-k predictions and labels from given predictions and labels arrays, and then initializes HitOneMetric class to store the evaluation metrics.",
+ "type": "comment"
+ },
+ "6030": {
+ "file_id": 480,
+ "content": " def accumulate(self):\n logger.info(\n '[TEST] finished, hit_at_one = {:.5f}, perr = {:.5f}, gap = {:.5f}'.\n format(np.mean(np.array(self.hit_at_one)),\n np.mean(np.array(self.perr)), np.mean(np.array(self.gap))))\n def clear(self):\n \"\"\"Clear the evaluation metrics and reset the HitOneMetric object.\"\"\"\n self.hit_at_one = []\n self.perr = []\n self.gap = []\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n hit_at_one = paddle.to_tensor(outputs['hit_at_one'])\n perr = paddle.to_tensor(outputs['perr'])\n gap = paddle.to_tensor(outputs['gap'])\n # NOTE(shipping): deal with multi cards validate\n if self.world_size > 1:\n hit_at_one = paddle.distributed.all_reduce(\n hit_at_one,\n op=paddle.distributed.ReduceOp.SUM) / self.world_size\n perr = paddle.distributed.all_reduce(\n perr, op=paddle.distributed.ReduceOp.SUM) / self.world_size",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:169-193"
+ },
+ "6031": {
+ "file_id": 480,
+ "content": "The code defines a HitOneMetric class for evaluating metrics in a video prediction task. The accumulate method calculates mean values of hit_at_one, perr, and gap, and logs the results as information. The clear method resets all metrics to an empty list. The update method updates the metric with each iteration, taking into account multi-card validation using PaddlePaddle's distributed functions.",
+ "type": "comment"
+ },
+ "6032": {
+ "file_id": 480,
+ "content": " gap = paddle.distributed.all_reduce(\n gap, op=paddle.distributed.ReduceOp.SUM) / self.world_size\n self.hit_at_one.append(hit_at_one.numpy())\n self.perr.append(perr.numpy())\n self.gap.append(gap.numpy())\n # preds ensemble\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{}...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size),\n ))",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/eval_util.py:194-205"
+ },
+ "6033": {
+ "file_id": 480,
+ "content": "This code snippet is a part of the Youtube8m evaluation module in PaddleVideo. It calculates the gap between ground truth and prediction for each batch, performs all-reduce on the gap, appends it to the corresponding list. Also, logs information about processing batches during testing.",
+ "type": "comment"
+ },
+ "6034": {
+ "file_id": 481,
+ "content": "/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py",
+ "type": "filepath"
+ },
+ "6035": {
+ "file_id": 481,
+ "content": "This code uses MeanAveragePrecisionCalculator to calculate mAP for ranked lists, initializes AveragePrecisionCalculator objects, supports interpolated precisions, and ensures shape compatibility. It averages average precisions of each class to provide the final result as mAP.",
+ "type": "summary"
+ },
+ "6036": {
+ "file_id": 481,
+ "content": "# Copyright 2016 Google Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Calculate the mean average precision.\nIt provides an interface for calculating mean average precision\nfor an entire list or the top-n ranked items.\nExample usages:\nWe first call the function accumulate many times to process parts of the ranked\nlist. After processing all the parts, we call peek_map_at_n\nto calculate the mean average precision.\n```\nimport random\np = np.array([[random.random() for _ in xrange(50)] for _ in xrange(1000)])",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py:1-27"
+ },
+ "6037": {
+ "file_id": 481,
+ "content": "This code calculates the mean average precision for a ranked list of items. It provides an interface to calculate this metric for the entire list or top-n ranked items. The example usage demonstrates accumulating data in parts and then using peek_map_at_n function to calculate the final result. The provided numpy array is used for demonstration purposes, representing a ranked list of values.",
+ "type": "comment"
+ },
+ "6038": {
+ "file_id": 481,
+ "content": "a = np.array([[random.choice([0, 1]) for _ in xrange(50)]\n for _ in xrange(1000)])\n# mean average precision for 50 classes.\ncalculator = mean_average_precision_calculator.MeanAveragePrecisionCalculator(\n num_class=50)\ncalculator.accumulate(p, a)\naps = calculator.peek_map_at_n()\n```\n\"\"\"\nimport numpy\nfrom . import average_precision_calculator\nclass MeanAveragePrecisionCalculator(object):\n \"\"\"This class is to calculate mean average precision.\n \"\"\"\n def __init__(self, num_class):\n \"\"\"Construct a calculator to calculate the (macro) average precision.\n Args:\n num_class: A positive Integer specifying the number of classes.\n top_n_array: A list of positive integers specifying the top n for each\n class. The top n in each class will be used to calculate its average\n precision at n.\n The size of the array must be num_class.\n Raises:\n ValueError: An error occurred when num_class is not a positive integer;\n or the top_n_array is not a list of positive integers.",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py:28-59"
+ },
+ "6039": {
+ "file_id": 481,
+ "content": "Creates a numpy array with 1000 samples, each containing 50 binary random choices. Initializes MeanAveragePrecisionCalculator object with specified number of classes (in this case, 50). Accumulates predictions and ground truth for calculating average precision. Retrieves the average precision map at a given point in time.",
+ "type": "comment"
+ },
+ "6040": {
+ "file_id": 481,
+ "content": " \"\"\"\n if not isinstance(num_class, int) or num_class <= 1:\n raise ValueError(\"num_class must be a positive integer.\")\n self._ap_calculators = [] # member of AveragePrecisionCalculator\n self._num_class = num_class # total number of classes\n for i in range(num_class):\n self._ap_calculators.append(\n average_precision_calculator.AveragePrecisionCalculator())\n def accumulate(self, predictions, actuals, num_positives=None):\n \"\"\"Accumulate the predictions and their ground truth labels.\n Args:\n predictions: A list of lists storing the prediction scores. The outer\n dimension corresponds to classes.\n actuals: A list of lists storing the ground truth labels. The dimensions\n should correspond to the predictions input. Any value\n larger than 0 will be treated as positives, otherwise as negatives.\n num_positives: If provided, it is a list of numbers representing the\n number of true positives for each class. If not provided, the number of",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py:60-80"
+ },
+ "6041": {
+ "file_id": 481,
+ "content": "This code defines a class for calculating Mean Average Precision (mAP) in the context of video classification. The constructor checks if num_class is a positive integer and initializes a list to store AveragePrecisionCalculator objects. The accumulate method takes predictions and actuals as input, accumulating prediction scores with their corresponding ground truth labels. If num_positives is provided, it represents the number of true positives for each class; otherwise, it defaults to no value.",
+ "type": "comment"
+ },
+ "6042": {
+ "file_id": 481,
+ "content": " true positives will be inferred from the 'actuals' array.\n Raises:\n ValueError: An error occurred when the shape of predictions and actuals\n does not match.\n \"\"\"\n if not num_positives:\n num_positives = [None for i in predictions.shape[1]]\n calculators = self._ap_calculators\n for i in range(len(predictions)):\n calculators[i].accumulate(predictions[i], actuals[i],\n num_positives[i])\n def clear(self):\n for calculator in self._ap_calculators:\n calculator.clear()\n def is_empty(self):\n return ([calculator.heap_size for calculator in self._ap_calculators] ==\n [0 for _ in range(self._num_class)])\n def peek_map_at_n(self):\n \"\"\"Peek the non-interpolated mean average precision at n.\n Returns:\n An array of non-interpolated average precision at n (default 0) for each\n class.\n \"\"\"\n aps = [\n self._ap_calculators[i].peek_ap_at_n()\n for i in range(self._num_class)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py:81-112"
+ },
+ "6043": {
+ "file_id": 481,
+ "content": "This code calculates the mean average precision for each class in a dataset, and provides methods to clear and check if the calculators are empty. The peek_map_at_n function returns an array of non-interpolated average precisions at n for each class. It also checks for shape compatibility between predictions and actuals arrays.",
+ "type": "comment"
+ },
+ "6044": {
+ "file_id": 481,
+ "content": " ]\n return aps",
+ "type": "code",
+ "location": "/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py:113-114"
+ },
+ "6045": {
+ "file_id": 481,
+ "content": "This code calculates the mean average precision (mAP) by averaging the average precisions of each class. It returns the mAP value as a result.",
+ "type": "comment"
+ },
+ "6046": {
+ "file_id": 482,
+ "content": "/paddlevideo/metrics/yowo_metric.py",
+ "type": "filepath"
+ },
+ "6047": {
+ "file_id": 482,
+ "content": "The code adds a YOWOMetric class to the PaddleVideo framework for measuring YOWO metrics in two stages: saving test results and calculating metrics from saved results files. The code also handles batch processing, logging progress, and evaluates mAP metrics.",
+ "type": "summary"
+ },
+ "6048": {
+ "file_id": 482,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport os\nfrom paddlevideo.utils import get_logger\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom .ucf24_utils import get_mAP\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass YOWOMetric(BaseMetric):\n \"\"\"\n Metrics for YOWO. Two Stages in this metric:\n (1) Get test results using trained model, results will be saved in YOWOMetric.result_path;\n (2) Calculate metrics using results file from stage (1).",
+ "type": "code",
+ "location": "/paddlevideo/metrics/yowo_metric.py:1-30"
+ },
+ "6049": {
+ "file_id": 482,
+ "content": "This code defines a YOWOMetric class within the PaddleVideo framework. The class measures metrics for YOWO in two stages: first, it saves test results using a trained model, and then calculates metrics from the saved results file.",
+ "type": "comment"
+ },
+ "6050": {
+ "file_id": 482,
+ "content": " \"\"\"\n def __init__(self,\n data_size,\n batch_size,\n gt_folder,\n result_path,\n threshold=0.5,\n save_path=None,\n log_interval=1):\n \"\"\"\n Init for BMN metrics.\n Params:\n gtfolder:groundtruth folder path for ucf24\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.result_path = result_path\n self.gt_folder = gt_folder\n self.threshold = threshold\n self.save_path = save_path\n if not osp.isdir(self.result_path):\n os.makedirs(self.result_path)\n def update(self, batch_id, data, outputs):\n frame_idx = outputs['frame_idx']\n boxes = outputs[\"boxes\"]\n for j in range(len(frame_idx)):\n detection_path = osp.join(self.result_path, frame_idx[j])\n with open(detection_path, 'w+') as f_detect:\n for box in boxes[j]:\n x1 = round(float(box[0] - box[2] / 2.0) * 320.0)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/yowo_metric.py:31-62"
+ },
+ "6051": {
+ "file_id": 482,
+ "content": "The code initializes an instance of a BMN metrics class with specified parameters. It checks if the result path exists and creates it if not, then updates the metric by writing detection results to corresponding files in the result path for each batch.",
+ "type": "comment"
+ },
+ "6052": {
+ "file_id": 482,
+ "content": " y1 = round(float(box[1] - box[3] / 2.0) * 240.0)\n x2 = round(float(box[0] + box[2] / 2.0) * 320.0)\n y2 = round(float(box[1] + box[3] / 2.0) * 240.0)\n det_conf = float(box[4])\n for j in range((len(box) - 5) // 2):\n cls_conf = float(box[5 + 2 * j].item())\n prob = det_conf * cls_conf\n f_detect.write(\n str(int(box[6]) + 1) + ' ' + str(prob) + ' ' + str(x1) + ' ' + str(y1) + ' ' + str(\n x2) + ' ' + str(y2) + '\\n')\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n metric_list = get_mAP(self.gt_folder, self.result_path, self.threshold, self.save_path)\n for info in metric_list:\n logger.info(info)",
+ "type": "code",
+ "location": "/paddlevideo/metrics/yowo_metric.py:63-82"
+ },
+ "6053": {
+ "file_id": 482,
+ "content": "This code snippet is part of the PaddleVideo library. It calculates and writes yolo v5 box information into a file, handling batch processing and logging progress with an interval. The accumulate function collects mAP metrics for evaluation.",
+ "type": "comment"
+ },
+ "6054": {
+ "file_id": 483,
+ "content": "/paddlevideo/modeling/__init__.py",
+ "type": "filepath"
+ },
+ "6055": {
+ "file_id": 483,
+ "content": "The code imports modules from PaddleVideo library, initializes a model registry, and provides functions for building video recognition models and defining loss functions.",
+ "type": "summary"
+ },
+ "6056": {
+ "file_id": 483,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .assigners import MaxIoUAssignerAVA\nfrom .backbones import ResNet\nfrom .builder import (build_backbone, build_head, build_localizer, build_loss,\n build_recognizer)\nfrom .framework.detectors import BaseDetector, FastRCNN, TwoStageDetector\nfrom .framework.recognizers import BaseRecognizer, Recognizer2D\nfrom .heads import (AVARoIHead, BaseHead, BBoxHeadAVA, SingleRoIExtractor3D,\n TSNHead)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/__init__.py:1-22"
+ },
+ "6057": {
+ "file_id": 483,
+ "content": "This code is an import statement from the PaddleVideo library, including various modules for backbones, builders, detectors, recognizers, and heads. It also includes license information and copyright details. The code allows users to access and build models using these imported modules.",
+ "type": "comment"
+ },
+ "6058": {
+ "file_id": 483,
+ "content": "from .losses import CrossEntropyLoss\nfrom .registry import (BACKBONES, DETECTORS, HEADS, LOCALIZERS, LOSSES,\n PARTITIONERS, RECOGNIZERS, ROI_EXTRACTORS)\nfrom .samplers import RandomSampler\nfrom .weight_init import kaiming_normal_, trunc_normal_, weight_init_\n__all__ = [\n 'BACKBONES', 'HEADS', 'RECOGNIZERS', 'LOCALIZERS', 'PARTITIONERS', 'LOSSES',\n 'build_recognizer', 'build_localizer', 'build_head', 'build_backbone',\n 'build_loss', 'ResNet', 'TSNHead', 'BaseHead', 'BaseRecognizer',\n 'Recognizer2d', 'CrossEntropyLoss', 'ROI_EXTRACTORS',\n 'SingleRoIExtractor3D', 'AVARoIHead', 'BBoxHeadAVA', 'MaxIoUAssignerAVA',\n 'RandomSampler', 'DETECTORS', 'kaiming_normal_', 'trunc_normal_',\n 'weight_init_'\n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/__init__.py:23-37"
+ },
+ "6059": {
+ "file_id": 483,
+ "content": "This code imports various modules, initializes a registry of models and functions, and lists all the available ones. It also defines a few key functions like `build_recognizer` and `build_localizer`, as well as some important loss functions such as `CrossEntropyLoss`. The code is part of PaddleVideo's modeling package and seems to be involved in building different parts of a video recognition model.",
+ "type": "comment"
+ },
+ "6060": {
+ "file_id": 484,
+ "content": "/paddlevideo/modeling/assigners/__init__.py",
+ "type": "filepath"
+ },
+ "6061": {
+ "file_id": 484,
+ "content": "This code imports the MaxIoUAssignerAVA class from the max_iou_assigner_ava module and adds it to the __all__ list, making it importable by default. The comment at the top of the file contains license information and copyright notices.",
+ "type": "summary"
+ },
+ "6062": {
+ "file_id": 484,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .max_iou_assigner_ava import MaxIoUAssignerAVA\n__all__ = ['MaxIoUAssignerAVA']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/__init__.py:1-17"
+ },
+ "6063": {
+ "file_id": 484,
+ "content": "This code imports the MaxIoUAssignerAVA class from the max_iou_assigner_ava module and adds it to the __all__ list, making it importable by default. The comment at the top of the file contains license information and copyright notices.",
+ "type": "comment"
+ },
+ "6064": {
+ "file_id": 485,
+ "content": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py",
+ "type": "filepath"
+ },
+ "6065": {
+ "file_id": 485,
+ "content": "The code defines AssignResult class, initializes MaxIoUAssignerAVA, assigns GT boxes to bboxes using max IOU method and handles multi-class cases. It's registered at BBOX_ASSIGNERS.",
+ "type": "summary"
+ },
+ "6066": {
+ "file_id": 485,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport numpy as np\nfrom ..registry import BBOX_ASSIGNERS\nfrom ..bbox_utils import bbox_overlaps\nclass AssignResult():\n def __init__(self, num_gts, gt_inds, max_overlaps, labels=None):\n self.num_gts = num_gts\n self.gt_inds = gt_inds\n self.max_overlaps = max_overlaps\n self.labels = labels\n def add_gt_(self, gt_labels):\n \"\"\"Add ground truth as assigned results. \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:1-27"
+ },
+ "6067": {
+ "file_id": 485,
+ "content": "This code defines a class called \"AssignResult\" for storing the assigned results, including number of gts, ground truth indexes, maximum overlaps, and labels if available. It also includes a method called \"add_gt_\" to add ground truth as assigned results.",
+ "type": "comment"
+ },
+ "6068": {
+ "file_id": 485,
+ "content": " self_inds = paddle.arange(1, len(gt_labels) + 1, dtype=\"int32\")\n gt_inds_squeeze = paddle.squeeze(self.gt_inds, axis=0)\n self.gt_inds = paddle.concat([self_inds, gt_inds_squeeze])\n gt_label_ones = paddle.full((len(gt_labels), ), 1, dtype='float32')\n max_overlaps_squeeze = paddle.squeeze(self.max_overlaps, axis=0)\n self.max_overlaps = paddle.concat([gt_label_ones, max_overlaps_squeeze])\n if self.labels is not None:\n self.labels = paddle.concat([gt_labels, self.labels])\n@BBOX_ASSIGNERS.register()\nclass MaxIoUAssignerAVA():\n \"\"\"Assign a corresponding gt bbox or background to each bbox. \"\"\"\n def __init__(self,\n pos_iou_thr,\n neg_iou_thr,\n min_pos_iou=.0,\n gt_max_assign_all=True,\n ignore_iof_thr=-1,\n ignore_wrt_candidates=True,\n match_low_quality=True,\n gpu_assign_thr=-1,\n iou_calculator=dict(type='BboxOverlaps2D')):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:28-49"
+ },
+ "6069": {
+ "file_id": 485,
+ "content": "This code initializes a MaxIoUAssignerAVA object by setting the self_inds and gt_inds attributes using paddle.arange and paddle.squeeze functions, concatenating them with paddle.concat function. It also sets max_overlaps attribute by concatenating gt_label_ones and max_overlaps_squeeze, and updates labels attribute if not None. The class is then registered at BBOX_ASSIGNERS with the decorator @BBOX_ASSIGNERS.register().",
+ "type": "comment"
+ },
+ "6070": {
+ "file_id": 485,
+ "content": " self.pos_iou_thr = pos_iou_thr\n self.neg_iou_thr = neg_iou_thr\n self.min_pos_iou = min_pos_iou\n self.gt_max_assign_all = gt_max_assign_all\n self.ignore_iof_thr = ignore_iof_thr\n self.ignore_wrt_candidates = ignore_wrt_candidates\n self.gpu_assign_thr = gpu_assign_thr\n self.match_low_quality = match_low_quality\n def assign(self, \n bboxes, \n gt_bboxes, \n gt_labels=None):\n \"\"\"Assign gt to bboxes. \"\"\"\n overlaps = bbox_overlaps(gt_bboxes, bboxes)\n assign_result = self.assign_wrt_overlaps(overlaps, gt_labels)\n return assign_result\n def assign_wrt_overlaps(self, overlaps, gt_labels=None):\n \"\"\"Assign w.r.t. the overlaps of bboxes with gts. \"\"\"\n num_gts, num_bboxes = overlaps.shape[0], overlaps.shape[1]\n # 1. assign -1\n assigned_gt_inds = paddle.full((num_bboxes, ), -1, dtype='int32')\n # for each anchor, which gt best overlaps with it\n # for each anchor, the max iou of all gts",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:50-75"
+ },
+ "6071": {
+ "file_id": 485,
+ "content": "The code defines a class that assigns ground truth (GT) boxes to bboxes. It takes in bboxes and GT bboxes as input, and returns the assignment result. The function assign_wrt_overlaps calculates assigned_gt_inds based on the overlaps of bboxes with gts.",
+ "type": "comment"
+ },
+ "6072": {
+ "file_id": 485,
+ "content": " max_overlaps, argmax_overlaps = paddle.topk(overlaps, k=1, axis=0)\n # for each gt, which anchor best overlaps with it\n # for each gt, the max iou of all proposals\n gt_max_overlaps, gt_argmax_overlaps = paddle.topk(overlaps, k=1, axis=1) \n # 2. assign negative: below the negative inds are set to be 0\n match_labels = paddle.full(argmax_overlaps.shape, -1, dtype='int32')\n match_labels = paddle.where(max_overlaps < self.neg_iou_thr,\n paddle.zeros_like(match_labels), match_labels)\n # 3. assign positive: above positive IoU threshold\n argmax_overlaps_int32 = paddle.cast(argmax_overlaps, 'int32')\n match_labels = paddle.where(max_overlaps >= self.pos_iou_thr,\n argmax_overlaps_int32 + 1, match_labels)\n assigned_gt_inds = match_labels\n if self.match_low_quality:\n # Low-quality matching will overwirte the assigned_gt_inds\n # assigned in Step 3. Thus, the assigned gt might not be the",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:76-93"
+ },
+ "6073": {
+ "file_id": 485,
+ "content": "This code assigns positive and negative labels to anchors based on their IoU with ground truth boxes. If the max IoU is above a certain threshold, it's considered positive. If it's below another threshold, it's negative. This process helps determine which anchor best overlaps with each ground truth box.",
+ "type": "comment"
+ },
+ "6074": {
+ "file_id": 485,
+ "content": " # best one for prediction.\n # For example, if bbox A has 0.9 and 0.8 iou with GT bbox\n # 1 & 2, bbox 1 will be assigned as the best target for bbox A\n # in step 3. However, if GT bbox 2's gt_argmax_overlaps = A,\n # bbox A's assigned_gt_inds will be overwritten to be bbox B.\n # This might be the reason that it is not used in ROI Heads.\n for i in range(num_gts):\n if gt_max_overlaps.numpy()[i] >= self.min_pos_iou:\n if self.gt_max_assign_all:\n equal_x_np = overlaps[i, :].numpy()\n equal_y_np = gt_max_overlaps[i].numpy()\n max_iou_inds = np.equal(equal_x_np, equal_y_np)\n max_iou_inds = paddle.to_tensor(max_iou_inds)\n max_iou_inds = paddle.reshape( max_iou_inds, [1,max_iou_inds.shape[0]] )\n match_labels_gts = paddle.full(max_iou_inds.shape, i+1, dtype='int32')\n match_labels = paddle.where(max_iou_inds, match_labels_gts, match_labels)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:94-109"
+ },
+ "6075": {
+ "file_id": 485,
+ "content": "This code iterates over each ground truth (GT) bounding box, and if the IOU with a detection is above the minimum allowed position IOU, it checks whether all overlapping detections should be assigned to this GT. It creates a tensor of boolean values representing the assignment for each detection and GT pair. This is done by comparing the overlaps matrix and gt_max_overlaps, then reshaping and replacing match labels accordingly.",
+ "type": "comment"
+ },
+ "6076": {
+ "file_id": 485,
+ "content": " assigned_gt_inds = match_labels\n else:\n assigned_gt_inds[gt_argmax_overlaps[i]] = i + 1\n if gt_labels is not None:\n # consider multi-class case (AVA)\n assert len(gt_labels[0]) > 1\n assigned_labels = paddle.full([num_bboxes, len(gt_labels[0])], 0, dtype='float32')\n assigned_gt_inds_reshape = assigned_gt_inds.reshape([assigned_gt_inds.shape[1]])\n pos_inds = paddle.nonzero( assigned_gt_inds_reshape , as_tuple=False)\n pos_inds_num = float(paddle.numel(pos_inds))\n if pos_inds_num > 0:\n pos_inds = paddle.squeeze(pos_inds, axis = 1 )\n assigned_gt_inds_squeeze = paddle.squeeze(assigned_gt_inds, axis=0)\n assigned_gt_inds_select = paddle.index_select(assigned_gt_inds_squeeze, pos_inds) - 1\n gt_labels_select = paddle.index_select(gt_labels, assigned_gt_inds_select)\n A = assigned_gt_inds_squeeze",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:110-126"
+ },
+ "6077": {
+ "file_id": 485,
+ "content": "This code assigns ground truth (GT) labels and indices for maximum IOU Assigner in the AVA dataset. It handles both multi-class cases with multiple classes per label. If there is a match, GT indices are assigned, otherwise, it assigns index + 1. Finally, it considers the multi-class case by asserting the existence of more than one class and assigns zeros to the initial labels array before updating them based on the selected gt_labels.",
+ "type": "comment"
+ },
+ "6078": {
+ "file_id": 485,
+ "content": " X = assigned_gt_inds_squeeze - 1\n Y = paddle.zeros_like(X)\n if A.shape[0]==1:\n if float(A) > 0:\n T=X\n else:\n T=Y\n else:\n T = paddle.where(A>0, X, Y)\n S = paddle.index_select(gt_labels, T)\n AE = paddle.expand(A, [S.shape[1], A.shape[0]]) \n AET = paddle.transpose(AE, perm=[1, 0])\n R = paddle.where(AET>0, S, assigned_labels) \n assigned_labels = R\n else:\n assigned_labels = None\n ret = AssignResult(\n num_gts,\n assigned_gt_inds,\n max_overlaps,\n labels=assigned_labels)\n return ret",
+ "type": "code",
+ "location": "/paddlevideo/modeling/assigners/max_iou_assigner_ava.py:127-148"
+ },
+ "6079": {
+ "file_id": 485,
+ "content": "This code snippet is part of a max IOU assigner implementation in PaddleVideo. It assigns labels to objects based on the maximum IoU (intersection over union) threshold. If there's only one object, it assigns the ground truth index if the overlap is greater than 0, otherwise sets it to 0. For multiple objects, it uses a where statement to select the max IOU assignment. The assigned labels are then returned as part of the AssignResult.",
+ "type": "comment"
+ },
+ "6080": {
+ "file_id": 486,
+ "content": "/paddlevideo/modeling/backbones/__init__.py",
+ "type": "filepath"
+ },
+ "6081": {
+ "file_id": 486,
+ "content": "This code initializes and defines various backbone models for video analysis tasks in PaddleVideo, including ResNet, Vision Transformer, AGCN, and popular models such as ResNetTSN_MRI, ResNetTSM_MRI, and SwinTransformer3D. These models form the foundation for object detection, segmentation, motion estimation, and various computer vision applications in PaddlePaddle framework.",
+ "type": "summary"
+ },
+ "6082": {
+ "file_id": 486,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .actbert import BertForMultiModalPreTraining\nfrom .adds import ADDS_DepthNet\nfrom .agcn import AGCN\nfrom .asrf import ASRF\nfrom .bmn import BMN\nfrom .cfbi import CFBI\nfrom .movinet import MoViNet\nfrom .ms_tcn import MSTCN\nfrom .resnet import ResNet\nfrom .resnet_slowfast import ResNetSlowFast\nfrom .resnet_slowfast_MRI import ResNetSlowFast_MRI\nfrom .resnet_tsm import ResNetTSM\nfrom .resnet_tsm_MRI import ResNetTSM_MRI",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/__init__.py:1-27"
+ },
+ "6083": {
+ "file_id": 486,
+ "content": "This code is an initialization file for backbone models in PaddleVideo. It imports various model classes from submodules, including BertForMultiModalPreTraining, ADDS_DepthNet, AGCN, ASRF, BMN, CFBI, MoViNet, MSTCN, ResNet, ResNetSlowFast, ResNetSlowFast_MRI, and ResNetTSM, ResNetTSM_MRI. These models can be used for video analysis tasks in the PaddlePaddle framework.",
+ "type": "comment"
+ },
+ "6084": {
+ "file_id": 486,
+ "content": "from .resnet_tsn_MRI import ResNetTSN_MRI\nfrom .resnet_tweaks_tsm import ResNetTweaksTSM\nfrom .resnet_tweaks_tsn import ResNetTweaksTSN\nfrom .stgcn import STGCN\nfrom .swin_transformer import SwinTransformer3D\nfrom .transnetv2 import TransNetV2\nfrom .vit import VisionTransformer\nfrom .vit_tweaks import VisionTransformer_tweaks\nfrom .ms_tcn import MSTCN\nfrom .asrf import ASRF\nfrom .resnet_tsn_MRI import ResNetTSN_MRI\nfrom .resnet_tsm_MRI import ResNetTSM_MRI\nfrom .resnet_slowfast_MRI import ResNetSlowFast_MRI\nfrom .cfbi import CFBI\nfrom .ctrgcn import CTRGCN\nfrom .agcn2s import AGCN2s\nfrom .movinet import MoViNet\nfrom .resnet3d_slowonly import ResNet3dSlowOnly\nfrom .toshift_vit import TokenShiftVisionTransformer\nfrom .pptsm_mv2 import PPTSM_MobileNetV2\nfrom .pptsm_mv3 import PPTSM_MobileNetV3\nfrom .pptsm_v2 import PPTSM_v2\nfrom .yowo import YOWO\n__all__ = [\n 'ResNet', 'ResNetTSM', 'ResNetTweaksTSM', 'ResNetSlowFast', 'BMN',\n 'ResNetTweaksTSN', 'VisionTransformer', 'STGCN', 'AGCN', 'TransNetV2',\n 'ADDS_DepthNet', 'VisionTransformer_tweaks', 'BertForMultiModalPreTraining',",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/__init__.py:28-55"
+ },
+ "6085": {
+ "file_id": 486,
+ "content": "The code imports various backbone models for video analysis from different modules within the PaddleVideo library, including ResNet, Vision Transformer, STGCN, AGCN, and more. The models are used for tasks like object detection, segmentation, and motion estimation in video processing.",
+ "type": "comment"
+ },
+ "6086": {
+ "file_id": 486,
+ "content": " 'ResNetTSN_MRI', 'ResNetTSM_MRI', 'ResNetSlowFast_MRI', 'CFBI', 'MSTCN',\n 'ASRF', 'MoViNet', 'SwinTransformer3D', 'CTRGCN',\n 'TokenShiftVisionTransformer', 'AGCN2s', 'PPTSM_MobileNetV2',\n 'PPTSM_MobileNetV3', 'PPTSM_v2', 'ResNet3dSlowOnly', 'YOWO'\n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/__init__.py:56-60"
+ },
+ "6087": {
+ "file_id": 486,
+ "content": "This code defines a list of available backbones for video processing tasks, including popular models such as ResNetTSN_MRI, ResNetTSM_MRI, and SwinTransformer3D. These backbones serve as the foundation for various computer vision applications in PaddleVideo.",
+ "type": "comment"
+ },
+ "6088": {
+ "file_id": 487,
+ "content": "/paddlevideo/modeling/backbones/actbert.py",
+ "type": "filepath"
+ },
+ "6089": {
+ "file_id": 487,
+ "content": "The code presents a PaddlePaddle BertEmbeddings class for BERT model embeddings in video action recognition, utilizing self-attention and ACTBERT's backbone for multimodal inputs including text, video, and action data.",
+ "type": "summary"
+ },
+ "6090": {
+ "file_id": 487,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport sys\nimport numpy as np\nimport math\nimport copy\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout)\nfrom paddle.nn.initializer import Constant, Normal\nfrom ...utils.save_load import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nACT2FN = {\"gelu\": F.gelu, \"relu\": F.relu, \"swish\": F.swish}\nclass BertEmbeddings(nn.Layer):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1-32"
+ },
+ "6091": {
+ "file_id": 487,
+ "content": "This code is a Python file containing a class named \"BertEmbeddings\" within the PaddlePaddle framework. The class inherits from nn.Layer and appears to contain embeddings for the BERT model. This code also includes comments with copyright information, license details, and an import section with necessary libraries. It introduces a dictionary, ACT2FN, that maps activation functions for use in the BertEmbeddings class.",
+ "type": "comment"
+ },
+ "6092": {
+ "file_id": 487,
+ "content": " \"\"\"Construct the embeddings from word, position and token_type embeddings.\n \"\"\"\n def __init__(self, vocab_size, max_position_embeddings, type_vocab_size,\n hidden_size, hidden_dropout_prob):\n super(BertEmbeddings, self).__init__()\n self.word_embeddings = nn.Embedding(vocab_size,\n hidden_size,\n padding_idx=0)\n self.position_embeddings = nn.Embedding(max_position_embeddings,\n hidden_size)\n self.token_type_embeddings = nn.Embedding(type_vocab_size, hidden_size)\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(hidden_dropout_prob)\n def forward(self, input_ids, token_type_ids=None):\n seq_length = input_ids.shape[1]\n position_ids = paddle.arange(end=seq_length, dtype=\"int64\")\n position_ids = position_ids.unsqueeze(0).expand_as(input_ids)\n if token_type_ids is None:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:33-52"
+ },
+ "6093": {
+ "file_id": 487,
+ "content": "BertEmbeddings initializes word, position, and token_type embeddings for a given vocabulary size, maximum position embedding size, type vocab size, hidden size, and hidden dropout probability. Forward function uses input ids and token type ids to generate position ids and then combines the different embeddings with layer normalization and dropout.",
+ "type": "comment"
+ },
+ "6094": {
+ "file_id": 487,
+ "content": " token_type_ids = paddle.zeros_like(input_ids)\n words_embeddings = self.word_embeddings(input_ids) #8,36 -> 8,36,768\n position_embeddings = self.position_embeddings(\n position_ids) #8,36 -> 8,36,768\n token_type_embeddings = self.token_type_embeddings(\n token_type_ids) #8,36 -> 8,36,768\n embeddings = words_embeddings + position_embeddings + token_type_embeddings\n embeddings = self.LayerNorm(embeddings)\n embeddings = self.dropout(embeddings)\n return embeddings\nclass BertImageEmbeddings(nn.Layer):\n def __init__(self, v_feature_size, v_hidden_size, v_hidden_dropout_prob):\n super(BertImageEmbeddings, self).__init__()\n self.image_embeddings = nn.Linear(v_feature_size, v_hidden_size)\n self.image_location_embeddings = nn.Linear(5, v_hidden_size)\n self.LayerNorm = nn.LayerNorm(v_hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(v_hidden_dropout_prob)\n def forward(self, input_ids, input_loc):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:53-75"
+ },
+ "6095": {
+ "file_id": 487,
+ "content": "This code defines the `ActBert` class, which is a backbone model for video action recognition. It initializes word embeddings, position embeddings, and token type embeddings. The class also includes a forward function that combines these embeddings, applies layer normalization and dropout, and returns the result. Additionally, there's the `BertImageEmbeddings` class which takes image features and their locations as input and uses linear layers to generate embeddings for both, followed by layer normalization and dropout.",
+ "type": "comment"
+ },
+ "6096": {
+ "file_id": 487,
+ "content": " img_embeddings = self.image_embeddings(\n input_ids) #8,37,2048 -> 8,37,1024\n loc_embeddings = self.image_location_embeddings(\n input_loc) #8,37,5 -> 8,37,1024\n embeddings = self.LayerNorm(img_embeddings + loc_embeddings)\n embeddings = self.dropout(embeddings)\n return embeddings # shape: bs*seq_len*hs\nclass BertActionEmbeddings(nn.Layer):\n def __init__(self, a_feature_size, a_hidden_size, a_hidden_dropout_prob):\n super(BertActionEmbeddings, self).__init__()\n self.action_embeddings = nn.Linear(a_feature_size, a_hidden_size)\n self.LayerNorm = nn.LayerNorm(a_hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(a_hidden_dropout_prob)\n def forward(self, input_ids):\n action_embeddings = self.action_embeddings(\n input_ids) #8,5,2048 -> 8,5,768\n embeddings = self.LayerNorm(action_embeddings)\n embeddings = self.dropout(embeddings)\n return embeddings\nclass BertSelfAttention(nn.Layer):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:76-100"
+ },
+ "6097": {
+ "file_id": 487,
+ "content": "This code defines two classes: `BertActionEmbeddings` and `BertSelfAttention`. The former takes action features as input, linearly projects them into hidden states, normalizes these using LayerNorm, applies dropout, and returns the embeddings. The latter performs self-attention over the embeddings produced by `BertActionEmbeddings`, applies a feed-forward network, applies LayerNorm, and finally applies dropout before returning the output.",
+ "type": "comment"
+ },
+ "6098": {
+ "file_id": 487,
+ "content": " def __init__(self, hidden_size, num_attention_heads,\n attention_probs_dropout_prob):\n super(BertSelfAttention, self).__init__()\n if hidden_size % num_attention_heads != 0:\n raise ValueError(\n \"The hidden size (%d) is not a multiple of the number of attention \"\n \"heads (%d)\" % (hidden_size, num_attention_heads))\n self.num_attention_heads = num_attention_heads\n self.attention_head_size = int(hidden_size / num_attention_heads)\n self.all_head_size = self.num_attention_heads * self.attention_head_size\n self.query = nn.Linear(hidden_size, self.all_head_size)\n self.key = nn.Linear(hidden_size, self.all_head_size)\n self.value = nn.Linear(hidden_size, self.all_head_size)\n self.dropout = nn.Dropout(attention_probs_dropout_prob)\n def transpose_for_scores(self, x):\n new_x_shape = x.shape[:-1] + [\n self.num_attention_heads,\n self.attention_head_size,\n ]\n x = x.reshape(new_x_shape)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:101-123"
+ },
+ "6099": {
+ "file_id": 487,
+ "content": "This code defines a BertSelfAttention class with parameters: hidden_size, num_attention_heads, and attention_probs_dropout_prob. It checks if the hidden size is divisible by the number of attention heads. If not, it raises a ValueError. Then, it calculates attention_head_size and all_head_size. Finally, it initializes query, key, value linear layers and dropout layer for attention probabilities.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/61.json b/docs/data/61.json
new file mode 100644
index 000000000..332fc87b0
--- /dev/null
+++ b/docs/data/61.json
@@ -0,0 +1,551 @@
+{
+ "6100": {
+ "file_id": 487,
+ "content": " return x.transpose((0, 2, 1, 3))\n def forward(self, hidden_states, attention_mask):\n mixed_query_layer = self.query(hidden_states)\n mixed_key_layer = self.key(hidden_states)\n mixed_value_layer = self.value(hidden_states)\n query_layer = self.transpose_for_scores(mixed_query_layer)\n key_layer = self.transpose_for_scores(mixed_key_layer)\n value_layer = self.transpose_for_scores(mixed_value_layer)\n # Take the dot product between \"query\" and \"key\" to get the raw attention scores.\n attention_scores = paddle.matmul(query_layer,\n key_layer.transpose((0, 1, 3, 2)))\n attention_scores = attention_scores / math.sqrt(\n self.attention_head_size)\n # Apply the attention mask is (precomputed for all layers in BertModel forward() function)\n attention_scores = attention_scores + attention_mask\n # Normalize the attention scores to probabilities.\n attention_probs = nn.Softmax(axis=-1)(attention_scores)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:124-144"
+ },
+ "6101": {
+ "file_id": 487,
+ "content": "This code performs multi-head attention in an attention mechanism. It transposes the query, key, and value layers before calculating raw attention scores via dot product. The results are then normalized into probabilities using softmax. The attention mask is applied to the attention scores for masked self-attention.",
+ "type": "comment"
+ },
+ "6102": {
+ "file_id": 487,
+ "content": " # This is actually dropping out entire tokens to attend to, which might\n # seem a bit unusual, but is taken from the original Transformer paper.\n attention_probs = self.dropout(attention_probs)\n context_layer = paddle.matmul(attention_probs, value_layer)\n context_layer = context_layer.transpose((0, 2, 1, 3))\n new_context_layer_shape = context_layer.shape[:-2] + [\n self.all_head_size\n ]\n context_layer = context_layer.reshape(new_context_layer_shape)\n return context_layer, attention_probs\nclass BertSelfOutput(nn.Layer):\n def __init__(self, hidden_size, hidden_dropout_prob):\n super(BertSelfOutput, self).__init__()\n self.dense = nn.Linear(hidden_size, hidden_size)\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(hidden_dropout_prob)\n def forward(self, hidden_states, input_tensor):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.dropout(hidden_states)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:146-169"
+ },
+ "6103": {
+ "file_id": 487,
+ "content": "This code defines a BertSelfOutput layer that takes input hidden states, applies linear transformation and dropout for regularization, then passes the output through layer normalization.",
+ "type": "comment"
+ },
+ "6104": {
+ "file_id": 487,
+ "content": " hidden_states = self.LayerNorm(hidden_states + input_tensor)\n return hidden_states\nclass BertAttention(nn.Layer):\n def __init__(self, hidden_size, hidden_dropout_prob, num_attention_heads,\n attention_probs_dropout_prob):\n super(BertAttention, self).__init__()\n self.self = BertSelfAttention(hidden_size, num_attention_heads,\n attention_probs_dropout_prob)\n self.output = BertSelfOutput(hidden_size, hidden_dropout_prob)\n def forward(self, input_tensor, attention_mask):\n self_output, attention_probs = self.self(input_tensor, attention_mask)\n attention_output = self.output(self_output, input_tensor)\n return attention_output, attention_probs\nclass BertIntermediate(nn.Layer):\n def __init__(self, hidden_size, intermediate_size, hidden_act):\n super(BertIntermediate, self).__init__()\n self.dense = nn.Linear(hidden_size, intermediate_size)\n if isinstance(hidden_act, str) or (sys.version_info[0] == 2",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:170-192"
+ },
+ "6105": {
+ "file_id": 487,
+ "content": "This code defines three classes: `ActBert`, `BertAttention`, and `BertIntermediate`. \n\n`ActBert` appears to be a model that includes `BertAttention` and `BertIntermediate` as its layers. The `BertAttention` class defines forward function for attention mechanism, which takes in an input tensor and attention mask, and returns output and attention probabilities. The `BertIntermediate` class appears to be a dense layer with linear activation function.",
+ "type": "comment"
+ },
+ "6106": {
+ "file_id": 487,
+ "content": " and isinstance(hidden_act, str)):\n self.intermediate_act_fn = ACT2FN[hidden_act]\n else:\n self.intermediate_act_fn = hidden_act\n def forward(self, hidden_states):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.intermediate_act_fn(hidden_states)\n return hidden_states\nclass BertOutput(nn.Layer):\n def __init__(self, intermediate_size, hidden_size, hidden_dropout_prob):\n super(BertOutput, self).__init__()\n self.dense = nn.Linear(intermediate_size, hidden_size)\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(hidden_dropout_prob)\n def forward(self, hidden_states, input_tensor):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.dropout(hidden_states)\n hidden_states = self.LayerNorm(hidden_states + input_tensor)\n return hidden_states\nclass BertEntAttention(nn.Layer):\n \"\"\"Core mudule of tangled transformer.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:193-219"
+ },
+ "6107": {
+ "file_id": 487,
+ "content": "Code defines a class for an attention-based transformer model. It includes an intermediate activation function and forward pass layers for processing input, applying dropout regularization, and normalizing outputs. The BertEntAttention layer is the core module for the transformer.",
+ "type": "comment"
+ },
+ "6108": {
+ "file_id": 487,
+ "content": " \"\"\"\n def __init__(\n self,\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n bi_num_attention_heads,\n ):\n super(BertEntAttention, self).__init__()\n if bi_hidden_size % bi_num_attention_heads != 0:\n raise ValueError(\n \"The hidden size (%d) is not a multiple of the number of attention \"\n \"heads (%d)\" % (bi_hidden_size, bi_num_attention_heads))\n self.num_attention_heads = bi_num_attention_heads\n self.attention_head_size = int(bi_hidden_size / bi_num_attention_heads)\n self.all_head_size = self.num_attention_heads * self.attention_head_size\n # self attention layers for vision input\n self.query1 = nn.Linear(v_hidden_size, self.all_head_size)\n self.key1 = nn.Linear(v_hidden_size, self.all_head_size)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:220-246"
+ },
+ "6109": {
+ "file_id": 487,
+ "content": "This code defines a BertEntAttention class with parameters for hidden size, vision input hidden size, attention probabilities dropout probabilities, and bi-directional hidden size. It also checks if the hidden size is a multiple of the number of attention heads. The class initializes attributes such as the number of attention heads, attention head size, all head size, and linear layers for self-attention in vision input.",
+ "type": "comment"
+ },
+ "6110": {
+ "file_id": 487,
+ "content": " self.value1 = nn.Linear(v_hidden_size, self.all_head_size)\n self.dropout1 = nn.Dropout(v_attention_probs_dropout_prob)\n # self attention layers for text input\n self.query2 = nn.Linear(hidden_size, self.all_head_size)\n self.key2 = nn.Linear(hidden_size, self.all_head_size)\n self.value2 = nn.Linear(hidden_size, self.all_head_size)\n self.dropout2 = nn.Dropout(attention_probs_dropout_prob)\n # self attention layers for action input\n self.query3 = nn.Linear(a_hidden_size, self.all_head_size)\n self.key3 = nn.Linear(a_hidden_size, self.all_head_size)\n self.value3 = nn.Linear(a_hidden_size, self.all_head_size)\n self.dropout3 = nn.Dropout(a_attention_probs_dropout_prob)\n # self attention layers for action_text\n self.key_at = nn.Linear(bi_hidden_size, self.all_head_size)\n self.value_at = nn.Linear(bi_hidden_size, self.all_head_size)\n self.dropout_at = nn.Dropout(av_attention_probs_dropout_prob)\n # self attention layers for action_vision",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:247-267"
+ },
+ "6111": {
+ "file_id": 487,
+ "content": "This code defines layers for self-attention in the ACTBERT model, including linear and dropout layers for text, action, action_text, and action_vision inputs.",
+ "type": "comment"
+ },
+ "6112": {
+ "file_id": 487,
+ "content": " self.key_av = nn.Linear(bi_hidden_size, self.all_head_size)\n self.value_av = nn.Linear(bi_hidden_size, self.all_head_size)\n self.dropout_av = nn.Dropout(at_attention_probs_dropout_prob)\n def transpose_for_scores(self, x):\n new_x_shape = x.shape[:-1] + [\n self.num_attention_heads,\n self.attention_head_size,\n ]\n x = x.reshape(new_x_shape)\n return x.transpose((0, 2, 1, 3))\n def forward(\n self,\n input_tensor1,\n attention_mask1,\n input_tensor2,\n attention_mask2,\n input_tensor3,\n attention_mask3,\n ):\n # for vision input.\n mixed_query_layer1 = self.query1(input_tensor1)\n mixed_key_layer1 = self.key1(input_tensor1)\n mixed_value_layer1 = self.value1(input_tensor1)\n query_layer1 = self.transpose_for_scores(mixed_query_layer1)\n key_layer1 = self.transpose_for_scores(mixed_key_layer1)\n value_layer1 = self.transpose_for_scores(mixed_value_layer1)\n # for text input:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:268-299"
+ },
+ "6113": {
+ "file_id": 487,
+ "content": "This code defines a model for attention mechanism in a transformer architecture, used for both vision and text inputs. The key steps involve creating linear layers for keys and values, applying dropout, transposing the input tensors for scoring, and forwarding the input through these operations for both vision and text inputs separately.",
+ "type": "comment"
+ },
+ "6114": {
+ "file_id": 487,
+ "content": " mixed_query_layer2 = self.query2(input_tensor2)\n mixed_key_layer2 = self.key2(input_tensor2)\n mixed_value_layer2 = self.value2(input_tensor2)\n query_layer2 = self.transpose_for_scores(mixed_query_layer2)\n key_layer2 = self.transpose_for_scores(mixed_key_layer2)\n value_layer2 = self.transpose_for_scores(mixed_value_layer2)\n # for action input:\n mixed_query_layer3 = self.query3(input_tensor3)\n mixed_key_layer3 = self.key3(input_tensor3)\n mixed_value_layer3 = self.value3(input_tensor3)\n query_layer3 = self.transpose_for_scores(mixed_query_layer3)\n key_layer3 = self.transpose_for_scores(mixed_key_layer3)\n value_layer3 = self.transpose_for_scores(mixed_value_layer3)\n def do_attention(query_layer, key_layer, value_layer, attention_mask,\n dropout):\n \"\"\" compute attention \"\"\"\n attention_scores = paddle.matmul(query_layer,\n key_layer.transpose((0, 1, 3, 2)))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:300-321"
+ },
+ "6115": {
+ "file_id": 487,
+ "content": "This code is performing multi-head attention operation. It first separates the input tensor into two parts, with each part going through its own set of linear layers to create query, key, and value tensors. Then it transposes the query and key tensors before computing attention scores by taking the dot product of the transposed query and key tensors.",
+ "type": "comment"
+ },
+ "6116": {
+ "file_id": 487,
+ "content": " attention_scores = attention_scores / math.sqrt(\n self.attention_head_size)\n attention_scores = attention_scores + attention_mask\n # Normalize the attention scores to probabilities.\n attention_probs = nn.Softmax(axis=-1)(attention_scores)\n # This is actually dropping out entire tokens to attend to, which might\n # seem a bit unusual, but is taken from the original Transformer paper.\n attention_probs = dropout(attention_probs)\n context_layer = paddle.matmul(attention_probs, value_layer)\n context_layer = context_layer.transpose((0, 2, 1, 3))\n new_context_layer_shape = context_layer.shape[:-2] + [\n self.all_head_size\n ]\n context_layer = context_layer.reshape(new_context_layer_shape)\n return context_layer\n context_av = do_attention(query_layer3, key_layer1, value_layer1,\n attention_mask1, self.dropout_av)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:322-342"
+ },
+ "6117": {
+ "file_id": 487,
+ "content": "This code calculates attention scores between queries, keys, and values, normalizes them to probabilities using softmax, applies dropout, performs matrix multiplication with the values, transposes the result, reshapes it, and returns the context layer. It follows the Transformer paper's approach of dropping out entire tokens to attend to.",
+ "type": "comment"
+ },
+ "6118": {
+ "file_id": 487,
+ "content": " context_at = do_attention(query_layer3, key_layer2, value_layer2,\n attention_mask2, self.dropout_at)\n context_key_av = self.key_av(context_av).transpose((0, 2, 1))\n # interpolate only support 4-D tensor now.\n context_key_av = F.interpolate(context_key_av.unsqueeze(-1),\n size=(key_layer2.shape[2],\n 1)).squeeze(-1)\n context_key_av = self.transpose_for_scores(\n context_key_av.transpose((0, 2, 1)))\n key_layer2 = key_layer2 + context_key_av\n context_key_at = self.key_at(context_at).transpose((0, 2, 1))\n context_key_at = F.interpolate(context_key_at.unsqueeze(-1),\n size=(key_layer1.shape[2],\n 1)).squeeze(-1)\n context_key_at = self.transpose_for_scores(\n context_key_at.transpose((0, 2, 1)))\n key_layer1 = key_layer1 + context_key_at",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:343-361"
+ },
+ "6119": {
+ "file_id": 487,
+ "content": "This code is performing attention mechanism for multi-scale context fusion. It uses interpolation to resize the context features, then adds them to the original key layers (key_layer2 and key_layer1). The purpose of this is to incorporate contextual information from different scales into the model's understanding.",
+ "type": "comment"
+ },
+ "6120": {
+ "file_id": 487,
+ "content": " context_val_av = self.value_at(context_av).transpose((0, 2, 1))\n context_val_av = F.interpolate(context_val_av.unsqueeze(-1),\n size=(value_layer2.shape[2],\n 1)).squeeze(-1)\n context_val_av = self.transpose_for_scores(\n context_val_av.transpose((0, 2, 1)))\n value_layer2 = value_layer2 + context_val_av\n context_val_at = self.value_at(context_at).transpose((0, 2, 1))\n context_val_at = F.interpolate(context_val_at.unsqueeze(-1),\n size=(value_layer1.shape[2],\n 1)).squeeze(-1)\n context_val_at = self.transpose_for_scores(\n context_val_at.transpose((0, 2, 1)))\n value_layer1 = value_layer1 + context_val_at\n context_layer1 = do_attention(query_layer1, key_layer1, value_layer1,\n attention_mask1, self.dropout1)\n context_layer2 = do_attention(query_layer2, key_layer2, value_layer2,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:363-381"
+ },
+ "6121": {
+ "file_id": 487,
+ "content": "This code snippet is performing cross-attention in a transformer model. It first interpolates and adds context vectors to value layers, then applies attention mechanisms to compute context layers. This process helps to capture dependencies between different parts of the input data effectively.",
+ "type": "comment"
+ },
+ "6122": {
+ "file_id": 487,
+ "content": " attention_mask2, self.dropout2)\n context_layer3 = do_attention(query_layer3, key_layer3, value_layer3,\n attention_mask3, self.dropout3)\n return context_layer1, context_layer2, context_layer3 # vision, text, action\nclass BertEntOutput(nn.Layer):\n def __init__(\n self,\n bi_hidden_size,\n hidden_size,\n v_hidden_size,\n v_hidden_dropout_prob,\n hidden_dropout_prob,\n ):\n super(BertEntOutput, self).__init__()\n self.dense1 = nn.Linear(bi_hidden_size, v_hidden_size)\n self.LayerNorm1 = nn.LayerNorm(v_hidden_size, epsilon=1e-12)\n self.dropout1 = nn.Dropout(v_hidden_dropout_prob)\n self.dense2 = nn.Linear(bi_hidden_size, hidden_size)\n self.LayerNorm2 = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout2 = nn.Dropout(hidden_dropout_prob)\n self.dense3 = nn.Linear(bi_hidden_size, hidden_size)\n self.LayerNorm3 = nn.LayerNorm(hidden_size, epsilon=1e-12)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:382-409"
+ },
+ "6123": {
+ "file_id": 487,
+ "content": "The code defines a class \"BertEntOutput\" with several layers including dense and dropout. It uses bi-hidden size, hidden size, v_hidden_size, and corresponding dropout probabilities for initializing the layers. This class seems to be part of a model architecture where it performs layer normalization, applies dropout regularization, and linear transformations to process input data.",
+ "type": "comment"
+ },
+ "6124": {
+ "file_id": 487,
+ "content": " self.dropout3 = nn.Dropout(hidden_dropout_prob)\n def forward(\n self,\n hidden_states1,\n input_tensor1,\n hidden_states2,\n input_tensor2,\n hidden_states3,\n input_tensor3,\n ):\n context_state1 = self.dense1(hidden_states1)\n context_state1 = self.dropout1(context_state1)\n context_state2 = self.dense2(hidden_states2)\n context_state2 = self.dropout2(context_state2)\n context_state3 = self.dense3(hidden_states3)\n context_state3 = self.dropout3(context_state3)\n hidden_states1 = self.LayerNorm1(context_state1 + input_tensor1)\n hidden_states2 = self.LayerNorm2(context_state2 + input_tensor2)\n hidden_states3 = self.LayerNorm3(context_state3 + input_tensor3)\n return hidden_states1, hidden_states2, hidden_states3\nclass BertLayer(nn.Layer):\n def __init__(self, hidden_size, intermediate_size, hidden_act,\n hidden_dropout_prob, num_attention_heads,\n attention_probs_dropout_prob):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:410-440"
+ },
+ "6125": {
+ "file_id": 487,
+ "content": "This code defines a BertLayer class with Dropout layers and dense layers, performing attention mechanism. The forward method applies these layers to hidden_states1,2,3, and returns the updated hidden states after adding them with their corresponding input tensors.",
+ "type": "comment"
+ },
+ "6126": {
+ "file_id": 487,
+ "content": " super(BertLayer, self).__init__()\n self.attention = BertAttention(hidden_size, hidden_dropout_prob,\n num_attention_heads,\n attention_probs_dropout_prob)\n self.intermediate = BertIntermediate(hidden_size, intermediate_size,\n hidden_act)\n self.output = BertOutput(intermediate_size, hidden_size,\n hidden_dropout_prob)\n def forward(self, hidden_states, attention_mask):\n attention_output, attention_probs = self.attention(\n hidden_states, attention_mask)\n intermediate_output = self.intermediate(attention_output)\n layer_output = self.output(intermediate_output, attention_output)\n return layer_output, attention_probs\nclass BertConnectionLayer(nn.Layer):\n def __init__(self, hidden_size, v_hidden_size, a_hidden_size,\n bi_hidden_size, bi_num_attention_heads,\n attention_probs_dropout_prob, v_attention_probs_dropout_prob,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:441-461"
+ },
+ "6127": {
+ "file_id": 487,
+ "content": "This code defines a BertLayer class and a BertConnectionLayer class. The BertLayer class includes an attention layer, an intermediate layer, and an output layer. It has a forward function that performs the calculations for these layers. The BertConnectionLayer class is a subclass of nn.Layer with various parameters for hidden sizes and attention dropout probabilities.",
+ "type": "comment"
+ },
+ "6128": {
+ "file_id": 487,
+ "content": " a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob, intermediate_size,\n v_intermediate_size, a_intermediate_size, hidden_act,\n v_hidden_act, a_hidden_act, hidden_dropout_prob,\n v_hidden_dropout_prob, a_hidden_dropout_prob):\n super(BertConnectionLayer, self).__init__()\n self.ent_attention = BertEntAttention(\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n bi_num_attention_heads,\n )\n self.ent_output = BertEntOutput(\n bi_hidden_size,\n hidden_size,\n v_hidden_size,\n v_hidden_dropout_prob,\n hidden_dropout_prob,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:462-487"
+ },
+ "6129": {
+ "file_id": 487,
+ "content": "This code initializes a BertConnectionLayer object with various parameters including hidden size, attention probabilities dropout probability, and intermediate size. It also initializes two other objects: BertEntAttention and BertEntOutput. The BertEntAttention object is responsible for performing entity-based attention while the BertEntOutput object is responsible for producing the output of the connection layer.",
+ "type": "comment"
+ },
+ "6130": {
+ "file_id": 487,
+ "content": " )\n self.v_intermediate = BertIntermediate(v_hidden_size,\n v_intermediate_size,\n v_hidden_act)\n self.v_output = BertOutput(v_intermediate_size, v_hidden_size,\n v_hidden_dropout_prob)\n self.t_intermediate = BertIntermediate(hidden_size, intermediate_size,\n hidden_act)\n self.t_output = BertOutput(intermediate_size, hidden_size,\n hidden_dropout_prob)\n self.a_intermediate = BertIntermediate(a_hidden_size,\n a_intermediate_size,\n a_hidden_act)\n self.a_output = BertOutput(a_intermediate_size, a_hidden_size,\n a_hidden_dropout_prob)\n def forward(\n self,\n input_tensor1,\n attention_mask1,\n input_tensor2,\n attention_mask2,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:488-512"
+ },
+ "6131": {
+ "file_id": 487,
+ "content": "This code defines a model with three input streams (v, t, and a) and initializes intermediate layers and output layers for each stream. The forward function takes in two pairs of input tensors and attention masks for each stream.",
+ "type": "comment"
+ },
+ "6132": {
+ "file_id": 487,
+ "content": " input_tensor3,\n attention_mask3,\n ):\n ent_output1, ent_output2, ent_output3 = self.ent_attention(\n input_tensor1, attention_mask1, input_tensor2, attention_mask2,\n input_tensor3, attention_mask3)\n attention_output1, attention_output2, attention_output3 = self.ent_output(\n ent_output1, input_tensor1, ent_output2, input_tensor2, ent_output3,\n input_tensor3)\n intermediate_output1 = self.v_intermediate(attention_output1)\n layer_output1 = self.v_output(intermediate_output1, attention_output1)\n intermediate_output2 = self.t_intermediate(attention_output2)\n layer_output2 = self.t_output(intermediate_output2, attention_output2)\n intermediate_output3 = self.a_intermediate(attention_output3)\n layer_output3 = self.a_output(intermediate_output3, attention_output3)\n return layer_output1, layer_output2, layer_output3\nclass BertEncoder(nn.Layer):\n \"\"\"\n ActBert Encoder, consists 3 pathway of multi-BertLayers and BertConnectionLayer.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:513-539"
+ },
+ "6133": {
+ "file_id": 487,
+ "content": "This function computes the layer outputs for three different pathways using the ActBert Encoder. It utilizes attention masks and input tensors for each pathway, passing them through various intermediate layers before returning the final layer outputs. The BertEncoder class represents a combination of three multi-BertLayers and BertConnectionLayer.",
+ "type": "comment"
+ },
+ "6134": {
+ "file_id": 487,
+ "content": " \"\"\"\n def __init__(\n self,\n v_ent_attention_id,\n t_ent_attention_id,\n a_ent_attention_id,\n fixed_t_layer,\n fixed_v_layer,\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n intermediate_size,\n v_intermediate_size,\n a_intermediate_size,\n hidden_act,\n v_hidden_act,\n a_hidden_act,\n hidden_dropout_prob,\n v_hidden_dropout_prob,\n a_hidden_dropout_prob,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n num_attention_heads,\n v_num_attention_heads,\n a_num_attention_heads,\n bi_num_attention_heads,\n num_hidden_layers,\n v_num_hidden_layers,\n a_num_hidden_layers,\n ):\n super(BertEncoder, self).__init__()\n self.v_ent_attention_id = v_ent_attention_id\n self.t_ent_attention_id = t_ent_attention_id",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:540-576"
+ },
+ "6135": {
+ "file_id": 487,
+ "content": "This code defines the `BertEncoder` class, which initializes various parameters for the BERT model's encoder. These parameters include attention IDs, fixed layer positions, hidden sizes, and dropout probabilities for different components of the model. The class extends `super(BertEncoder, self).__init__()`, indicating it inherits from another class.",
+ "type": "comment"
+ },
+ "6136": {
+ "file_id": 487,
+ "content": " self.a_ent_attention_id = a_ent_attention_id\n self.fixed_t_layer = fixed_t_layer\n self.fixed_v_layer = fixed_v_layer\n layer = BertLayer(hidden_size, intermediate_size, hidden_act,\n hidden_dropout_prob, num_attention_heads,\n attention_probs_dropout_prob)\n v_layer = BertLayer(v_hidden_size, v_intermediate_size, v_hidden_act,\n v_hidden_dropout_prob, v_num_attention_heads,\n v_attention_probs_dropout_prob)\n a_layer = BertLayer(a_hidden_size, a_intermediate_size, a_hidden_act,\n a_hidden_dropout_prob, a_num_attention_heads,\n a_attention_probs_dropout_prob)\n connect_layer = BertConnectionLayer(\n hidden_size, v_hidden_size, a_hidden_size, bi_hidden_size,\n bi_num_attention_heads, attention_probs_dropout_prob,\n v_attention_probs_dropout_prob, a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob, at_attention_probs_dropout_prob,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:577-594"
+ },
+ "6137": {
+ "file_id": 487,
+ "content": "This code initializes three BertLayer objects and one BertConnectionLayer object with different hidden sizes, intermediate sizes, activation functions, dropout probabilities, and attention head numbers. These layers will be used for encoding input sequences in the Actor-Critic Transformer (ACT) model.",
+ "type": "comment"
+ },
+ "6138": {
+ "file_id": 487,
+ "content": " intermediate_size, v_intermediate_size, a_intermediate_size,\n hidden_act, v_hidden_act, a_hidden_act, hidden_dropout_prob,\n v_hidden_dropout_prob, a_hidden_dropout_prob)\n self.layer = nn.LayerList(\n [copy.deepcopy(layer) for _ in range(num_hidden_layers)]) #12\n self.v_layer = nn.LayerList(\n [copy.deepcopy(v_layer) for _ in range(v_num_hidden_layers)]) #2\n self.a_layer = nn.LayerList(\n [copy.deepcopy(a_layer) for _ in range(a_num_hidden_layers)]) #3\n self.c_layer = nn.LayerList([\n copy.deepcopy(connect_layer) for _ in range(len(v_ent_attention_id))\n ] #2 [0,1]\n )\n def forward(\n self,\n txt_embedding,\n image_embedding,\n action_embedding,\n txt_attention_mask,\n image_attention_mask,\n action_attention_mask,\n output_all_encoded_layers=True,\n ):\n v_start, a_start, t_start = 0, 0, 0\n count = 0\n all_encoder_layers_t = []",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:595-622"
+ },
+ "6139": {
+ "file_id": 487,
+ "content": "This code defines a model with separate layers for text (txt_layer), vision (v_layer), and action (a_layer) embeddings. It also includes a connect_layer to combine the visual and action information for attention masks. The forward method takes input embeddings, attention masks, and an optional parameter to output all encoded layers.",
+ "type": "comment"
+ },
+ "6140": {
+ "file_id": 487,
+ "content": " all_encoder_layers_v = []\n all_encoder_layers_a = []\n for v_layer_id, a_layer_id, t_layer_id in zip(self.v_ent_attention_id,\n self.a_ent_attention_id,\n self.t_ent_attention_id):\n v_end = v_layer_id\n a_end = a_layer_id\n t_end = t_layer_id\n assert self.fixed_t_layer <= t_end\n assert self.fixed_v_layer <= v_end\n ### region embedding\n for idx in range(v_start,\n self.fixed_v_layer): #两次训练，这个循环都没有进去 #前面的层固定住\n with paddle.no_grad():\n image_embedding, image_attention_probs = self.v_layer[idx](\n image_embedding, image_attention_mask)\n v_start = self.fixed_v_layer\n for idx in range(v_start, v_end):\n image_embedding, image_attention_probs = self.v_layer[idx](\n image_embedding, image_attention_mask)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:623-645"
+ },
+ "6141": {
+ "file_id": 487,
+ "content": "This code initializes empty lists for all encoder layers in vision and audio. It then iterates through the given layer IDs, splitting them into vision (v), audio (a), and time (t) layers. The code asserts that the fixed time layer is less than or equal to the last time layer, and the fixed vision layer is less than or equal to the last vision layer. Next, it iterates through all vision layers from the start index up to but not including the fixed vision layer. Inside this loop, it applies the corresponding vision layer to the image embedding and attention mask using Paddle's no_grad context manager. Finally, it loops over all vision layers from the start index to the end index (fixed vision layer excluded), applying the corresponding vision layer to the image embedding and attention mask.",
+ "type": "comment"
+ },
+ "6142": {
+ "file_id": 487,
+ "content": " ### action embedding\n for idx in range(a_start, a_end):\n action_embedding, action_attention_probs = self.a_layer[idx](\n action_embedding, action_attention_mask)\n ### text embedding\n for idx in range(t_start, self.fixed_t_layer):\n with paddle.no_grad():\n txt_embedding, txt_attention_probs = self.layer[idx](\n txt_embedding, txt_attention_mask)\n t_start = self.fixed_t_layer\n for idx in range(t_start, t_end):\n txt_embedding, txt_attention_probs = self.layer[idx](\n txt_embedding, txt_attention_mask)\n image_embedding, txt_embedding, action_embedding = self.c_layer[\n count](image_embedding, image_attention_mask, txt_embedding,\n txt_attention_mask, action_embedding,\n action_attention_mask)\n v_start = v_end\n t_start = t_end\n a_start = a_end",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:647-669"
+ },
+ "6143": {
+ "file_id": 487,
+ "content": "This code is performing multi-modal embedding by separately handling action, text, and image embeddings. It iterates through layers for each modality to compute the embeddings and attention probs. Finally, it combines the embeddings in a specific order before potentially updating start/end indices.",
+ "type": "comment"
+ },
+ "6144": {
+ "file_id": 487,
+ "content": " count += 1\n if output_all_encoded_layers:\n all_encoder_layers_t.append(txt_embedding)\n all_encoder_layers_v.append(image_embedding)\n all_encoder_layers_a.append(action_embedding)\n for idx in range(v_start, len(self.v_layer)): # 1\n image_embedding, image_attention_probs = self.v_layer[idx](\n image_embedding, image_attention_mask)\n for idx in range(a_start, len(self.a_layer)):\n action_embedding, action_attention_probs = self.a_layer[idx](\n action_embedding, action_attention_mask)\n for idx in range(t_start, len(self.layer)):\n txt_embedding, txt_attention_probs = self.layer[idx](\n txt_embedding, txt_attention_mask)\n # add the end part to finish.\n if not output_all_encoded_layers:\n all_encoder_layers_t.append(txt_embedding) #8, 36, 768\n all_encoder_layers_v.append(image_embedding) #8, 37, 1024\n all_encoder_layers_a.append(action_embedding) #8, 5, 768",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:670-693"
+ },
+ "6145": {
+ "file_id": 487,
+ "content": "This code is responsible for encoding text, image, and action inputs in a neural network model. It iterates over the layers of each input type to produce their respective encoded representations. If output_all_encoded_layers is set to True, it appends all intermediate encoded layers to separate lists; otherwise, only the final encoded layer is stored. This allows for flexibility in selecting which encoded layers to use in further processing.",
+ "type": "comment"
+ },
+ "6146": {
+ "file_id": 487,
+ "content": " return all_encoder_layers_t, all_encoder_layers_v, all_encoder_layers_a\nclass BertPooler(nn.Layer):\n \"\"\" \"Pool\" the model by simply taking the hidden state corresponding\n to the first token.\n \"\"\"\n def __init__(self, hidden_size, bi_hidden_size):\n super(BertPooler, self).__init__()\n self.dense = nn.Linear(hidden_size, bi_hidden_size)\n self.activation = nn.ReLU()\n def forward(self, hidden_states):\n first_token_tensor = hidden_states[:, 0] #8, 768\n pooled_output = self.dense(first_token_tensor)\n pooled_output = self.activation(pooled_output)\n return pooled_output\nclass BertModel(nn.Layer):\n def __init__(\n self,\n vocab_size,\n max_position_embeddings,\n type_vocab_size,\n v_feature_size,\n a_feature_size,\n num_hidden_layers,\n v_num_hidden_layers,\n a_num_hidden_layers,\n v_ent_attention_id,\n t_ent_attention_id,\n a_ent_attention_id,\n fixed_t_layer,\n fixed_v_layer,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:695-729"
+ },
+ "6147": {
+ "file_id": 487,
+ "content": "The code defines a BertPooler class that pools the model by taking the hidden state corresponding to the first token. It also includes a BertModel class with various parameters for initializing the BERT model, including vocab size, max position embeddings, type vocab size, and feature sizes for different entities (v, a). The code also defines attention IDs and fixed layers for tokens and aspects.",
+ "type": "comment"
+ },
+ "6148": {
+ "file_id": 487,
+ "content": " hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n intermediate_size,\n v_intermediate_size,\n a_intermediate_size,\n hidden_act,\n v_hidden_act,\n a_hidden_act,\n hidden_dropout_prob,\n v_hidden_dropout_prob,\n a_hidden_dropout_prob,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n num_attention_heads,\n v_num_attention_heads,\n a_num_attention_heads,\n bi_num_attention_heads,\n ):\n super(BertModel, self).__init__()\n # initilize word embedding\n self.embeddings = BertEmbeddings(vocab_size, max_position_embeddings,\n type_vocab_size, hidden_size,\n hidden_dropout_prob)\n # initlize the region embedding\n self.v_embeddings = BertImageEmbeddings(v_feature_size, v_hidden_size,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:730-759"
+ },
+ "6149": {
+ "file_id": 487,
+ "content": "This code is initializing a BertModel class with various parameters including hidden size, attention-related parameters, and embedding types. It uses the superclass constructor to initialize the base model and then further customizes it by adding word and image embeddings.",
+ "type": "comment"
+ },
+ "6150": {
+ "file_id": 487,
+ "content": " v_hidden_dropout_prob)\n # initlize the action embedding\n self.a_embeddings = BertActionEmbeddings(a_feature_size, a_hidden_size,\n a_hidden_dropout_prob)\n self.encoder = BertEncoder(\n v_ent_attention_id, t_ent_attention_id, a_ent_attention_id,\n fixed_t_layer, fixed_v_layer, hidden_size, v_hidden_size,\n a_hidden_size, bi_hidden_size, intermediate_size,\n v_intermediate_size, a_intermediate_size, hidden_act, v_hidden_act,\n a_hidden_act, hidden_dropout_prob, v_hidden_dropout_prob,\n a_hidden_dropout_prob, attention_probs_dropout_prob,\n v_attention_probs_dropout_prob, a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob, at_attention_probs_dropout_prob,\n num_attention_heads, v_num_attention_heads, a_num_attention_heads,\n bi_num_attention_heads, num_hidden_layers, v_num_hidden_layers,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:760-775"
+ },
+ "6151": {
+ "file_id": 487,
+ "content": "This code is initializing a model for the ACTBERT backbone, which includes an encoder and action embedding. The model has parameters for various hidden sizes, dropout probabilities, attention head numbers, and hidden layer counts for both textual (v), visual (t), and action (a) components. The model also uses different activation functions for each component.",
+ "type": "comment"
+ },
+ "6152": {
+ "file_id": 487,
+ "content": " a_num_hidden_layers)\n self.t_pooler = BertPooler(hidden_size, bi_hidden_size)\n self.v_pooler = BertPooler(v_hidden_size, bi_hidden_size)\n self.a_pooler = BertPooler(a_hidden_size, bi_hidden_size)\n def forward(\n self,\n text_ids,\n action_feat,\n image_feat,\n image_loc,\n token_type_ids=None,\n text_mask=None,\n image_mask=None,\n action_mask=None,\n output_all_encoded_layers=False,\n ):\n \"\"\"\n text_ids: input text ids. Shape: [batch_size, seqence_length]\n action_feat: input action feature. Shape: [batch_size, action_length, action_feature_dim]\n image_feat: input image feature. Shape: [batch_size, region_length, image_feature_dim]]\n image_loc: input region location. Shape: [batch_size, region_length, region_location_dim]\n token_type_ids: segment ids of each video clip. Shape: [batch_size, seqence_length]\n text_mask: text mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, seqence_length]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:776-800"
+ },
+ "6153": {
+ "file_id": 487,
+ "content": "This code defines a class for a model that takes in text, action, and image features as inputs. It initializes three poolers for text, action, and visual features. The forward function processes the input data and returns encoded layers based on the inputs received. The output_all_encoded_layers parameter allows getting all encoded layers if set to True.",
+ "type": "comment"
+ },
+ "6154": {
+ "file_id": 487,
+ "content": " image_mask: image mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, region_length]\n action_mask: action mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, action_length]\n output_all_encoded_layers: is output encoded layers feature or not. Type: Bool.\n \"\"\"\n if text_mask is None:\n text_mask = paddle.ones_like(text_ids)\n if token_type_ids is None:\n token_type_ids = paddle.zeros_like(text_ids)\n if image_mask is None:\n image_mask = paddle.ones(image_feat.shape[0],\n image_feat.shape[1]).astype(text_ids.dtype)\n if action_mask is None:\n action_mask = paddle.ones(action_feat.shape[0],\n action_feat.shape[1]).astype(\n text_ids.dtype)\n # We create a 3D attention mask from a 2D tensor mask.\n # Sizes are [batch_size, 1, 1, to_seq_length]\n # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length].",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:801-819"
+ },
+ "6155": {
+ "file_id": 487,
+ "content": "This code checks if the input masks for text, token_type, image, and action are None. If any of them are None, it generates a mask with the same shape as the corresponding feature tensor and fills it with ones (real tokens) or zeros (padding tokens). The attention mask is created from the 2D tensor mask to be used in the multi-head attention mechanism, which broadcasts to [batch_size, num_heads, from_seq_length, to_seq_length].",
+ "type": "comment"
+ },
+ "6156": {
+ "file_id": 487,
+ "content": " extended_text_mask = text_mask.unsqueeze(1).unsqueeze(2)\n extended_image_mask = image_mask.unsqueeze(1).unsqueeze(2)\n extended_action_mask = action_mask.unsqueeze(1).unsqueeze(2)\n # Since attention_mask is 1.0 for positions we want to attend and 0.0 for\n # masked positions, this operation will create a tensor which is 0.0 for\n # positions we want to attend and -10000.0 for masked positions.\n # Since we are adding it to the raw scores before the softmax, this is\n # effectively the same as removing these entirely.\n def set_mask(extended_attention_mask):\n extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0\n return extended_attention_mask\n extended_text_mask = set_mask(extended_text_mask)\n extended_image_mask = set_mask(extended_image_mask)\n extended_action_mask = set_mask(extended_action_mask)\n t_embedding_output = self.embeddings(text_ids, token_type_ids)\n v_embedding_output = self.v_embeddings(image_feat, image_loc)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:820-838"
+ },
+ "6157": {
+ "file_id": 487,
+ "content": "This code segment is part of a backbone model for the ACTBERT. It creates extended masks for text, image, and action inputs by unsqueezing the existing masks along dimensions 1 and 2. The function set_mask is then used to multiply each mask with -10000.0 at positions where we want to attend, effectively removing those positions from the attention process. This is done for all three input types: text, image, and action. Finally, the code applies the embeddings to the text inputs using self.embeddings.",
+ "type": "comment"
+ },
+ "6158": {
+ "file_id": 487,
+ "content": " a_embedding_output = self.a_embeddings(action_feat)\n # var = [t_embedding_output, v_embedding_output, a_embedding_output]\n # import numpy as np\n # for i, item in enumerate(var):\n # np.save('tmp/' + str(i)+'.npy', item.numpy())\n encoded_layers_t, encoded_layers_v, encoded_layers_a = self.encoder(\n t_embedding_output,\n v_embedding_output,\n a_embedding_output,\n extended_text_mask,\n extended_image_mask,\n extended_action_mask,\n output_all_encoded_layers=output_all_encoded_layers,\n )\n sequence_output_t = encoded_layers_t[-1] #get item from list\n sequence_output_v = encoded_layers_v[-1]\n sequence_output_a = encoded_layers_a[-1]\n pooled_output_t = self.t_pooler(sequence_output_t)\n pooled_output_v = self.v_pooler(sequence_output_v)\n pooled_output_a = self.a_pooler(sequence_output_a)\n if not output_all_encoded_layers:\n encoded_layers_t = encoded_layers_t[-1]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:839-865"
+ },
+ "6159": {
+ "file_id": 487,
+ "content": "This code is part of a backbone for a multimodal model, specifically ACTBERT. It first computes the embedding outputs for text (t), vision (v), and action (a) features. Then it passes these embeddings to an encoder to obtain encoded layers for each modality. The last hidden state from each encoder is used as a sequence output, and a pooled output is also computed using separate poolers for each modality. If output_all_encoded_layers is False, the code reduces the encoded layers to their last hidden states.",
+ "type": "comment"
+ },
+ "6160": {
+ "file_id": 487,
+ "content": " encoded_layers_v = encoded_layers_v[-1]\n encoded_layers_a = encoded_layers_a[-1]\n return encoded_layers_t, encoded_layers_v, encoded_layers_a, \\\n pooled_output_t, pooled_output_v, pooled_output_a\n# For Head\nclass BertPredictionHeadTransform(nn.Layer):\n def __init__(self, hidden_size, hidden_act):\n super(BertPredictionHeadTransform, self).__init__()\n self.dense = nn.Linear(hidden_size, hidden_size)\n if isinstance(hidden_act, str) or (sys.version_info[0] == 2\n and isinstance(hidden_act, str)):\n self.transform_act_fn = ACT2FN[hidden_act]\n else:\n self.transform_act_fn = hidden_act\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n def forward(self, hidden_states):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.transform_act_fn(hidden_states)\n hidden_states = self.LayerNorm(hidden_states)\n return hidden_states\nclass BertLMPredictionHead(nn.Layer):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:866-892"
+ },
+ "6161": {
+ "file_id": 487,
+ "content": "This code defines two classes: `BertPredictionHeadTransform` and `BertLMPredictionHead`. The former is a transform layer used in BERT's prediction heads. It applies a dense layer followed by an activation function and layer normalization. The latter is the prediction head itself, which takes input hidden states, applies the transform defined in `BertPredictionHeadTransform`, and returns output.",
+ "type": "comment"
+ },
+ "6162": {
+ "file_id": 487,
+ "content": " def __init__(self, hidden_size, hidden_act, bert_model_embedding_weights):\n super(BertLMPredictionHead, self).__init__()\n self.transform = BertPredictionHeadTransform(hidden_size, hidden_act)\n # The output weights are the same as the input embeddings, but there is\n # an output-only bias for each token.\n assert bert_model_embedding_weights.shape[1] == hidden_size\n vocab_size = bert_model_embedding_weights.shape[0]\n # another implementation which would create another big params:\n # self.decoder = nn.Linear(hidden_size, vocab_size) # NOTE bias default: constant 0.0\n # self.decoder.weight = self.create_parameter(shape=[hidden_size, vocab_size],\n # default_initializer=nn.initializer.Assign(\n # bert_model_embedding_weights.t())) # transpose\n self.decoder_weight = bert_model_embedding_weights\n self.decoder_bias = self.create_parameter(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:893-909"
+ },
+ "6163": {
+ "file_id": 487,
+ "content": "This code initializes the BertLMPredictionHead class, which is a part of the BERT model. It takes in hidden_size, hidden_act, and bert_model_embedding_weights as parameters. The class uses these to initialize its transform and decoder components. The decoder component has a weight equal to the input embedding weights with an output-only bias for each token. This implementation avoids creating additional large parameters by directly assigning the input embedding weights to the decoder's weight attribute.",
+ "type": "comment"
+ },
+ "6164": {
+ "file_id": 487,
+ "content": " shape=[vocab_size],\n dtype=bert_model_embedding_weights.dtype,\n is_bias=True) # NOTE bias default: constant 0.0\n def forward(self, hidden_states):\n hidden_states = self.transform(hidden_states)\n hidden_states = paddle.tensor.matmul(\n hidden_states, self.decoder_weight,\n transpose_y=True) + self.decoder_bias\n return hidden_states\nclass BertImageActionPredictionHead(nn.Layer):\n def __init__(self, hidden_size, hidden_act, target_size):\n super(BertImageActionPredictionHead, self).__init__()\n self.transform = BertPredictionHeadTransform(hidden_size, hidden_act)\n self.decoder = nn.Linear(hidden_size, target_size)\n def forward(self, hidden_states):\n hidden_states = self.transform(hidden_states)\n hidden_states = self.decoder(hidden_states)\n return hidden_states\nclass BertPreTrainingHeads(nn.Layer):\n def __init__(self, hidden_size, v_hidden_size, a_hidden_size,\n bi_hidden_size, hidden_act, v_hidden_act, a_hidden_act,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:910-937"
+ },
+ "6165": {
+ "file_id": 487,
+ "content": "This code defines three classes: ActBert, BertImageActionPredictionHead, and BertPreTrainingHeads. ActBert is a layer that performs attention and feedforward operations in BERT's transformer layers. The BertImageActionPredictionHead class is responsible for the image action prediction task in BERT. Finally, the BertPreTrainingHeads class includes multiple layers for pre-training tasks such as masked language modeling, next sentence prediction, and SQUAD question answering.",
+ "type": "comment"
+ },
+ "6166": {
+ "file_id": 487,
+ "content": " v_target_size, a_target_size, fusion_method,\n bert_model_embedding_weights):\n super(BertPreTrainingHeads, self).__init__()\n self.predictions = BertLMPredictionHead(hidden_size, hidden_act,\n bert_model_embedding_weights)\n self.seq_relationship = nn.Linear(bi_hidden_size, 2)\n self.imagePredictions = BertImageActionPredictionHead(\n v_hidden_size, v_hidden_act, v_target_size) # visual class number\n self.actionPredictions = BertImageActionPredictionHead(\n a_hidden_size, a_hidden_act, a_target_size) # action class number\n self.fusion_method = fusion_method\n self.dropout = nn.Dropout(0.1)\n def forward(self, sequence_output_t, sequence_output_v, sequence_output_a,\n pooled_output_t, pooled_output_v, pooled_output_a):\n if self.fusion_method == 'sum':\n pooled_output = self.dropout(pooled_output_t + pooled_output_v +\n pooled_output_a)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:938-956"
+ },
+ "6167": {
+ "file_id": 487,
+ "content": "The code defines a class BertPreTrainingHeads that extends an existing class. It initializes the necessary modules for prediction and fusion. The forward function performs pooling and fusion operations based on the specified fusion method ('sum').",
+ "type": "comment"
+ },
+ "6168": {
+ "file_id": 487,
+ "content": " elif self.fusion_method == 'mul':\n pooled_output = self.dropout(pooled_output_t * pooled_output_v +\n pooled_output_a)\n else:\n assert False\n prediction_scores_t = self.predictions(\n sequence_output_t) # 8， 36 ，30522\n seq_relationship_score = self.seq_relationship(pooled_output) # 8, 2\n prediction_scores_v = self.imagePredictions(\n sequence_output_v) # 8, 37, 1601\n prediction_scores_a = self.actionPredictions(\n sequence_output_a) # 8, 5, 401\n return prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score\n@BACKBONES.register()\nclass BertForMultiModalPreTraining(nn.Layer):\n \"\"\"BERT model with multi modal pre-training heads.\n \"\"\"\n def __init__(\n self,\n vocab_size=30522,\n max_position_embeddings=512,\n type_vocab_size=2,\n v_target_size=1601,\n a_target_size=700,\n v_feature_size=2048,\n a_feature_size=2048,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:957-986"
+ },
+ "6169": {
+ "file_id": 487,
+ "content": "In this code snippet, the model is returning prediction scores for text (t), video (v), and action (a) inputs along with a sequence relationship score. The model is a multi-modal pre-training BERT model with fusion method 'mul'. If the fusion method is not 'mul', it will raise an assertion error.",
+ "type": "comment"
+ },
+ "6170": {
+ "file_id": 487,
+ "content": " num_hidden_layers=12,\n v_num_hidden_layers=2,\n a_num_hidden_layers=3,\n t_ent_attention_id=[10, 11],\n v_ent_attention_id=[0, 1],\n a_ent_attention_id=[0, 1],\n fixed_t_layer=0,\n fixed_v_layer=0,\n hidden_size=768,\n v_hidden_size=1024,\n a_hidden_size=768,\n bi_hidden_size=1024,\n intermediate_size=3072,\n v_intermediate_size=1024,\n a_intermediate_size=3072,\n hidden_act=\"gelu\",\n v_hidden_act=\"gelu\",\n a_hidden_act=\"gelu\",\n hidden_dropout_prob=0.1,\n v_hidden_dropout_prob=0.1,\n a_hidden_dropout_prob=0.1,\n attention_probs_dropout_prob=0.1,\n v_attention_probs_dropout_prob=0.1,\n a_attention_probs_dropout_prob=0.1,\n av_attention_probs_dropout_prob=0.1,\n at_attention_probs_dropout_prob=0.1,\n num_attention_heads=12,\n v_num_attention_heads=8,\n a_num_attention_heads=12,\n bi_num_attention_heads=8,\n fusion_method=\"mul\",\n pretrained=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:987-1018"
+ },
+ "6171": {
+ "file_id": 487,
+ "content": "This code defines a custom transformer backbone model for ACT-BERT, with specific configurations for the text (t), video (v), and audio (a) modalities. It includes parameters such as hidden layer numbers, sizes, activation functions, dropout rates, attention heads, and fusion method. The pretrained parameter is set to None.",
+ "type": "comment"
+ },
+ "6172": {
+ "file_id": 487,
+ "content": " ):\n \"\"\"\n vocab_size: vocabulary size. Default: 30522.\n max_position_embeddings: max position id. Default: 512.\n type_vocab_size: max segment id. Default: 2.\n v_target_size: class number of visual word. Default: 1601.\n a_target_size: class number of action word. Default: 700.\n v_feature_size: input visual feature dimension. Default: 2048.\n a_feature_size: input action feature dimension. Default: 2048.\n num_hidden_layers: number of BertLayer in text transformer. Default: 12.\n v_num_hidden_layers: number of BertLayer in visual transformer. Default: 2.\n a_num_hidden_layers: number of BertLayer in action transformer. Default:3.\n t_ent_attention_id: index id of BertConnectionLayer in text transformer. Default: [10, 11].\n v_ent_attention_id: index id of BertConnectionLayer in visual transformer. Default:[0, 1].\n a_ent_attention_id: index id of BertConnectionLayer in action transformer. Default:[0, 1].\n fixed_t_layer: index id of fixed BertLayer in text transformer. Default: 0.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1019-1034"
+ },
+ "6173": {
+ "file_id": 487,
+ "content": "This function defines the input parameters for an ActBERT model, including vocabulary size, maximum position embedding, type vocab size, visual and action target sizes, feature sizes for vision and actions, number of hidden layers in text, visual, and action transformers, index IDs for BertConnectionLayer, and a fixed layer index for the text transformer.",
+ "type": "comment"
+ },
+ "6174": {
+ "file_id": 487,
+ "content": " fixed_v_layer: index id of fixed BertLayer in visual transformer. Default: 0.\n hidden_size: hidden size in text BertLayer. Default: 768.\n v_hidden_size: hidden size in visual BertLayer. Default: 1024.\n a_hidden_size: hidden size in action BertLayer. Default: 768.\n bi_hidden_size: hidden size in BertConnectionLayer. Default: 1024,\n intermediate_size: intermediate size in text BertLayer. Default: 3072.\n v_intermediate_size: intermediate size in visual BertLayer. Default: 1024.\n a_intermediate_size: intermediate size in text BertLayer. Default: 3072.\n hidden_act: hidden activation function in text BertLayer. Default: \"gelu\".\n v_hidden_act: hidden activation function in visual BertLayer. Default: \"gelu\".\n a_hidden_act: hidden activation function in action BertLayer. Default: \"gelu\".\n hidden_dropout_prob: hidden dropout probability in text Embedding Layer. Default: 0.1\n v_hidden_dropout_prob: hidden dropout probability in visual Embedding Layer. Default: 0.1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1035-1047"
+ },
+ "6175": {
+ "file_id": 487,
+ "content": "This code defines parameters for different BertLayers in a model. Fixed_v_layer is the index of a fixed BertLayer in the visual transformer, hidden_size is the hidden size for text and other BERT layers, v_hidden_size is the hidden size for visual BERT layer, a_hidden_size is the hidden size for action BERT layer, bi_hidden_size is the hidden size for BertConnectionLayer, intermediate_size is the intermediate size for text and other BERT layers, v_intermediate_size is the intermediate size for visual BERT layer, a_intermediate_size is the intermediate size for action BERT layer, hidden_act is the activation function for text BERT layer, v_hidden_act is the activation function for visual BERT layer, a_hidden_act is the activation function for action BERT layer, and hidden_dropout_prob is the dropout probability for text embedding layer. All default values are provided in case no other values are specified.",
+ "type": "comment"
+ },
+ "6176": {
+ "file_id": 487,
+ "content": " a_hidden_dropout_prob: hidden dropout probability in action Embedding Layer. Default: 0.1\n attention_probs_dropout_prob: attention dropout probability in text BertLayer. Default: 0.1\n v_attention_probs_dropout_prob: attention dropout probability in visual BertLayer. Default: 0.1\n a_attention_probs_dropout_prob: attention dropout probability in action BertLayer. Default: 0.1\n av_attention_probs_dropout_prob: attention dropout probability in action-visual BertConnectionLayer. Default: 0.1\n at_attention_probs_dropout_prob: attention dropout probability in action-text BertConnectionLayer. Default: 0.1\n num_attention_heads: number of heads in text BertLayer. Default: 12.\n v_num_attention_heads: number of heads in visual BertLayer. Default: 8.\n a_num_attention_heads: number of heads in action BertLayer. Default: 12.\n bi_num_attention_heads: number of heads in BertConnectionLayer. Default: 8.\n fusion_method: methods of fusing pooled output from 3 transformer. Default: \"mul\".",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1048-1058"
+ },
+ "6177": {
+ "file_id": 487,
+ "content": "This code snippet defines default values for various parameters in a transformer model. These parameters include hidden dropout probabilities, attention dropout probabilities, number of attention heads, and fusion methods. The default values are provided for the text, visual, and action BertLayers as well as the BertConnectionLayer.",
+ "type": "comment"
+ },
+ "6178": {
+ "file_id": 487,
+ "content": " \"\"\"\n super(BertForMultiModalPreTraining, self).__init__()\n self.pretrained = pretrained\n self.vocab_size = vocab_size\n self.a_target_size = a_target_size\n self.bert = BertModel(\n vocab_size,\n max_position_embeddings,\n type_vocab_size,\n v_feature_size,\n a_feature_size,\n num_hidden_layers,\n v_num_hidden_layers,\n a_num_hidden_layers,\n v_ent_attention_id,\n t_ent_attention_id,\n a_ent_attention_id,\n fixed_t_layer,\n fixed_v_layer,\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n intermediate_size,\n v_intermediate_size,\n a_intermediate_size,\n hidden_act,\n v_hidden_act,\n a_hidden_act,\n hidden_dropout_prob,\n v_hidden_dropout_prob,\n a_hidden_dropout_prob,\n attention_probs_dropout_prob,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1059-1092"
+ },
+ "6179": {
+ "file_id": 487,
+ "content": "This code initializes an instance of the BertForMultiModalPreTraining class, which is a pre-trained model for multi-modal tasks. It takes in various parameters such as vocab_size, max_position_embeddings, type_vocab_size, v_feature_size, a_feature_size, num_hidden_layers, v_num_hidden_layers, a_num_hidden_layers, v_ent_attention_id, t_ent_attention_id, a_ent_attention_id, fixed_t_layer, fixed_v_layer, hidden_size, v_hidden_size, a_hidden_size, bi_hidden_size, intermediate_size, v_intermediate_size, a_intermediate_size, hidden_act, v_hidden_act, a_hidden_act, hidden_dropout_prob, v_hidden_dropout_prob, and a_hidden_dropout_prob. These parameters define the architecture and behavior of the model. The super() function is used to call a method from the parent class, in this case, BertModel. The pretrained variable indicates whether the model should use pre-trained weights or not.",
+ "type": "comment"
+ },
+ "6180": {
+ "file_id": 487,
+ "content": " v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n num_attention_heads,\n v_num_attention_heads,\n a_num_attention_heads,\n bi_num_attention_heads,\n )\n self.cls = BertPreTrainingHeads(\n hidden_size, v_hidden_size, a_hidden_size, bi_hidden_size,\n hidden_act, v_hidden_act, a_hidden_act, v_target_size,\n a_target_size, fusion_method,\n self.bert.embeddings.word_embeddings.weight)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, (nn.Linear, nn.Embedding)):\n weight_init_(layer, 'Normal', std=0.02)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1093-1116"
+ },
+ "6181": {
+ "file_id": 487,
+ "content": "This code initializes the parameters of a pre-trained ACTBERT model. It checks if the model has been pre-trained and, if not, initializes the weights for the layers (using normal distribution with standard deviation 0.02).",
+ "type": "comment"
+ },
+ "6182": {
+ "file_id": 487,
+ "content": " elif isinstance(layer, nn.LayerNorm):\n weight_init_(layer, 'Constant', value=1)\n def forward(\n self,\n text_ids, #8,36\n action_feat, #8,5,2048\n image_feat, #8,37,2048\n image_loc, #8,37,5\n token_type_ids=None, #8,36\n text_mask=None, #8,36\n image_mask=None, #8,37\n action_mask=None, #8,5\n ):\n \"\"\"\n text_ids: input text ids. Shape: [batch_size, seqence_length]\n action_feat: input action feature. Shape: [batch_size, action_length, action_feature_dim]\n image_feat: input image feature. Shape: [batch_size, region_length+1, image_feature_dim]], add 1 for image global feature.\n image_loc: input region location. Shape: [batch_size, region_length+1, region_location_dim], add 1 for image global feature location.\n token_type_ids: segment ids of each video clip. Shape: [batch_size, seqence_length]\n text_mask: text mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, seqence_length]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1117-1137"
+ },
+ "6183": {
+ "file_id": 487,
+ "content": "The code defines a function \"forward\" that takes text_ids, action_feat, image_feat, image_loc, token_type_ids, text_mask, image_mask, and action_mask as input. The text_ids represent input text ids of shape [batch_size, sequence_length]. Action_feat is the input action feature of shape [batch_size, action_length, action_feature_dim], while image_feat is the input image feature of shape [batch_size, region_length+1, image_feature_dim] (adding 1 for global image feature). Image_loc represents the input region location of shape [batch_size, region_length+1, region_location_dim] (adding 1 for global image feature location). Token_type_ids represent segment ids of each video clip and are of shape [batch_size, sequence_length]. Text_mask is a binary mask representing real tokens as 1 and padding tokens as 0 with shape [batch_size, sequence_length]. Image_mask and action_mask also serve similar functions but for image and action respectively.",
+ "type": "comment"
+ },
+ "6184": {
+ "file_id": 487,
+ "content": " image_mask: image mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, region_length]\n action_mask: action mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, action_length]\n \"\"\"\n sequence_output_t, sequence_output_v, sequence_output_a, \\\n pooled_output_t, pooled_output_v, pooled_output_a = self.bert(\n text_ids,\n action_feat,\n image_feat,\n image_loc,\n token_type_ids,\n text_mask,\n image_mask,\n action_mask,\n output_all_encoded_layers=False,\n )\n prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score = self.cls(\n sequence_output_t, sequence_output_v, sequence_output_a,\n pooled_output_t, pooled_output_v, pooled_output_a)\n return prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/actbert.py:1138-1158"
+ },
+ "6185": {
+ "file_id": 487,
+ "content": "This code is a function that takes in text IDs, action feature, image feature, image location, token type IDs, text mask, image mask, and action mask as inputs. It uses the BERT model to process these inputs and returns prediction scores for each input (text, vision, action) and sequence relationship score.",
+ "type": "comment"
+ },
+ "6186": {
+ "file_id": 488,
+ "content": "/paddlevideo/modeling/backbones/adds.py",
+ "type": "filepath"
+ },
+ "6187": {
+ "file_id": 488,
+ "content": "The code imports necessary libraries, registers object detection backbones, performs vector transformations, defines network creation functions, computes depth prediction, initializes PaddleVideo backbone, includes Project3D layer, calculates SSIM loss, and creates a deep learning model for image processing with ResNet V1.5, DepthDecoder, and PoseDecoder classes. The pose estimation model supports diverse inputs, handles day/night scenarios, computes parameters, generates warped images, and selects data based on conditions.",
+ "type": "summary"
+ },
+ "6188": {
+ "file_id": 488,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nfrom collections import OrderedDict\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn import BatchNorm2D, Conv2D\nfrom paddle.nn.initializer import Constant, Normal\nfrom paddle.vision.models import ResNet\nfrom ...utils import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import kaiming_normal_, _calculate_fan_in_and_fan_out\nzeros_ = Constant(value=0.)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1-30"
+ },
+ "6189": {
+ "file_id": 488,
+ "content": "This code imports necessary libraries, defines constants, and registers backbones in a PaddlePaddle's object detection model library. It also includes comments for licensing and copyright information as well as function definitions for weight initialization and calculating fan-in and fan-out of layers.",
+ "type": "comment"
+ },
+ "6190": {
+ "file_id": 488,
+ "content": "ones_ = Constant(value=1.)\nnormal_ = Normal(mean=0, std=1e-3)\ndef disp_to_depth(disp, min_depth, max_depth):\n \"\"\"Convert network's sigmoid output into depth prediction\n The formula for this conversion is given in the 'additional considerations'\n section of the paper.\n \"\"\"\n min_disp = 1 / max_depth\n max_disp = 1 / min_depth\n scaled_disp = min_disp + (max_disp - min_disp) * disp\n depth = 1 / scaled_disp\n return scaled_disp, depth\ndef gram_matrix(y):\n (b, ch, h, w) = y.shape\n features = y.reshape([b, ch, w * h])\n features_t = paddle.transpose(features, [0, 2, 1])\n gram = features.bmm(features_t) / (ch * h * w)\n return gram\ndef convt_bn_relu(in_channels,\n out_channels,\n kernel_size,\n stride=1,\n padding=0,\n output_padding=0,\n bn=True,\n relu=True):\n bias = not bn\n layers = []\n layers.append(\n nn.Conv2DTranspose(in_channels,\n out_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:31-67"
+ },
+ "6191": {
+ "file_id": 488,
+ "content": "Code snippet defines three functions - \"disp_to_depth\" for converting network's sigmoid output into depth prediction, \"gram_matrix\" for computing the Gram matrix of feature maps and \"convt_bn_relu\" for creating a convolution layer with batch normalization and ReLU activation.",
+ "type": "comment"
+ },
+ "6192": {
+ "file_id": 488,
+ "content": " kernel_size,\n stride,\n padding,\n output_padding,\n bias_attr=bias))\n if bn:\n layers.append(nn.BatchNorm2D(out_channels))\n if relu:\n layers.append(nn.LeakyReLU(0.2))\n layers = nn.Sequential(*layers)\n # initialize the weights\n for m in layers.sublayers(include_self=True):\n if isinstance(m, nn.Conv2DTranspose):\n normal_(m.weight)\n if m.bias is not None:\n zeros_(m.bias)\n elif isinstance(m, nn.BatchNorm2D):\n ones_(m.weight)\n zeros_(m.bias)\n return layers\ndef transformation_from_parameters(axisangle, translation, invert=False):\n \"\"\"Convert the network's (axisangle, translation) output into a 4x4 matrix\n \"\"\"\n R = rot_from_axisangle(axisangle)\n t = translation.clone()\n if invert:\n R = R.transpose([0, 2, 1])\n t *= -1\n T = get_translation_matrix(t)\n if invert:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:68-104"
+ },
+ "6193": {
+ "file_id": 488,
+ "content": "The code defines a function for creating a convolutional transpose layer, adding batch normalization and Leaky ReLU activation if specified. It also includes weight initialization for the created layers. The second function converts network's (axisangle, translation) output into a 4x4 matrix based on parameters and an optional invert flag.",
+ "type": "comment"
+ },
+ "6194": {
+ "file_id": 488,
+ "content": " M = paddle.matmul(R, T)\n else:\n M = paddle.matmul(T, R)\n return M\ndef get_translation_matrix(translation_vector):\n \"\"\"Convert a translation vector into a 4x4 transformation matrix\n \"\"\"\n t = translation_vector.reshape([-1, 3, 1])\n gather_object = paddle.stack([\n paddle.zeros([\n translation_vector.shape[0],\n ], paddle.float32),\n paddle.ones([\n translation_vector.shape[0],\n ], paddle.float32),\n paddle.squeeze(t[:, 0], axis=-1),\n paddle.squeeze(t[:, 1], axis=-1),\n paddle.squeeze(t[:, 2], axis=-1),\n ])\n gather_index = paddle.to_tensor([\n [1],\n [0],\n [0],\n [2],\n [0],\n [1],\n [0],\n [3],\n [0],\n [0],\n [1],\n [4],\n [0],\n [0],\n [0],\n [1],\n ])\n T = paddle.gather_nd(gather_object, gather_index)\n T = T.reshape([4, 4, -1]).transpose((2, 0, 1))\n return T\ndef rot_from_axisangle(vec):\n \"\"\"Convert an axisangle rotation into a 4x4 transformation matrix",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:105-151"
+ },
+ "6195": {
+ "file_id": 488,
+ "content": "get_translation_matrix: Converts translation vector to a 4x4 transformation matrix.\nrot_from_axisangle: Converts axis-angle rotation into a 4x4 transformation matrix.",
+ "type": "comment"
+ },
+ "6196": {
+ "file_id": 488,
+ "content": " (adapted from https://github.com/Wallacoloo/printipi)\n Input 'vec' has to be Bx1x3\n \"\"\"\n angle = paddle.norm(vec, 2, 2, True)\n axis = vec / (angle + 1e-7)\n ca = paddle.cos(angle)\n sa = paddle.sin(angle)\n C = 1 - ca\n x = axis[..., 0].unsqueeze(1)\n y = axis[..., 1].unsqueeze(1)\n z = axis[..., 2].unsqueeze(1)\n xs = x * sa\n ys = y * sa\n zs = z * sa\n xC = x * C\n yC = y * C\n zC = z * C\n xyC = x * yC\n yzC = y * zC\n zxC = z * xC\n gather_object = paddle.stack([\n paddle.squeeze(x * xC + ca, axis=(-1, -2)),\n paddle.squeeze(xyC - zs, axis=(-1, -2)),\n paddle.squeeze(zxC + ys, axis=(-1, -2)),\n paddle.squeeze(xyC + zs, axis=(-1, -2)),\n paddle.squeeze(y * yC + ca, axis=(-1, -2)),\n paddle.squeeze(yzC - xs, axis=(-1, -2)),\n paddle.squeeze(zxC - ys, axis=(-1, -2)),\n paddle.squeeze(yzC + xs, axis=(-1, -2)),\n paddle.squeeze(z * zC + ca, axis=(-1, -2)),\n paddle.ones([\n vec.shape[0],\n ], dtype=paddle.float32),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:152-188"
+ },
+ "6197": {
+ "file_id": 488,
+ "content": "This code performs rotation operations on a 3D vector 'vec'. It calculates the angle and axis of rotation, then applies trigonometry to compute rotation matrices. Finally, it gathers transformed vectors using stacked tensor operations.",
+ "type": "comment"
+ },
+ "6198": {
+ "file_id": 488,
+ "content": " paddle.zeros([\n vec.shape[0],\n ], dtype=paddle.float32)\n ])\n gather_index = paddle.to_tensor([\n [0],\n [1],\n [2],\n [10],\n [3],\n [4],\n [5],\n [10],\n [6],\n [7],\n [8],\n [10],\n [10],\n [10],\n [10],\n [9],\n ])\n rot = paddle.gather_nd(gather_object, gather_index)\n rot = rot.reshape([4, 4, -1]).transpose((2, 0, 1))\n return rot\ndef upsample(x):\n \"\"\"Upsample input tensor by a factor of 2\n \"\"\"\n return F.interpolate(x, scale_factor=2, mode=\"nearest\")\ndef get_smooth_loss(disp, img):\n \"\"\"Computes the smoothness loss for a disparity image\n The color image is used for edge-aware smoothness\n \"\"\"\n grad_disp_x = paddle.abs(disp[:, :, :, :-1] - disp[:, :, :, 1:])\n grad_disp_y = paddle.abs(disp[:, :, :-1, :] - disp[:, :, 1:, :])\n grad_img_x = paddle.mean(paddle.abs(img[:, :, :, :-1] - img[:, :, :, 1:]),\n 1,\n keepdim=True)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:189-231"
+ },
+ "6199": {
+ "file_id": 488,
+ "content": "Code defines three functions: \"get_rot\", \"upsample\", and \"get_smooth_loss\". get_rot performs a gather operation on a tensor, reshapes the result, then transposes it. upsample interpolates an input tensor by doubling its size. get_smooth_loss computes the smoothness loss for disparity images using gradients of disparities and color image edges.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/62.json b/docs/data/62.json
new file mode 100644
index 000000000..2f9639ea9
--- /dev/null
+++ b/docs/data/62.json
@@ -0,0 +1,549 @@
+{
+ "6200": {
+ "file_id": 488,
+ "content": " grad_img_y = paddle.mean(paddle.abs(img[:, :, :-1, :] - img[:, :, 1:, :]),\n 1,\n keepdim=True)\n grad_disp_x *= paddle.exp(-grad_img_x)\n grad_disp_y *= paddle.exp(-grad_img_y)\n return grad_disp_x.mean() + grad_disp_y.mean()\ndef conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1):\n \"\"\"3x3 convolution with padding\"\"\"\n return nn.Conv2D(in_planes,\n out_planes,\n kernel_size=3,\n stride=stride,\n padding=dilation,\n groups=groups,\n bias_attr=False,\n dilation=dilation)\ndef conv1x1(in_planes, out_planes, stride=1):\n \"\"\"1x1 convolution\"\"\"\n return nn.Conv2D(in_planes,\n out_planes,\n kernel_size=1,\n stride=stride,\n bias_attr=False)\ndef resnet_multiimage_input(num_layers, num_input_images=1):\n \"\"\"Constructs a ResNet model.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:232-264"
+ },
+ "6201": {
+ "file_id": 488,
+ "content": "This code defines functions for creating convolutional layers and a ResNet model with multiple input images. The functions include 3x3 and 1x1 convolutions, along with a function that constructs the ResNet model itself. The ResNet model can handle multiple input images by combining gradients from each image channel.",
+ "type": "comment"
+ },
+ "6202": {
+ "file_id": 488,
+ "content": " Args:\n num_layers (int): Number of resnet layers. Must be 18 or 50\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n num_input_images (int): Number of frames stacked as input\n \"\"\"\n assert num_layers in [18, 50], \"Can only run with 18 or 50 layer resnet\"\n blocks = {18: [2, 2, 2, 2], 50: [3, 4, 6, 3]}[num_layers]\n block_type = {18: BasicBlock, 50: Bottleneck}[num_layers]\n model = ResNetMultiImageInput(block_type,\n num_layers,\n blocks,\n num_input_images=num_input_images)\n model.init_weights()\n return model\nclass ConvBlock(nn.Layer):\n \"\"\"Layer to perform a convolution followed by ELU\n \"\"\"\n def __init__(self, in_channels, out_channels):\n super(ConvBlock, self).__init__()\n self.conv = Conv3x3(in_channels, out_channels)\n self.nonlin = nn.ELU()\n def forward(self, x):\n out = self.conv(x)\n out = self.nonlin(out)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:265-294"
+ },
+ "6203": {
+ "file_id": 488,
+ "content": "This code defines a function that creates a ResNet model with multiple image inputs. The model takes in the number of resnet layers (18 or 50), whether to use pretrained weights, and the number of input frames to stack. It then creates blocks based on the layer type and number of layers provided, and initializes the model's weights. The ConvBlock class performs a convolution followed by ELU activation.",
+ "type": "comment"
+ },
+ "6204": {
+ "file_id": 488,
+ "content": " return out\nclass Conv3x3(nn.Layer):\n \"\"\"Layer to pad and convolve input\n \"\"\"\n def __init__(self, in_channels, out_channels, use_refl=True):\n super(Conv3x3, self).__init__()\n if use_refl:\n self.pad = nn.Pad2D(1, mode='reflect')\n else:\n self.pad = nn.Pad2D(1)\n self.conv = nn.Conv2D(int(in_channels), int(out_channels), 3)\n def forward(self, x):\n out = self.pad(x)\n out = self.conv(out)\n return out\nclass BackprojectDepth(nn.Layer):\n \"\"\"Layer to transform a depth image into a point cloud\n \"\"\"\n def __init__(self, batch_size, height, width):\n super(BackprojectDepth, self).__init__()\n self.batch_size = batch_size\n self.height = height\n self.width = width\n meshgrid = np.meshgrid(range(self.width),\n range(self.height),\n indexing='xy')\n id_coords = np.stack(meshgrid, axis=0).astype(np.float32)\n self.id_coords = self.create_parameter(shape=list(id_coords.shape),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:295-330"
+ },
+ "6205": {
+ "file_id": 488,
+ "content": "Conv3x3 is a layer that pads and convolves the input.\nBackprojectDepth transforms a depth image into a point cloud.",
+ "type": "comment"
+ },
+ "6206": {
+ "file_id": 488,
+ "content": " dtype=paddle.float32)\n self.id_coords.set_value(id_coords)\n self.add_parameter(\"id_coords\", self.id_coords)\n self.id_coords.stop_gradient = True\n self.ones = self.create_parameter(\n shape=[self.batch_size, 1, self.height * self.width],\n default_initializer=ones_)\n self.add_parameter(\"ones\", self.ones)\n self.ones.stop_gradient = True\n pix_coords = paddle.unsqueeze(\n paddle.stack([\n self.id_coords[0].reshape([\n -1,\n ]), self.id_coords[1].reshape([\n -1,\n ])\n ], 0), 0)\n pix_coords = pix_coords.tile([batch_size, 1, 1])\n pix_coords = paddle.concat([pix_coords, self.ones], 1)\n self.pix_coords = self.create_parameter(shape=list(pix_coords.shape), )\n self.pix_coords.set_value(pix_coords)\n self.add_parameter(\"pix_coords\", self.pix_coords)\n self.pix_coords.stop_gradient = True",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:331-355"
+ },
+ "6207": {
+ "file_id": 488,
+ "content": "This code creates and initializes parameters for a backbone in PaddleVideo, specifically for the ID and pixel coordinates. It sets the gradients to stop, meaning they won't be updated during backpropagation. The code uses paddling operations like unsqueeze, stack, tile, and concat for parameter creation and manipulation.",
+ "type": "comment"
+ },
+ "6208": {
+ "file_id": 488,
+ "content": " def forward(self, depth, inv_K):\n cam_points = paddle.matmul(inv_K[:, :3, :3], self.pix_coords)\n cam_points = depth.reshape([self.batch_size, 1, -1]) * cam_points\n cam_points = paddle.concat([cam_points, self.ones], 1)\n return cam_points\nclass Project3D(nn.Layer):\n \"\"\"Layer which projects 3D points into a camera with intrinsics K and at position T\n \"\"\"\n def __init__(self, batch_size, height, width, eps=1e-7):\n super(Project3D, self).__init__()\n self.batch_size = batch_size\n self.height = height\n self.width = width\n self.eps = eps\n def forward(self, points, K, T):\n P = paddle.matmul(K, T)[:, :3, :]\n cam_points = paddle.matmul(P, points)\n pix_coords = cam_points[:, :2, :] / (cam_points[:, 2, :].unsqueeze(1) +\n self.eps)\n pix_coords = pix_coords.reshape(\n [self.batch_size, 2, self.height, self.width])\n pix_coords = pix_coords.transpose([0, 2, 3, 1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:357-385"
+ },
+ "6209": {
+ "file_id": 488,
+ "content": "The code defines a Project3D layer that projects 3D points into a camera with intrinsics K and at position T. It includes the forward pass, initialization, and required parameters such as batch_size, height, and width. The forward function calculates camera projection points by multiplying the intrinsic matrix K with the translation matrix T, then projects the points to pixels coordinates.",
+ "type": "comment"
+ },
+ "6210": {
+ "file_id": 488,
+ "content": " pix_coords[..., 0] /= self.width - 1\n pix_coords[..., 1] /= self.height - 1\n pix_coords = (pix_coords - 0.5) * 2\n return pix_coords\nclass SSIM(nn.Layer):\n \"\"\"Layer to compute the SSIM loss between a pair of images\n \"\"\"\n def __init__(self):\n super(SSIM, self).__init__()\n self.mu_x_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.mu_y_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.sig_x_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.sig_y_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.sig_xy_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.refl = nn.Pad2D(1, mode='reflect')\n self.C1 = 0.01**2\n self.C2 = 0.03**2\n def forward(self, x, y):\n x = self.refl(x)\n y = self.refl(y)\n mu_x = self.mu_x_pool(x)\n mu_y = self.mu_y_pool(y)\n sigma_x = self.sig_x_pool(x**2) - mu_x**2\n sigma_y = self.sig_y_pool(y**2) - mu_y**2\n sigma_xy = self.sig_xy_pool(x * y) - mu_x * mu_y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:386-417"
+ },
+ "6211": {
+ "file_id": 488,
+ "content": "The code defines a function `pix_coords` that normalizes pixel coordinates and a class `SSIM` for computing the Structural Similarity Index (SSIM) loss between two images. It initializes variables for mean, variance pooling, and applies padding to input images before calculating SSIM loss using provided formulas.",
+ "type": "comment"
+ },
+ "6212": {
+ "file_id": 488,
+ "content": " SSIM_n = (2 * mu_x * mu_y + self.C1) * (2 * sigma_xy + self.C2)\n SSIM_d = (mu_x**2 + mu_y**2 + self.C1) * (sigma_x + sigma_y + self.C2)\n return paddle.clip((1 - SSIM_n / SSIM_d) / 2, 0, 1)\nclass ResNetMultiImageInput(ResNet):\n \"\"\"Constructs a resnet model with varying number of input images.\n Adapted from https://github.com/pypaddle/vision/blob/master/paddlevision/models/resnet.py\n \"\"\"\n def __init__(self, block, depth, layers, num_input_images=1):\n super(ResNetMultiImageInput, self).__init__(block, depth)\n self.inplanes = 64\n self.conv1 = nn.Conv2D(num_input_images * 3,\n 64,\n kernel_size=7,\n stride=2,\n padding=3,\n bias_attr=False)\n self.bn1 = nn.BatchNorm2D(64)\n self.relu = nn.ReLU()\n self.maxpool = nn.MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.layer1 = self._make_layer(block, 64, layers[0])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:419-441"
+ },
+ "6213": {
+ "file_id": 488,
+ "content": "The code defines a ResNet model with multiple input images. It includes a convolution layer, batch normalization, ReLU activation, and max pooling for initial processing. The class \"ResNetMultiImageInput\" inherits from the base \"ResNet\" class and can handle different numbers of input images.",
+ "type": "comment"
+ },
+ "6214": {
+ "file_id": 488,
+ "content": " self.layer2 = self._make_layer(block, 128, layers[1], stride=2)\n self.layer3 = self._make_layer(block, 256, layers[2], stride=2)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=2)\n def init_weights(self):\n for layer in self.sublayers(include_self=True):\n if isinstance(layer, nn.Conv2D):\n kaiming_normal_(layer.weight,\n mode='fan_out',\n nonlinearity='relu')\n elif isinstance(layer, nn.BatchNorm2D):\n ones_(layer.weight)\n zeros_(layer.bias)\nclass ConvBNLayer(nn.Layer):\n \"\"\"Conv2D and BatchNorm2D layer.\n Args:\n in_channels (int): Number of channels for the input.\n out_channels (int): Number of channels for the output.\n kernel_size (int): Kernel size.\n stride (int): Stride in the Conv2D layer. Default: 1.\n groups (int): Groups in the Conv2D, Default: 1.\n act (str): Indicate activation after BatchNorm2D layer.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:442-466"
+ },
+ "6215": {
+ "file_id": 488,
+ "content": "The code defines a model architecture with multiple layers, including ConvBNLayer. It initializes the weights of these layers using specific methods and constraints for convolutional and batch normalization layers. This is typically done to improve performance and stability in deep learning models.",
+ "type": "comment"
+ },
+ "6216": {
+ "file_id": 488,
+ "content": " name (str): the name of an instance of ConvBNLayer.\n Note: weight and bias initialization include initialize values\n and name the restored parameters, values initialization\n are explicit declared in the ```init_weights``` method.\n \"\"\"\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n act=None,\n name=None):\n super(ConvBNLayer, self).__init__()\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n bias_attr=False)\n self._act = act\n self._batch_norm = BatchNorm2D(out_channels)\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:467-497"
+ },
+ "6217": {
+ "file_id": 488,
+ "content": "The `ConvBNLayer` class is a custom layer that consists of a convolution operation and batch normalization. It initializes the Conv2D layer and BatchNorm2D layer with specified parameters, and applies them sequentially in the forward pass.",
+ "type": "comment"
+ },
+ "6218": {
+ "file_id": 488,
+ "content": " if self._act:\n y = getattr(paddle.nn.functional, self._act)(y)\n return y\nclass BasicBlock(nn.Layer):\n expansion = 1\n def __init__(self,\n inplanes,\n planes,\n stride=1,\n downsample=None,\n groups=1,\n base_width=64,\n dilation=1,\n norm_layer=None):\n super(BasicBlock, self).__init__()\n if norm_layer is None:\n norm_layer = nn.BatchNorm2D\n if groups != 1 or base_width != 64:\n raise ValueError(\n 'BasicBlock only supports groups=1 and base_width=64')\n if dilation > 1:\n raise NotImplementedError(\n \"Dilation > 1 not supported in BasicBlock\")\n # Both self.conv1 and self.downsample layers downsample the input when stride != 1\n self.conv1 = conv3x3(inplanes, planes, stride)\n self.bn1 = norm_layer(planes)\n self.relu = nn.ReLU()\n self.conv2 = conv3x3(planes, planes)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:498-528"
+ },
+ "6219": {
+ "file_id": 488,
+ "content": "The code defines a class called `BasicBlock` which is an instance of the `nn.Layer` class, and initializes it with parameters such as `inplanes`, `planes`, `stride`, `downsample`, `groups`, `base_width`, `dilation`, and `norm_layer`. It also performs some checks to ensure that certain values match the block's requirements, and then initializes specific layers like `conv1`, `bn1`, and `relu` accordingly. The code also handles cases where `stride` is not equal to 1 by downsampling the input through both `self.conv1` and `self.downsample`.",
+ "type": "comment"
+ },
+ "6220": {
+ "file_id": 488,
+ "content": " self.bn2 = norm_layer(planes)\n self.downsample = downsample\n self.stride = stride\n def forward(self, x):\n identity = x\n out = self.conv1(x)\n out = self.bn1(out)\n out = self.relu(out)\n out = self.conv2(out)\n out = self.bn2(out)\n if self.downsample is not None:\n identity = self.downsample(x)\n out += identity\n out = self.relu(out)\n return out\nclass Bottleneck(nn.Layer):\n # Bottleneck in torchvision places the stride for downsampling at 3x3 convolution(self.conv2)\n # while original implementation places the stride at the first 1x1 convolution(self.conv1)\n # according to \"Deep residual learning for image recognition\"https://arxiv.org/abs/1512.03385.\n # This variant is also known as ResNet V1.5 and improves accuracy according to\n # https://ngc.nvidia.com/catalog/model-scripts/nvidia:resnet_50_v1_5_for_pytorch.\n expansion = 4\n def __init__(self,\n inplanes,\n planes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:529-563"
+ },
+ "6221": {
+ "file_id": 488,
+ "content": "The code defines a Bottleneck layer with stride at the 3x3 convolution (self.conv2) for ResNet V1.5, improving accuracy according to sources like \"Deep residual learning for image recognition\" and \"NVIDIA: ResNet_50_v1_5_for_PyTorch\". The Bottleneck layer has an expansion of 4, and its class initializes inplanes, planes, and other parameters.",
+ "type": "comment"
+ },
+ "6222": {
+ "file_id": 488,
+ "content": " stride=1,\n downsample=None,\n groups=1,\n base_width=64,\n dilation=1,\n norm_layer=None):\n super(Bottleneck, self).__init__()\n if norm_layer is None:\n norm_layer = nn.BatchNorm2D\n width = int(planes * (base_width / 64.)) * groups\n self.conv1 = conv1x1(inplanes, width)\n self.bn1 = norm_layer(width)\n self.conv2 = conv3x3(width, width, stride, groups, dilation)\n self.bn2 = norm_layer(width)\n self.conv3 = conv1x1(width, planes * self.expansion)\n self.bn3 = norm_layer(planes * self.expansion)\n self.relu = nn.ReLU()\n self.downsample = downsample\n self.stride = stride\n def forward(self, x):\n identity = x\n out = self.conv1(x)\n out = self.bn1(out)\n out = self.relu(out)\n out = self.conv2(out)\n out = self.bn2(out)\n out = self.relu(out)\n out = self.conv3(out)\n out = self.bn3(out)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:564-597"
+ },
+ "6223": {
+ "file_id": 488,
+ "content": "The code defines a Bottleneck class for a convolutional neural network. It has multiple layers of 1x1 and 3x3 convolutions, with batch normalization and ReLU activation functions. The class also supports downsampling and stride configuration options.",
+ "type": "comment"
+ },
+ "6224": {
+ "file_id": 488,
+ "content": " if self.downsample is not None:\n identity = self.downsample(x)\n out += identity\n out = self.relu(out)\n return out\nclass DepthDecoder(nn.Layer):\n def __init__(self,\n num_ch_enc,\n scales=range(4),\n num_output_channels=1,\n use_skips=True):\n super(DepthDecoder, self).__init__()\n self.num_output_channels = num_output_channels\n self.use_skips = use_skips\n self.upsample_mode = 'nearest'\n self.scales = scales\n self.num_ch_enc = num_ch_enc\n self.num_ch_dec = np.array([16, 32, 64, 128, 256])\n # decoder\n self.convs = OrderedDict()\n for i in range(4, -1, -1):\n # upconv_0\n num_ch_in = self.num_ch_enc[-1] if i == 4 else self.num_ch_dec[i +\n 1]\n num_ch_out = self.num_ch_dec[i]\n self.convs[(\"upconv\", i, 0)] = ConvBlock(num_ch_in, num_ch_out)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:599-631"
+ },
+ "6225": {
+ "file_id": 488,
+ "content": "The code defines a class called DepthDecoder. It takes in parameters such as number of channels, scales, output channel count, and use_skips. The class initializes various attributes like num_output_channels, use_skips, upsample_mode, and scale. It also creates an OrderedDict named 'convs' which stores ConvBlock instances based on the given parameters.",
+ "type": "comment"
+ },
+ "6226": {
+ "file_id": 488,
+ "content": " # upconv_1\n num_ch_in = self.num_ch_dec[i]\n if self.use_skips and i > 0:\n num_ch_in += self.num_ch_enc[i - 1]\n num_ch_out = self.num_ch_dec[i]\n self.convs[(\"upconv\", i, 1)] = ConvBlock(num_ch_in, num_ch_out)\n for s in self.scales:\n self.convs[(\"dispconv\", s)] = Conv3x3(self.num_ch_dec[s],\n self.num_output_channels)\n self.decoder = nn.LayerList(list(self.convs.values()))\n self.sigmoid = nn.Sigmoid()\n def forward(self, input_features):\n outputs = {}\n # decoder\n x = input_features[-1]\n for i in range(4, -1, -1):\n x = self.convs[(\"upconv\", i, 0)](x)\n x = [upsample(x)]\n if self.use_skips and i > 0:\n x += [input_features[i - 1]]\n x = paddle.concat(x, 1)\n x = self.convs[(\"upconv\", i, 1)](x)\n if i in self.scales:\n outputs[(\"disp\", i)] = self.sigmoid(self.convs[(\"dispconv\",",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:633-660"
+ },
+ "6227": {
+ "file_id": 488,
+ "content": "Code defines a convolutional network architecture for image decoding. It uses ConvBlock layers and Conv3x3 layers in the decoder section. The input features are upsampled and combined with previous encoder outputs at each stage, and then passed through convolution layers. The results are stored in 'outputs' dictionary.",
+ "type": "comment"
+ },
+ "6228": {
+ "file_id": 488,
+ "content": " i)](x))\n return outputs\nclass PoseDecoder(nn.Layer):\n def __init__(self,\n num_ch_enc,\n num_input_features,\n num_frames_to_predict_for=None,\n stride=1):\n super(PoseDecoder, self).__init__()\n self.num_ch_enc = num_ch_enc\n self.num_input_features = num_input_features\n if num_frames_to_predict_for is None:\n num_frames_to_predict_for = num_input_features - 1\n self.num_frames_to_predict_for = num_frames_to_predict_for\n self.convs = OrderedDict()\n self.convs[(\"squeeze\")] = nn.Conv2D(self.num_ch_enc[-1], 256, 1)\n self.convs[(\"pose\", 0)] = nn.Conv2D(num_input_features * 256, 256, 3,\n stride, 1)\n self.convs[(\"pose\", 1)] = nn.Conv2D(256, 256, 3, stride, 1)\n self.convs[(\"pose\", 2)] = nn.Conv2D(256, 6 * num_frames_to_predict_for,\n 1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:661-686"
+ },
+ "6229": {
+ "file_id": 488,
+ "content": "The PoseDecoder class in this code is a neural network layer that uses convolutional layers to predict pose for a given number of frames. It takes in the number of input channels, the number of input features, and an optional parameter for the number of frames to predict. The layer contains three convolution layers with different parameters for each.",
+ "type": "comment"
+ },
+ "6230": {
+ "file_id": 488,
+ "content": " self.relu = nn.ReLU()\n self.net = nn.LayerList(list(self.convs.values()))\n def forward(self, input_features):\n last_features = [f[-1] for f in input_features]\n cat_features = [\n self.relu(self.convs[\"squeeze\"](f)) for f in last_features\n ]\n cat_features = paddle.concat(cat_features, 1)\n out = cat_features\n for i in range(3):\n out = self.convs[(\"pose\", i)](out)\n if i != 2:\n out = self.relu(out)\n out = out.mean(3).mean(2)\n out = 0.01 * out.reshape([-1, self.num_frames_to_predict_for, 1, 6])\n axisangle = out[..., :3]\n translation = out[..., 3:]\n return axisangle, translation\nclass ResnetEncoder(nn.Layer):\n \"\"\"Pypaddle module for a resnet encoder\n \"\"\"\n def __init__(self, num_layers, pretrained=False, num_input_images=1):\n super(ResnetEncoder, self).__init__()\n self.num_ch_enc = np.array([64, 64, 128, 256, 512])\n resnets = {\n 18: paddle.vision.models.resnet18,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:688-725"
+ },
+ "6231": {
+ "file_id": 488,
+ "content": "The code defines a class \"Adds\" with a forward function that performs feature extraction and concatenation, followed by convolution and activation operations. The code also includes a class \"ResnetEncoder\" which is a Pypaddle implementation of a ResNet encoder.",
+ "type": "comment"
+ },
+ "6232": {
+ "file_id": 488,
+ "content": " 34: paddle.vision.models.resnet34,\n 50: paddle.vision.models.resnet50,\n 101: paddle.vision.models.resnet101,\n 152: paddle.vision.models.resnet152\n }\n if num_layers not in resnets:\n raise ValueError(\n \"{} is not a valid number of resnet layers\".format(num_layers))\n if num_input_images > 1:\n self.encoder = resnet_multiimage_input(num_layers, pretrained,\n num_input_images)\n else:\n self.encoder = resnets[num_layers](pretrained)\n if num_layers > 34:\n self.num_ch_enc[1:] *= 4\n ######################################\n # night public first conv\n ######################################\n self.conv1 = nn.Conv2D(3,\n 64,\n kernel_size=7,\n stride=2,\n padding=3,\n bias_attr=False)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:726-753"
+ },
+ "6233": {
+ "file_id": 488,
+ "content": "The code defines a function that creates a ResNet backbone model with specified layers and checks if the input has multiple images. It uses pretrained weights, adds a convolutional layer to the output of the ResNet, and scales certain channels based on the number of layers.",
+ "type": "comment"
+ },
+ "6234": {
+ "file_id": 488,
+ "content": " self.bn1 = nn.BatchNorm2D(64)\n self.relu = nn.ReLU() # NOTE\n self.conv_shared = nn.Conv2D(512, 64, kernel_size=1)\n ##########################################\n # private source encoder, day\n ##########################################\n self.encoder_day = resnets[num_layers](pretrained)\n self.conv_diff_day = nn.Conv2D(\n 512, 64, kernel_size=1) # no bn after conv, so bias=true\n ##########################################\n # private target encoder, night\n ##########################################\n self.encoder_night = resnets[num_layers](pretrained)\n self.conv_diff_night = nn.Conv2D(512, 64, kernel_size=1)\n ######################################\n # shared decoder (small decoder), use a simple de-conv to upsample the features with no skip connection\n ######################################\n self.convt5 = convt_bn_relu(in_channels=512,\n out_channels=256,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:754-776"
+ },
+ "6235": {
+ "file_id": 488,
+ "content": "This code initializes a network backbone with shared and private encoders for day and night, as well as a shared decoder. It uses BatchNorm2D, ReLU activation, Conv2D layers, and sets up convolutional blocks for the encoders and decoder.",
+ "type": "comment"
+ },
+ "6236": {
+ "file_id": 488,
+ "content": " kernel_size=3,\n stride=2,\n padding=1,\n output_padding=1)\n self.convt4 = convt_bn_relu(in_channels=256,\n out_channels=128,\n kernel_size=3,\n stride=2,\n padding=1,\n output_padding=1)\n self.convt3 = convt_bn_relu(in_channels=128,\n out_channels=64,\n kernel_size=3,\n stride=2,\n padding=1,\n output_padding=1)\n self.convt2 = convt_bn_relu(in_channels=64,\n out_channels=64,\n kernel_size=3,\n stride=2,\n padding=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:777-797"
+ },
+ "6237": {
+ "file_id": 488,
+ "content": "This code defines a series of convolutional layers with batch normalization and ReLU activation functions. The layers have different numbers of input and output channels, as well as identical kernel sizes, strides, padding, and output padding values. These layers likely form part of a deep learning model for image processing or analysis tasks.",
+ "type": "comment"
+ },
+ "6238": {
+ "file_id": 488,
+ "content": " output_padding=1)\n self.convt1 = convt_bn_relu(in_channels=64,\n out_channels=64,\n kernel_size=3,\n stride=2,\n padding=1,\n output_padding=1)\n self.convtf = nn.Conv2D(64, 3, kernel_size=1, stride=1, padding=0)\n def forward(self, input_image, is_night):\n if self.training:\n result = []\n input_data = (input_image - 0.45) / 0.225\n if is_night == 'day':\n # source private encoder, day\n private_feature = self.encoder_day.conv1(input_data)\n private_feature = self.encoder_day.bn1(private_feature)\n private_feature = self.encoder_day.relu(private_feature)\n private_feature = self.encoder_day.maxpool(private_feature)\n private_feature = self.encoder_day.layer1(private_feature)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:798-817"
+ },
+ "6239": {
+ "file_id": 488,
+ "content": "The code defines a class with an initializer for two ConvT blocks and a convolutional layer. The forward function is used for training, where it subtracts 0.45 and divides by 0.225 from the input image to normalize it, and if the 'is_night' parameter is 'day', it passes this normalized image through the day encoder blocks of the model.",
+ "type": "comment"
+ },
+ "6240": {
+ "file_id": 488,
+ "content": " private_feature = self.encoder_day.layer2(private_feature)\n private_feature = self.encoder_day.layer3(private_feature)\n private_feature = self.encoder_day.layer4(private_feature)\n private_code = self.conv_diff_day(private_feature)\n private_gram = gram_matrix(private_feature)\n result.append(private_code)\n result.append(private_gram)\n elif is_night == 'night':\n # target private encoder, night\n private_feature = self.encoder_night.conv1(input_data)\n private_feature = self.encoder_night.bn1(private_feature)\n private_feature = self.encoder_night.relu(private_feature)\n private_feature = self.encoder_night.maxpool(private_feature)\n private_feature = self.encoder_night.layer1(private_feature)\n private_feature = self.encoder_night.layer2(private_feature)\n private_feature = self.encoder_night.layer3(private_feature)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:818-834"
+ },
+ "6241": {
+ "file_id": 488,
+ "content": "The code is processing the input data through a day or night specific encoder, applying convolutions, batch normalization, ReLU activation, and max pooling. It then appends the resulting private code and gram matrix to the 'result' list.",
+ "type": "comment"
+ },
+ "6242": {
+ "file_id": 488,
+ "content": " private_feature = self.encoder_night.layer4(private_feature)\n private_code = self.conv_diff_night(private_feature)\n private_gram = gram_matrix(private_feature)\n result.append(private_code)\n result.append(private_gram)\n # shared encoder\n self.features = []\n x = (input_image - 0.45) / 0.225\n if is_night == 'day':\n x = self.encoder.conv1(x)\n x = self.encoder.bn1(x)\n self.features.append(self.encoder.relu(x))\n else:\n x = self.conv1(x)\n x = self.bn1(x)\n self.features.append(self.relu(x))\n self.features.append(\n self.encoder.layer1(self.encoder.maxpool(self.features[-1])))\n self.features.append(self.encoder.layer2(self.features[-1]))\n self.features.append(self.encoder.layer3(self.features[-1]))\n self.features.append(self.encoder.layer4(self.features[-1]))\n if self.training:\n shared_code = self.conv_shared(self.features[-1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:835-861"
+ },
+ "6243": {
+ "file_id": 488,
+ "content": "This code defines a model with two branches: one for day and one for night. It extracts features from the input image, applies different layers depending on whether it's day or night, and appends them to a list of features. Finally, it calculates a shared code for training using the last feature extracted.",
+ "type": "comment"
+ },
+ "6244": {
+ "file_id": 488,
+ "content": " shared_gram = gram_matrix(self.features[-1])\n result.append(shared_code) # use this to calculate loss of diff\n result.append(shared_gram)\n result.append(\n self.features[-1]) # use this to calculate loss of similarity\n union_code = private_feature + self.features[-1]\n rec_code = self.convt5(union_code)\n rec_code = self.convt4(rec_code)\n rec_code = self.convt3(rec_code)\n rec_code = self.convt2(rec_code)\n rec_code = self.convt1(rec_code)\n rec_code = self.convtf(rec_code)\n result.append(rec_code)\n return self.features, result\n else:\n return self.features\nclass ResnetEncoder_pose(nn.Layer):\n \"\"\"Pypaddle module for a resnet encoder\n \"\"\"\n def __init__(self, num_layers, pretrained=False, num_input_images=1):\n super(ResnetEncoder_pose, self).__init__()\n self.num_ch_enc = np.array([64, 64, 128, 256, 512])\n resnets = {",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:862-889"
+ },
+ "6245": {
+ "file_id": 488,
+ "content": "This code defines a ResnetEncoder_pose class, which is a Pypaddle module for a resnet encoder. It initializes the number of layers and whether pre-trained weights are used. The code then defines several convolutional layers (convt1 to convt5) for processing feature maps. If pretrained is set to True, the method returns the features. Otherwise, it appends the processed feature maps to a result list and returns the features and result.",
+ "type": "comment"
+ },
+ "6246": {
+ "file_id": 488,
+ "content": " 18: paddle.vision.models.resnet18,\n 34: paddle.vision.models.resnet34,\n 50: paddle.vision.models.resnet50,\n 101: paddle.vision.models.resnet101,\n 152: paddle.vision.models.resnet152\n }\n if num_layers not in resnets:\n raise ValueError(\n \"{} is not a valid number of resnet layers\".format(num_layers))\n if num_input_images > 1:\n self.encoder = resnet_multiimage_input(num_layers, num_input_images)\n else:\n self.encoder = resnets[num_layers](pretrained)\n if num_layers > 34:\n self.num_ch_enc[1:] *= 4\n def forward(self, input_image):\n features = []\n x = (input_image - 0.45) / 0.225\n x = self.encoder.conv1(x)\n x = self.encoder.bn1(x)\n features.append(self.encoder.relu(x))\n features.append(self.encoder.layer1(self.encoder.maxpool(features[-1])))\n features.append(self.encoder.layer2(features[-1]))\n features.append(self.encoder.layer3(features[-1]))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:890-917"
+ },
+ "6247": {
+ "file_id": 488,
+ "content": "This code defines a ResNet backbone model with different layers (18, 34, 50, 101, 152) and handles multi-image input cases. The encoder is initialized based on the specified number of layers, and adjusts the number of channels for layers larger than 34. The forward function extracts features from an input image through a series of ResNet layers.",
+ "type": "comment"
+ },
+ "6248": {
+ "file_id": 488,
+ "content": " features.append(self.encoder.layer4(features[-1]))\n return features\n@BACKBONES.register()\nclass ADDS_DepthNet(nn.Layer):\n def __init__(self,\n num_layers=18,\n frame_ids=[0, -1, 1],\n height=256,\n width=512,\n batch_size=6,\n pose_model_input=\"pairs\",\n use_stereo=False,\n only_depth_encoder=False,\n pretrained=None,\n scales=[0, 1, 2, 3],\n min_depth=0.1,\n max_depth=100.0,\n pose_model_type='separate_resnet',\n v1_multiscale=False,\n predictive_mask=False,\n disable_automasking=False):\n super(ADDS_DepthNet, self).__init__()\n self.num_layers = num_layers\n self.height = height\n self.width = width\n self.batch_size = batch_size\n self.frame_ids = frame_ids\n self.pose_model_input = pose_model_input\n self.use_stereo = use_stereo",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:918-949"
+ },
+ "6249": {
+ "file_id": 488,
+ "content": "This code defines the class `ADDS_DepthNet`, which is a depth estimation network, with parameters such as number of layers, frame IDs, input size, batch size, etc. It inherits from `nn.Layer` and has methods to encode poses and features. The class also registers itself at `BACKBONES`.",
+ "type": "comment"
+ },
+ "6250": {
+ "file_id": 488,
+ "content": " self.only_depth_encoder = only_depth_encoder\n self.pretrained = pretrained\n self.scales = scales\n self.pose_model_type = pose_model_type\n self.predictive_mask = predictive_mask\n self.disable_automasking = disable_automasking\n self.v1_multiscale = v1_multiscale\n self.min_depth = min_depth\n self.max_depth = max_depth\n self.num_input_frames = len(self.frame_ids)\n self.num_pose_frames = 2 if self.pose_model_input == \"pairs\" else self.num_input_frames\n assert self.frame_ids[0] == 0, \"frame_ids must start with 0\"\n self.use_pose_net = not (self.use_stereo and self.frame_ids == [0])\n self.encoder = ResnetEncoder(self.num_layers)\n if not self.only_depth_encoder:\n self.depth = DepthDecoder(self.encoder.num_ch_enc, self.scales)\n if self.use_pose_net and not self.only_depth_encoder:\n if self.pose_model_type == \"separate_resnet\":\n self.pose_encoder = ResnetEncoder_pose(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:950-972"
+ },
+ "6251": {
+ "file_id": 488,
+ "content": "The code initializes the model parameters and instances, including whether to only use the depth encoder (only_depth_encoder), if pre-trained weights are used (pretrained), and the scales for the depth decoding (scales). It also determines the number of input frames needed for both depth and pose prediction based on the provided inputs. The code creates instances of DepthDecoder, ResnetEncoder, and ResnetEncoder_pose depending on the model configuration.",
+ "type": "comment"
+ },
+ "6252": {
+ "file_id": 488,
+ "content": " self.num_layers, num_input_images=self.num_pose_frames)\n self.pose = PoseDecoder(self.pose_encoder.num_ch_enc,\n num_input_features=1,\n num_frames_to_predict_for=2)\n self.backproject_depth = {}\n self.project_3d = {}\n for scale in self.scales:\n h = self.height // (2**scale)\n w = self.width // (2**scale)\n self.backproject_depth[scale] = BackprojectDepth(\n self.batch_size, h, w)\n self.project_3d[scale] = Project3D(batch_size, h, w)\n def init_weights(self):\n \"\"\"First init model's weight\"\"\"\n for m in self.sublayers(include_self=True):\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight, a=math.sqrt(5))\n if m.bias is not None:\n fan_in, _ = _calculate_fan_in_and_fan_out(m.weight)\n bound = 1 / math.sqrt(fan_in)\n uniform_ = paddle.nn.initializer.Uniform(-bound, bound)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:973-996"
+ },
+ "6253": {
+ "file_id": 488,
+ "content": "The code initializes a backbone model by defining its layers and scales, then initializing the weights of convolutional layers using Kaiming normalization and uniform initialization for bias. This backbone model is designed for handling pose estimation tasks.",
+ "type": "comment"
+ },
+ "6254": {
+ "file_id": 488,
+ "content": " uniform_(m.bias)\n \"\"\"Second, if provide pretrained ckpt, load it\"\"\"\n if self.pretrained: # load pretrained weights\n load_ckpt(self, self.pretrained)\n def forward(self, inputs, day_or_night='day'):\n if self.training:\n features, result = self.encoder(inputs[\"color_aug\", 0, 0], 'day')\n features_night, result_night = self.encoder(\n inputs[(\"color_n_aug\", 0, 0)], 'night')\n outputs = self.depth(features)\n outputs_night = self.depth(features_night)\n if self.use_pose_net and not self.only_depth_encoder:\n outputs.update(self.predict_poses(inputs, 'day'))\n outputs_night.update(self.predict_poses(inputs, 'night'))\n self.generate_images_pred(inputs, outputs, 'day')\n self.generate_images_pred(inputs, outputs_night, 'night')\n outputs['frame_ids'] = self.frame_ids\n outputs['scales'] = self.scales\n outputs['result'] = result",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:997-1019"
+ },
+ "6255": {
+ "file_id": 488,
+ "content": "This code defines a forward function for a backbone model. It applies the encoder to inputs and uses the depth module to extract features. If pose prediction is enabled, it adds poses to the output dictionary, generates images, and stores frame IDs and scales in the outputs dictionary. This function handles both day and night scenarios.",
+ "type": "comment"
+ },
+ "6256": {
+ "file_id": 488,
+ "content": " outputs['result_night'] = result_night\n outputs_night['frame_ids'] = self.frame_ids\n outputs_night['scales'] = self.scales\n outputs['outputs_night'] = outputs_night\n else:\n if isinstance(inputs, dict):\n input_color = inputs[(\"color\", 0, 0)]\n features = self.encoder(input_color, day_or_night[0])\n outputs = self.depth(features)\n pred_disp, _ = disp_to_depth(outputs[(\"disp\", 0)],\n self.min_depth, self.max_depth)\n pred_disp = pred_disp[:, 0].numpy()\n outputs['pred_disp'] = np.squeeze(pred_disp)\n outputs['gt'] = np.squeeze(inputs['depth_gt'].numpy())\n else:\n input_color = inputs\n features = self.encoder(input_color, day_or_night)\n outputs = self.depth(features)\n pred_disp, _ = disp_to_depth(outputs[(\"disp\", 0)],\n self.min_depth, self.max_depth)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1020-1044"
+ },
+ "6257": {
+ "file_id": 488,
+ "content": "This code handles both dictionary and non-dictionary inputs for a model. If the input is a dictionary, it selects the 'color' input and processes accordingly. It uses an encoder to extract features from the input, then passes those features through a depth function to get predictions. The predictions are converted to depth format, and the final outputs include pred_disp and gt (ground truth) for further processing.",
+ "type": "comment"
+ },
+ "6258": {
+ "file_id": 488,
+ "content": " pred_disp = pred_disp[:, 0]\n outputs = paddle.squeeze(pred_disp)\n return outputs\n def predict_poses(self, inputs, is_night):\n \"\"\"Predict poses between input frames for monocular sequences.\n \"\"\"\n outputs = {}\n if self.num_pose_frames == 2:\n if is_night:\n pose_feats = {\n f_i: inputs[\"color_n_aug\", f_i, 0]\n for f_i in self.frame_ids\n }\n else:\n pose_feats = {\n f_i: inputs[\"color_aug\", f_i, 0]\n for f_i in self.frame_ids\n }\n for f_i in self.frame_ids[1:]:\n if f_i != \"s\":\n if f_i < 0:\n pose_inputs = [pose_feats[f_i], pose_feats[0]]\n else:\n pose_inputs = [pose_feats[0], pose_feats[f_i]]\n if self.pose_model_type == \"separate_resnet\":\n pose_inputs = [",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1046-1074"
+ },
+ "6259": {
+ "file_id": 488,
+ "content": "This code is defining a function to predict poses between input frames for monocular sequences. It takes inputs as parameters and checks if the number of pose frames is 2. If so, it applies different treatments based on whether it's night or day. For night, it uses color_n_aug; for day, it uses color_aug. Then, it iterates through the frame IDs, excluding 's', and prepares inputs accordingly. The pose model type is \"separate_resnet\".",
+ "type": "comment"
+ },
+ "6260": {
+ "file_id": 488,
+ "content": " self.pose_encoder(paddle.concat(pose_inputs,\n axis=1))\n ]\n axisangle, translation = self.pose(pose_inputs)\n outputs[(\"axisangle\", 0, f_i)] = axisangle\n outputs[(\"translation\", 0, f_i)] = translation\n # Invert the matrix if the frame id is negative\n outputs[(\"cam_T_cam\", 0,\n f_i)] = transformation_from_parameters(\n axisangle[:, 0],\n translation[:, 0],\n invert=(f_i < 0))\n return outputs\n def generate_images_pred(self, inputs, outputs, is_night):\n \"\"\"Generate the warped (reprojected) color images for a minibatch.\n Generated images are saved into the `outputs` dictionary.\n \"\"\"\n _, _, height, width = inputs['color', 0, 0].shape\n for scale in self.scales:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1075-1096"
+ },
+ "6261": {
+ "file_id": 488,
+ "content": "This code segment defines a function that calculates pose, axisangle, translation, and camera transformation parameters for an image. It takes input from the \"pose_encoder\" function, combines them, and assigns the results to specific positions in the \"outputs\" dictionary. If the frame ID is negative, it inverts the calculated matrix. The code also initializes a nested loop over different scales and generates warped color images for a given batch of inputs.",
+ "type": "comment"
+ },
+ "6262": {
+ "file_id": 488,
+ "content": " disp = outputs[(\"disp\", scale)]\n if self.v1_multiscale:\n source_scale = scale\n else:\n disp = F.interpolate(disp, [height, width],\n mode=\"bilinear\",\n align_corners=False)\n source_scale = 0\n _, depth = disp_to_depth(disp, self.min_depth, self.max_depth)\n outputs[(\"depth\", 0, scale)] = depth\n for i, frame_id in enumerate(self.frame_ids[1:]):\n T = outputs[(\"cam_T_cam\", 0, frame_id)]\n cam_points = self.backproject_depth[source_scale](\n depth, inputs[(\"inv_K\", source_scale)])\n pix_coords = self.project_3d[source_scale](\n cam_points, inputs[(\"K\", source_scale)], T)\n outputs[(\"sample\", frame_id, scale)] = pix_coords\n if is_night:\n inputs[(\"color_n\", frame_id,\n source_scale)].stop_gradient = False",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1097-1122"
+ },
+ "6263": {
+ "file_id": 488,
+ "content": "The code interpolates the displacement output based on the scale, and if multiscale is not enabled, it performs bilinear interpolation to match the input size. It then converts the displacement into depth using disp_to_depth function. Depth and its corresponding scale are added to the outputs. For each frame ID in the list, it retrieves camera transformation matrix T, backprojects depth to 3D coordinates, projects them onto image plane using project_3d, and adds the resulting pixel coordinates to the outputs. If is_night is True, it modifies the color_n input's stop_gradient attribute.",
+ "type": "comment"
+ },
+ "6264": {
+ "file_id": 488,
+ "content": " outputs[(\"color\", frame_id,\n scale)] = paddle.nn.functional.grid_sample(\n inputs[(\"color_n\", frame_id, source_scale)],\n outputs[(\"sample\", frame_id, scale)],\n padding_mode=\"border\",\n align_corners=False)\n else:\n inputs[(\"color\", frame_id,\n source_scale)].stop_gradient = False\n outputs[(\"color\", frame_id,\n scale)] = paddle.nn.functional.grid_sample(\n inputs[(\"color\", frame_id, source_scale)],\n outputs[(\"sample\", frame_id, scale)],\n padding_mode=\"border\",\n align_corners=False)\n if not self.disable_automasking:\n if is_night:\n outputs[(\"color_identity\", frame_id, scale)] = \\",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1123-1142"
+ },
+ "6265": {
+ "file_id": 488,
+ "content": "This code performs grid sampling on a tensor and assigns the result to a specific location in the outputs dictionary based on frame_id and scale. If disable_automasking is True, it also creates an identity mask for night scenes.",
+ "type": "comment"
+ },
+ "6266": {
+ "file_id": 488,
+ "content": " inputs[(\"color_n\", frame_id, source_scale)]\n else:\n outputs[(\"color_identity\", frame_id, scale)] = \\\n inputs[(\"color\", frame_id, source_scale)]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/adds.py:1143-1146"
+ },
+ "6267": {
+ "file_id": 488,
+ "content": "This code is selecting the input data from a dictionary based on specific conditions. If the frame_id and source_scale match, it assigns the value to \"color_n\". Otherwise, it assigns the value of \"color\" input to \"color_identity\".",
+ "type": "comment"
+ },
+ "6268": {
+ "file_id": 489,
+ "content": "/paddlevideo/modeling/backbones/agcn.py",
+ "type": "filepath"
+ },
+ "6269": {
+ "file_id": 489,
+ "content": "The code defines a PaddlePaddle GCN class with convolutional blocks for temporal sequences, utilizing layers such as batch normalization and residual connections. It also presents a custom AGCN backbone model for graph convolution tasks using adaptive graph convolutions.",
+ "type": "summary"
+ },
+ "6270": {
+ "file_id": 489,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nclass GCN(nn.Layer):\n def __init__(self, in_channels, out_channels, vertex_nums=25, stride=1):\n super(GCN, self).__init__()\n self.conv1 = nn.Conv2D(in_channels=in_channels,\n out_channels=3 * out_channels,\n kernel_size=1,\n stride=1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn.py:1-27"
+ },
+ "6271": {
+ "file_id": 489,
+ "content": "The code is defining a GCN (Graph Convolutional Network) class within the PaddlePaddle framework. It takes in channel dimensions, output channel dimensions, vertex numbers and stride as parameters for its constructor. The class has one convolution layer with kernel size of 1 and stride of 1.",
+ "type": "comment"
+ },
+ "6272": {
+ "file_id": 489,
+ "content": " self.conv2 = nn.Conv2D(in_channels=vertex_nums * 3,\n out_channels=vertex_nums,\n kernel_size=1)\n def forward(self, x):\n # x --- N,C,T,V\n x = self.conv1(x) # N,3C,T,V\n N, C, T, V = x.shape\n x = paddle.reshape(x, [N, C // 3, 3, T, V]) # N,C,3,T,V\n x = paddle.transpose(x, perm=[0, 1, 2, 4, 3]) # N,C,3,V,T\n x = paddle.reshape(x, [N, C // 3, 3 * V, T]) # N,C,3V,T\n x = paddle.transpose(x, perm=[0, 2, 1, 3]) # N,3V,C,T\n x = self.conv2(x) # N,V,C,T\n x = paddle.transpose(x, perm=[0, 2, 3, 1]) # N,C,T,V\n return x\nclass Block(paddle.nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n vertex_nums=25,\n temporal_size=9,\n stride=1,\n residual=True):\n super(Block, self).__init__()\n self.residual = residual\n self.out_channels = out_channels\n self.bn_res = nn.BatchNorm2D(out_channels)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn.py:28-57"
+ },
+ "6273": {
+ "file_id": 489,
+ "content": "The code defines a convolutional block for processing temporal sequences with 3D spatial-temporal convolutions. The block applies multiple convolution layers, batch normalization, and transposes the dimensions to perform feature extraction from the input sequence. It is parameterized by the number of channels, output channels, vertex numbers, temporal size, and a flag for residual connections.",
+ "type": "comment"
+ },
+ "6274": {
+ "file_id": 489,
+ "content": " self.conv_res = nn.Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n stride=(stride, 1))\n self.gcn = GCN(in_channels=in_channels,\n out_channels=out_channels,\n vertex_nums=vertex_nums)\n self.tcn = nn.Sequential(\n nn.BatchNorm2D(out_channels),\n nn.ReLU(),\n nn.Conv2D(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=(temporal_size, 1),\n padding=((temporal_size - 1) // 2, 0),\n stride=(stride, 1)),\n nn.BatchNorm2D(out_channels),\n )\n def forward(self, x):\n if self.residual:\n y = self.conv_res(x)\n y = self.bn_res(y)\n x = self.gcn(x)\n x = self.tcn(x)\n out = x + y if self.residual else x\n out = F.relu(out)\n return out",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn.py:58-84"
+ },
+ "6275": {
+ "file_id": 489,
+ "content": "This code initializes a convolutional residual block with a Graph Convolutional Network (GCN) and Temporal Convolutional Network (TCN). The conv_res is a 1x1 convolution, gcn is a GCN layer, and tcn is a TCN layer. In the forward pass, if residual is True, the input goes through the conv_res layer before being passed to the gcn layer, then the tcn layer. The output is either the sum of the input and the residual (if residual is True) or just the output of the GCN layer, which is then passed through a ReLU activation function.",
+ "type": "comment"
+ },
+ "6276": {
+ "file_id": 489,
+ "content": "@BACKBONES.register()\nclass AGCN(nn.Layer):\n \"\"\"\n AGCN model improves the performance of ST-GCN using\n Adaptive Graph Convolutional Networks.\n Args:\n in_channels: int, channels of vertex coordinate. 2 for (x,y), 3 for (x,y,z). Default 2.\n \"\"\"\n def __init__(self, in_channels=2, **kwargs):\n super(AGCN, self).__init__()\n self.data_bn = nn.BatchNorm1D(25 * 2)\n self.agcn = nn.Sequential(\n Block(in_channels=in_channels,\n out_channels=64,\n residual=False,\n **kwargs), Block(in_channels=64, out_channels=64, **kwargs),\n Block(in_channels=64, out_channels=64, **kwargs),\n Block(in_channels=64, out_channels=64, **kwargs),\n Block(in_channels=64, out_channels=128, stride=2, **kwargs),\n Block(in_channels=128, out_channels=128, **kwargs),\n Block(in_channels=128, out_channels=128, **kwargs),\n Block(in_channels=128, out_channels=256, stride=2, **kwargs),\n Block(in_channels=256, out_channels=256, **kwargs),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn.py:87-110"
+ },
+ "6277": {
+ "file_id": 489,
+ "content": "The code defines a class AGCN (Adaptive Graph Convolutional Network) as a subclass of nn.Layer, which is an improved version of ST-GCN for graph convolution tasks using adaptive graph convolutions. The model architecture consists of several Block layers with varying in_channels and out_channels, and downsampling is performed with stride=2.",
+ "type": "comment"
+ },
+ "6278": {
+ "file_id": 489,
+ "content": " Block(in_channels=256, out_channels=256, **kwargs))\n self.pool = nn.AdaptiveAvgPool2D(output_size=(1, 1))\n def forward(self, x):\n # data normalization\n N, C, T, V, M = x.shape\n x = x.transpose((0, 4, 1, 2, 3)) # N, M, C, T, V\n x = x.reshape((N * M, C, T, V))\n x = self.agcn(x)\n x = self.pool(x) # NM,C,T,V --> NM,C,1,1\n C = x.shape[1]\n x = paddle.reshape(x, (N, M, C, 1, 1)).mean(axis=1) # N,C,1,1\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn.py:111-128"
+ },
+ "6279": {
+ "file_id": 489,
+ "content": "This code defines a custom backbone for AGCN model with a block of 256 in_channels and out_channels, followed by an adaptive average pooling layer. The forward function performs data normalization, transposes the shape, reshapes it, applies the AGCN layer, pools it to size (1,1), and finally reshapes and averages along one axis before returning the result.",
+ "type": "comment"
+ },
+ "6280": {
+ "file_id": 490,
+ "content": "/paddlevideo/modeling/backbones/agcn2s.py",
+ "type": "filepath"
+ },
+ "6281": {
+ "file_id": 490,
+ "content": "The code implements temporal convolutional networks and GCN units in PaddlePaddle, creating a Graph class and AGCN2s graph convolution layer for the NTURGB+D dataset. This involves initializing variables, obtaining adjacency matrix, normalization, and executing convolutions.",
+ "type": "summary"
+ },
+ "6282": {
+ "file_id": 490,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport numpy as np\nfrom ..registry import BACKBONES\ndef import_class(name):\n components = name.split('.')\n mod = __import__(components[0])\n for comp in components[1:]:\n mod = getattr(mod, comp)\n return mod\nclass UnitTCN(nn.Layer):\n def __init__(self, in_channels, out_channels, kernel_size=9, stride=1):\n super(UnitTCN, self).__init__()\n pad = int((kernel_size - 1) / 2)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:1-32"
+ },
+ "6283": {
+ "file_id": 490,
+ "content": "This code defines a class named \"UnitTCN\" which is a type of layer for temporal convolutional network. It's implemented using PaddlePaddle library and includes methods to define the convolutional layers with specified number of input and output channels, kernel size and stride. The class is registered in the BACKBONES registry of the PaddleVideo module.",
+ "type": "comment"
+ },
+ "6284": {
+ "file_id": 490,
+ "content": " self.conv = nn.Conv2D(in_channels,\n out_channels,\n kernel_size=(kernel_size, 1),\n padding=(pad, 0),\n stride=(stride, 1))\n self.bn = nn.BatchNorm2D(out_channels)\n self.relu = nn.ReLU()\n def forward(self, x):\n \" input size : (N*M, C, T, V)\"\n x = self.bn(self.conv(x))\n return x\nclass UnitGCN(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n A,\n coff_embedding=4,\n num_subset=3):\n super(UnitGCN, self).__init__()\n inter_channels = out_channels // coff_embedding\n self.inter_c = inter_channels\n PA = self.create_parameter(shape=A.shape, dtype='float32')\n self.PA = PA\n self.A = paddle.to_tensor(A.astype(np.float32))\n self.num_subset = num_subset\n self.conv_a = nn.LayerList()\n self.conv_b = nn.LayerList()\n self.conv_d = nn.LayerList()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:33-65"
+ },
+ "6285": {
+ "file_id": 490,
+ "content": "This code defines a GCN unit class with convolutional layers for learning spatio-temporal features. It uses batch normalization and ReLU activation, allowing the model to learn representations from the input data. The GCN unit takes in channels, output channels, adjacency matrix A, coefficient embedding, and number of subsets as parameters.",
+ "type": "comment"
+ },
+ "6286": {
+ "file_id": 490,
+ "content": " for i in range(self.num_subset):\n self.conv_a.append(nn.Conv2D(in_channels, inter_channels, 1))\n self.conv_b.append(nn.Conv2D(in_channels, inter_channels, 1))\n self.conv_d.append(nn.Conv2D(in_channels, out_channels, 1))\n if in_channels != out_channels:\n self.down = nn.Sequential(nn.Conv2D(in_channels, out_channels, 1),\n nn.BatchNorm2D(out_channels))\n else:\n self.down = lambda x: x\n self.bn = nn.BatchNorm2D(out_channels)\n self.soft = nn.Softmax(-2)\n self.relu = nn.ReLU()\n def forward(self, x):\n N, C, T, V = x.shape\n A = self.A + self.PA\n y = None\n for i in range(self.num_subset):\n A1 = paddle.transpose(self.conv_a[i](x),\n perm=[0, 3, 1,\n 2]).reshape([N, V, self.inter_c * T])\n A2 = self.conv_b[i](x).reshape([N, self.inter_c * T, V])\n A1 = self.soft(paddle.matmul(A1, A2) / A1.shape[-1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:66-91"
+ },
+ "6287": {
+ "file_id": 490,
+ "content": "This code defines a neural network backbone for the AGCN2S model. It initializes and appends convolutional layers, checks if input and output channels are different to determine whether to add a downsampling layer, and defines softmax, batch normalization, and ReLU activation functions. The forward function performs operations on input data to produce the final output.",
+ "type": "comment"
+ },
+ "6288": {
+ "file_id": 490,
+ "content": " A1 = A1 + A[i]\n A2 = x.reshape([N, C * T, V])\n z = self.conv_d[i](paddle.matmul(A2, A1).reshape([N, C, T, V]))\n y = z + y if y is not None else z\n y = self.bn(y)\n y += self.down(x)\n return self.relu(y)\nclass Block(nn.Layer):\n def __init__(self, in_channels, out_channels, A, stride=1, residual=True):\n super(Block, self).__init__()\n self.gcn1 = UnitGCN(in_channels, out_channels, A)\n self.tcn1 = UnitTCN(out_channels, out_channels, stride=stride)\n self.relu = nn.ReLU()\n if not residual:\n self.residual = lambda x: 0\n elif (in_channels == out_channels) and (stride == 1):\n self.residual = lambda x: x\n else:\n self.residual = UnitTCN(in_channels,\n out_channels,\n kernel_size=1,\n stride=stride)\n def forward(self, x):\n x = self.tcn1(self.gcn1(x)) + self.residual(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:92-121"
+ },
+ "6289": {
+ "file_id": 490,
+ "content": "The code defines a block class for a neural network architecture. It consists of GCN and TCN units in series, followed by a ReLU activation function. The residual connection is either set to zero or equal to the input if not specified, allowing for identity shortcuts within the network. The forward method combines the outputs from GCN and TCN with residual connections.",
+ "type": "comment"
+ },
+ "6290": {
+ "file_id": 490,
+ "content": " return self.relu(x)\n# This Graph structure is for the NTURGB+D dataset. If you use a custom dataset, modify num_node and the corresponding graph adjacency structure.\nclass Graph:\n def __init__(self, labeling_mode='spatial'):\n num_node = 25\n self_link = [(i, i) for i in range(num_node)]\n inward_ori_index = [(1, 2), (2, 21), (3, 21), (4, 3), (5, 21), (6, 5),\n (7, 6), (8, 7), (9, 21), (10, 9), (11, 10),\n (12, 11), (13, 1), (14, 13), (15, 14), (16, 15),\n (17, 1), (18, 17), (19, 18), (20, 19), (22, 23),\n (23, 8), (24, 25), (25, 12)]\n inward = [(i - 1, j - 1) for (i, j) in inward_ori_index]\n outward = [(j, i) for (i, j) in inward]\n neighbor = inward + outward\n self.num_node = num_node\n self.self_link = self_link\n self.inward = inward\n self.outward = outward\n self.neighbor = neighbor\n self.A = self.get_adjacency_matrix(labeling_mode)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:122-144"
+ },
+ "6291": {
+ "file_id": 490,
+ "content": "This code defines a Graph class with a fixed number of nodes (25) and connectivity patterns for the NTURGB+D dataset. It initializes self_link, inward, outward, and neighbor variables based on the specified labeling mode ('spatial' by default). The adjacency matrix is obtained using get_adjacency_matrix method.",
+ "type": "comment"
+ },
+ "6292": {
+ "file_id": 490,
+ "content": " def edge2mat(self, link, num_node):\n A = np.zeros((num_node, num_node))\n for i, j in link:\n A[j, i] = 1\n return A\n def normalize_digraph(self, A):\n Dl = np.sum(A, 0)\n h, w = A.shape\n Dn = np.zeros((w, w))\n for i in range(w):\n if Dl[i] > 0:\n Dn[i, i] = Dl[i]**(-1)\n AD = np.dot(A, Dn)\n return AD\n def get_spatial_graph(self, num_node, self_link, inward, outward):\n I = self.edge2mat(self_link, num_node)\n In = self.normalize_digraph(self.edge2mat(inward, num_node))\n Out = self.normalize_digraph(self.edge2mat(outward, num_node))\n A = np.stack((I, In, Out))\n return A\n def get_adjacency_matrix(self, labeling_mode=None):\n if labeling_mode is None:\n return self.A\n if labeling_mode == 'spatial':\n A = self.get_spatial_graph(self.num_node, self.self_link,\n self.inward, self.outward)\n else:\n raise ValueError()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:146-176"
+ },
+ "6293": {
+ "file_id": 490,
+ "content": "The code defines three functions: `edge2mat()`, `normalize_digraph()`, and `get_spatial_graph()`. `edge2mat()` converts a list of edges into an adjacency matrix. `normalize_digraph()` normalizes a directed graph by computing the in-degree for each node. `get_spatial_graph()` combines the adjacency matrices from self-links, incoming edges, and outgoing edges into one matrix. The last function `get_adjacency_matrix()` returns the adjacency matrix depending on the given labeling mode (default or spatial).",
+ "type": "comment"
+ },
+ "6294": {
+ "file_id": 490,
+ "content": " return A\n@BACKBONES.register()\nclass AGCN2s(nn.Layer):\n def __init__(self,\n num_point=25,\n num_person=2,\n graph='ntu_rgb_d',\n graph_args=dict(),\n in_channels=3):\n super(AGCN2s, self).__init__()\n if graph == 'ntu_rgb_d':\n self.graph = Graph(**graph_args)\n else:\n raise ValueError()\n A = self.graph.A\n self.data_bn = nn.BatchNorm1D(num_person * in_channels * num_point)\n self.l1 = Block(in_channels, 64, A, residual=False)\n self.l2 = Block(64, 64, A)\n self.l3 = Block(64, 64, A)\n self.l4 = Block(64, 64, A)\n self.l5 = Block(64, 128, A, stride=2)\n self.l6 = Block(128, 128, A)\n self.l7 = Block(128, 128, A)\n self.l8 = Block(128, 256, A, stride=2)\n self.l9 = Block(256, 256, A)\n self.l10 = Block(256, 256, A)\n def forward(self, x):\n N, C, T, V, M = x.shape\n x = x.transpose([0, 4, 3, 1, 2]).reshape_([N, M * V * C, T])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:177-212"
+ },
+ "6295": {
+ "file_id": 490,
+ "content": "Class AGCN2s defines a neural network layer for graph convolutions. It takes parameters such as number of points, persons, and the type of graph. The code initializes graph adjacency matrix 'A' from the specified graph and creates several Block layers for convolution operations with different parameters and strides. In forward pass, it rearranges the input tensor dimensions and reshapes it before performing graph convolutions.",
+ "type": "comment"
+ },
+ "6296": {
+ "file_id": 490,
+ "content": " x = self.data_bn(x)\n x = x.reshape_([N, M, V, C,\n T]).transpose([0, 1, 3, 4,\n 2]).reshape_([N * M, C, T, V])\n x = self.l1(x)\n x = self.l2(x)\n x = self.l3(x)\n x = self.l4(x)\n x = self.l5(x)\n x = self.l6(x)\n x = self.l7(x)\n x = self.l8(x)\n x = self.l9(x)\n x = self.l10(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/agcn2s.py:213-229"
+ },
+ "6297": {
+ "file_id": 490,
+ "content": "The code performs the following operations: \n1. Applies data normalization to x using self.data_bn.\n2. Reshapes x with dimensions N, M, V, C, and T to (N*M,C,T,V).\n3. Passes x through ten linear layers (l1 to l10) for transformation.\n4. Finally, returns the transformed x.",
+ "type": "comment"
+ },
+ "6298": {
+ "file_id": 491,
+ "content": "/paddlevideo/modeling/backbones/asrf.py",
+ "type": "filepath"
+ },
+ "6299": {
+ "file_id": 491,
+ "content": "The code imports libraries, registers a backbone model in PaddleVideo, initializes an ASRF class for computer vision tasks, and sets layer biases using init_bias function. The ASRF forward method performs convolution on input x and iterates through shared layers before returning the output.",
+ "type": "summary"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/63.json b/docs/data/63.json
new file mode 100644
index 000000000..dafc529d0
--- /dev/null
+++ b/docs/data/63.json
@@ -0,0 +1,547 @@
+{
+ "6300": {
+ "file_id": 491,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# https://github.com/yabufarha/ms-tcn/blob/master/model.py\n# https://github.com/yiskw713/asrf/libs/models/tcn.py\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nimport copy\nimport random\nimport math\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom .ms_tcn import DilatedResidualLayer\nfrom ..framework.segmenters.utils import init_bias, KaimingUniform_like_torch",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/asrf.py:1-30"
+ },
+ "6301": {
+ "file_id": 491,
+ "content": "This code block is importing necessary libraries and modules, as well as registering a backbone model within the PaddleVideo framework. It also includes references to external repositories for inspiration or implementation guidance.",
+ "type": "comment"
+ },
+ "6302": {
+ "file_id": 491,
+ "content": "@BACKBONES.register()\nclass ASRF(nn.Layer):\n def __init__(self, in_channel, num_features, num_classes, num_stages,\n num_layers):\n super().__init__()\n self.in_channel = in_channel\n self.num_features = num_features\n self.num_classes = num_classes\n self.num_stages = num_stages\n self.num_layers = num_layers\n # define layers\n self.conv_in = nn.Conv1D(self.in_channel, self.num_features, 1)\n shared_layers = [\n DilatedResidualLayer(2**i, self.num_features, self.num_features)\n for i in range(self.num_layers)\n ]\n self.shared_layers = nn.LayerList(shared_layers)\n self.init_weights()\n def init_weights(self):\n \"\"\"\n initialize model layers' weight\n \"\"\"\n # init weight\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv1D):\n layer.weight.set_value(\n KaimingUniform_like_torch(layer.weight).astype('float32'))\n if layer.bias is not None:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/asrf.py:33-65"
+ },
+ "6303": {
+ "file_id": 491,
+ "content": "The ASRF class is a type of backbone model for computer vision tasks. It initializes convolutional layers and shared dilated residual layers, and sets their weights using KaimingUniform initialization. The number of features, stages, and layers are configurable parameters.",
+ "type": "comment"
+ },
+ "6304": {
+ "file_id": 491,
+ "content": " layer.bias.set_value(\n init_bias(layer.weight, layer.bias).astype('float32'))\n def forward(self, x):\n \"\"\" ASRF forward\n \"\"\"\n out = self.conv_in(x)\n for layer in self.shared_layers:\n out = layer(out)\n return out",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/asrf.py:66-75"
+ },
+ "6305": {
+ "file_id": 491,
+ "content": "This code sets the initial values of layer biases using init_bias function. The ASRF forward method performs convolution on input x, then iterates through shared layers to modify the output before returning it.",
+ "type": "comment"
+ },
+ "6306": {
+ "file_id": 492,
+ "content": "/paddlevideo/modeling/backbones/bmn.py",
+ "type": "filepath"
+ },
+ "6307": {
+ "file_id": 492,
+ "content": "This function creates mask matrices and BMN class in Paddle.ai, initializes 2D convolutional layers for the BMSN backbone, and defines a video analysis model with layers, activation functions, and returns processed input xp.",
+ "type": "summary"
+ },
+ "6308": {
+ "file_id": 492,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport numpy as np\nimport paddle\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\ndef _get_interp1d_bin_mask(seg_xmin, seg_xmax, tscale, num_sample,\n num_sample_perbin):\n \"\"\" generate sample mask for a boundary-matching pair \"\"\"\n plen = float(seg_xmax - seg_xmin)\n plen_sample = plen / (num_sample * num_sample_perbin - 1.0)\n total_samples = [\n seg_xmin + plen_sample * ii",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:1-28"
+ },
+ "6309": {
+ "file_id": 492,
+ "content": "This function generates a sample mask for a boundary-matching pair. It calculates the number of samples per bin and total samples based on segment bounds, total length, and desired numbers of samples.",
+ "type": "comment"
+ },
+ "6310": {
+ "file_id": 492,
+ "content": " for ii in range(num_sample * num_sample_perbin)\n ]\n p_mask = []\n for idx in range(num_sample):\n bin_samples = total_samples[idx * num_sample_perbin:(idx + 1) *\n num_sample_perbin]\n bin_vector = np.zeros([tscale])\n for sample in bin_samples:\n sample_upper = math.ceil(sample)\n sample_decimal, sample_down = math.modf(sample)\n if (tscale - 1) >= int(sample_down) >= 0:\n bin_vector[int(sample_down)] += 1 - sample_decimal\n if (tscale - 1) >= int(sample_upper) >= 0:\n bin_vector[int(sample_upper)] += sample_decimal\n bin_vector = 1.0 / num_sample_perbin * bin_vector\n p_mask.append(bin_vector)\n p_mask = np.stack(p_mask, axis=1)\n return p_mask\ndef get_interp1d_mask(tscale, dscale, prop_boundary_ratio, num_sample,\n num_sample_perbin):\n \"\"\" generate sample mask for each point in Boundary-Matching Map \"\"\"\n mask_mat = []\n for start_index in range(tscale):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:29-53"
+ },
+ "6311": {
+ "file_id": 492,
+ "content": "This code generates sample masks for each point in a Boundary-Matching Map. It iterates through samples, creates binary vectors for each, and then scales them to obtain the final mask. The resulting masks are stored in an array and returned.",
+ "type": "comment"
+ },
+ "6312": {
+ "file_id": 492,
+ "content": " mask_mat_vector = []\n for duration_index in range(dscale):\n if start_index + duration_index < tscale:\n p_xmin = start_index\n p_xmax = start_index + duration_index\n center_len = float(p_xmax - p_xmin) + 1\n sample_xmin = p_xmin - center_len * prop_boundary_ratio\n sample_xmax = p_xmax + center_len * prop_boundary_ratio\n p_mask = _get_interp1d_bin_mask(sample_xmin, sample_xmax,\n tscale, num_sample,\n num_sample_perbin)\n else:\n p_mask = np.zeros([tscale, num_sample])\n mask_mat_vector.append(p_mask)\n mask_mat_vector = np.stack(mask_mat_vector, axis=2)\n mask_mat.append(mask_mat_vector)\n mask_mat = np.stack(mask_mat, axis=3)\n mask_mat = mask_mat.astype(np.float32)\n sample_mask = np.reshape(mask_mat, [tscale, -1])\n return sample_mask\ndef init_params(name, in_channels, kernel_size):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:54-77"
+ },
+ "6313": {
+ "file_id": 492,
+ "content": "This code generates mask matrices for video frames. It iterates over different duration scales and starts from a given start index. For each duration scale, it creates binary masks using interpolation. If the duration is smaller than the total time scale, it adjusts the sample range to include boundaries. Zero paddings are used if the duration exceeds the total time scale. The generated mask vectors are stacked together and reshaped for final output.",
+ "type": "comment"
+ },
+ "6314": {
+ "file_id": 492,
+ "content": " fan_in = in_channels * kernel_size * 1\n k = 1. / math.sqrt(fan_in)\n param_attr = ParamAttr(name=name,\n initializer=paddle.nn.initializer.Uniform(low=-k,\n high=k))\n return param_attr\n@BACKBONES.register()\nclass BMN(paddle.nn.Layer):\n \"\"\"BMN model from\n `\"BMN: Boundary-Matching Network for Temporal Action Proposal Generation\" `_\n Args:\n tscale (int): sequence length, default 100.\n dscale (int): max duration length, default 100.\n prop_boundary_ratio (float): ratio of expanded temporal region in proposal boundary, default 0.5.\n num_sample (int): number of samples betweent starting boundary and ending boundary of each propoasl, default 32.\n num_sample_perbin (int): number of selected points in each sample, default 3.\n \"\"\"\n def __init__(\n self,\n tscale,\n dscale,\n prop_boundary_ratio,\n num_sample,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:78-103"
+ },
+ "6315": {
+ "file_id": 492,
+ "content": "This code defines a BMN class as a Paddle.ai layer implementing the BMN model for temporal action proposal generation from the paper \"BMN: Boundary-Matching Network for Temporal Action Proposal Generation\". It has parameters tscale, dscale, prop_boundary_ratio, num_sample, and num_sample_perbin which determine the sequence length, max duration length, ratio of expanded temporal region in proposal boundary, number of samples between starting and ending boundaries of each proposal, and number of selected points in each sample respectively. The code also initializes a ParamAttr with Uniform initializer for weight initialization.",
+ "type": "comment"
+ },
+ "6316": {
+ "file_id": 492,
+ "content": " num_sample_perbin,\n feat_dim=400,\n ):\n super(BMN, self).__init__()\n #init config\n self.feat_dim = feat_dim\n self.tscale = tscale\n self.dscale = dscale\n self.prop_boundary_ratio = prop_boundary_ratio\n self.num_sample = num_sample\n self.num_sample_perbin = num_sample_perbin\n self.hidden_dim_1d = 256\n self.hidden_dim_2d = 128\n self.hidden_dim_3d = 512\n # Base Module\n self.b_conv1 = paddle.nn.Conv1D(\n in_channels=self.feat_dim,\n out_channels=self.hidden_dim_1d,\n kernel_size=3,\n padding=1,\n groups=4,\n weight_attr=init_params('Base_1_w', self.feat_dim, 3),\n bias_attr=init_params('Base_1_b', self.feat_dim, 3))\n self.b_conv1_act = paddle.nn.ReLU()\n self.b_conv2 = paddle.nn.Conv1D(\n in_channels=self.hidden_dim_1d,\n out_channels=self.hidden_dim_1d,\n kernel_size=3,\n padding=1,\n groups=4,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:104-137"
+ },
+ "6317": {
+ "file_id": 492,
+ "content": "This code defines the BMN class, which is a backbone model. It initializes parameters and includes convolutional layers with ReLU activation functions for feature extraction. The code also includes instance variables for controlling the model's behavior and dimensionality of the hidden states.",
+ "type": "comment"
+ },
+ "6318": {
+ "file_id": 492,
+ "content": " weight_attr=init_params('Base_2_w', self.hidden_dim_1d, 3),\n bias_attr=init_params('Base_2_b', self.hidden_dim_1d, 3))\n self.b_conv2_act = paddle.nn.ReLU()\n # Temporal Evaluation Module\n self.ts_conv1 = paddle.nn.Conv1D(\n in_channels=self.hidden_dim_1d,\n out_channels=self.hidden_dim_1d,\n kernel_size=3,\n padding=1,\n groups=4,\n weight_attr=init_params('TEM_s1_w', self.hidden_dim_1d, 3),\n bias_attr=init_params('TEM_s1_b', self.hidden_dim_1d, 3))\n self.ts_conv1_act = paddle.nn.ReLU()\n self.ts_conv2 = paddle.nn.Conv1D(\n in_channels=self.hidden_dim_1d,\n out_channels=1,\n kernel_size=1,\n padding=0,\n groups=1,\n weight_attr=init_params('TEM_s2_w', self.hidden_dim_1d, 1),\n bias_attr=init_params('TEM_s2_b', self.hidden_dim_1d, 1))\n self.ts_conv2_act = paddle.nn.Sigmoid()\n self.te_conv1 = paddle.nn.Conv1D(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:138-163"
+ },
+ "6319": {
+ "file_id": 492,
+ "content": "This code defines a Conv1D block for the BMN model, including an input layer, a temporal evaluation module, and two convolutional layers with ReLU activation functions. The weights and biases are initialized using the 'init_params' function.",
+ "type": "comment"
+ },
+ "6320": {
+ "file_id": 492,
+ "content": " in_channels=self.hidden_dim_1d,\n out_channels=self.hidden_dim_1d,\n kernel_size=3,\n padding=1,\n groups=4,\n weight_attr=init_params('TEM_e1_w', self.hidden_dim_1d, 3),\n bias_attr=init_params('TEM_e1_b', self.hidden_dim_1d, 3))\n self.te_conv1_act = paddle.nn.ReLU()\n self.te_conv2 = paddle.nn.Conv1D(\n in_channels=self.hidden_dim_1d,\n out_channels=1,\n kernel_size=1,\n padding=0,\n groups=1,\n weight_attr=init_params('TEM_e2_w', self.hidden_dim_1d, 1),\n bias_attr=init_params('TEM_e2_b', self.hidden_dim_1d, 1))\n self.te_conv2_act = paddle.nn.Sigmoid()\n #Proposal Evaluation Module\n self.p_conv1 = paddle.nn.Conv1D(\n in_channels=self.hidden_dim_1d,\n out_channels=self.hidden_dim_2d,\n kernel_size=3,\n padding=1,\n groups=1,\n weight_attr=init_params('PEM_1d_w', self.hidden_dim_1d, 3),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:164-189"
+ },
+ "6321": {
+ "file_id": 492,
+ "content": "This code initializes the TEM and PEM modules of a backbone network. It defines several convolutional layers with specific configurations for each module, followed by activation functions. The weight and bias attributes are initialized using the init_params function.",
+ "type": "comment"
+ },
+ "6322": {
+ "file_id": 492,
+ "content": " bias_attr=init_params('PEM_1d_b', self.hidden_dim_1d, 3))\n self.p_conv1_act = paddle.nn.ReLU()\n # init to speed up\n sample_mask = get_interp1d_mask(self.tscale, self.dscale,\n self.prop_boundary_ratio,\n self.num_sample, self.num_sample_perbin)\n self.sample_mask = paddle.to_tensor(sample_mask)\n self.sample_mask.stop_gradient = True\n self.p_conv3d1 = paddle.nn.Conv3D(\n in_channels=128,\n out_channels=self.hidden_dim_3d,\n kernel_size=(self.num_sample, 1, 1),\n stride=(self.num_sample, 1, 1),\n padding=0,\n weight_attr=ParamAttr(name=\"PEM_3d1_w\"),\n bias_attr=ParamAttr(name=\"PEM_3d1_b\"))\n self.p_conv3d1_act = paddle.nn.ReLU()\n self.p_conv2d1 = paddle.nn.Conv2D(\n in_channels=512,\n out_channels=self.hidden_dim_2d,\n kernel_size=1,\n stride=1,\n padding=0,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:190-215"
+ },
+ "6323": {
+ "file_id": 492,
+ "content": "This code initializes a backbone model for the BMN architecture. It includes convolutional layers, ReLU activations, and a tensor mask for sampling. The model uses 1D, 2D, and 3D convolutions with specific parameters, as well as applies bias attributes to the weights and biases of the convolutions.",
+ "type": "comment"
+ },
+ "6324": {
+ "file_id": 492,
+ "content": " weight_attr=ParamAttr(name=\"PEM_2d1_w\"),\n bias_attr=ParamAttr(name=\"PEM_2d1_b\"))\n self.p_conv2d1_act = paddle.nn.ReLU()\n self.p_conv2d2 = paddle.nn.Conv2D(\n in_channels=128,\n out_channels=self.hidden_dim_2d,\n kernel_size=3,\n stride=1,\n padding=1,\n weight_attr=ParamAttr(name=\"PEM_2d2_w\"),\n bias_attr=ParamAttr(name=\"PEM_2d2_b\"))\n self.p_conv2d2_act = paddle.nn.ReLU()\n self.p_conv2d3 = paddle.nn.Conv2D(\n in_channels=128,\n out_channels=self.hidden_dim_2d,\n kernel_size=3,\n stride=1,\n padding=1,\n weight_attr=ParamAttr(name=\"PEM_2d3_w\"),\n bias_attr=ParamAttr(name=\"PEM_2d3_b\"))\n self.p_conv2d3_act = paddle.nn.ReLU()\n self.p_conv2d4 = paddle.nn.Conv2D(\n in_channels=128,\n out_channels=2,\n kernel_size=1,\n stride=1,\n padding=0,\n weight_attr=ParamAttr(name=\"PEM_2d4_w\"),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:216-246"
+ },
+ "6325": {
+ "file_id": 492,
+ "content": "This code initializes a series of 2D convolutional layers with ReLU activation functions for the Batch Multi-Scale Network (BMSN) backbone in PaddleVideo. Each convolutional layer has a specified number of output channels, kernel size, and stride. The weights and biases for each layer are defined using ParamAttr.",
+ "type": "comment"
+ },
+ "6326": {
+ "file_id": 492,
+ "content": " bias_attr=ParamAttr(name=\"PEM_2d4_b\"))\n self.p_conv2d4_act = paddle.nn.Sigmoid()\n def init_weights(self):\n pass\n def forward(self, x):\n #Base Module\n x = self.b_conv1(x)\n x = self.b_conv1_act(x)\n x = self.b_conv2(x)\n x = self.b_conv2_act(x)\n #TEM\n xs = self.ts_conv1(x)\n xs = self.ts_conv1_act(xs)\n xs = self.ts_conv2(xs)\n xs = self.ts_conv2_act(xs)\n xs = paddle.squeeze(xs, axis=[1])\n xe = self.te_conv1(x)\n xe = self.te_conv1_act(xe)\n xe = self.te_conv2(xe)\n xe = self.te_conv2_act(xe)\n xe = paddle.squeeze(xe, axis=[1])\n #PEM\n xp = self.p_conv1(x)\n xp = self.p_conv1_act(xp)\n #BM layer\n xp = paddle.matmul(xp, self.sample_mask)\n xp = paddle.reshape(xp, shape=[0, 0, -1, self.dscale, self.tscale])\n xp = self.p_conv3d1(xp)\n xp = self.p_conv3d1_act(xp)\n xp = paddle.squeeze(xp, axis=[2])\n xp = self.p_conv2d1(xp)\n xp = self.p_conv2d1_act(xp)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:247-283"
+ },
+ "6327": {
+ "file_id": 492,
+ "content": "The code is defining a backbone model for video analysis. It consists of base, TEM (temporal-inspired module), PEM (position-inspired module), and BM (block-matching module) layers. The layers are sequentially applied to the input data with appropriate activation functions and reshaping operations in between. Finally, it performs matrix multiplication with a sample mask and applies additional convolutions and activations.",
+ "type": "comment"
+ },
+ "6328": {
+ "file_id": 492,
+ "content": " xp = self.p_conv2d2(xp)\n xp = self.p_conv2d2_act(xp)\n xp = self.p_conv2d3(xp)\n xp = self.p_conv2d3_act(xp)\n xp = self.p_conv2d4(xp)\n xp = self.p_conv2d4_act(xp)\n return xp, xs, xe",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/bmn.py:284-290"
+ },
+ "6329": {
+ "file_id": 492,
+ "content": "This code is part of a neural network backbone model. It applies multiple convolution layers with activation functions and returns the processed input xp, along with other variables xs and xe.",
+ "type": "comment"
+ },
+ "6330": {
+ "file_id": 493,
+ "content": "/paddlevideo/modeling/backbones/cfbi.py",
+ "type": "filepath"
+ },
+ "6331": {
+ "file_id": 493,
+ "content": "The code imports libraries and defines an FPN class with three layers, creates a backbone model using convolutional layers and GroupNorm. It also defines a \"CFBI\" class that utilizes DeepLab for feature extraction and FPN to combine multi-scale features, returning extracted features at 4x, 8x, 16x scales along with low-level features using a forward function.",
+ "type": "summary"
+ },
+ "6332": {
+ "file_id": 493,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nfrom .deeplab import DeepLab\nclass FPN(nn.Layer):\n \"\"\"FPN Layer\"\"\"\n def __init__(self, in_dim_4x, in_dim_8x, in_dim_16x, out_dim):\n super(FPN, self).__init__()\n self.toplayer = self._make_layer(in_dim_16x, out_dim)\n self.latlayer1 = self._make_layer(in_dim_8x, out_dim)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/cfbi.py:1-28"
+ },
+ "6333": {
+ "file_id": 493,
+ "content": "This code imports necessary libraries and defines a class called FPN, which is an FPN layer in a neural network. It has three layers: toplayer, latlayer1, and latlayer2, each with specific input dimensions and output dimensions. The _make_layer function is used to create these layers.",
+ "type": "comment"
+ },
+ "6334": {
+ "file_id": 493,
+ "content": " self.latlayer2 = self._make_layer(in_dim_4x, out_dim)\n self.smooth1 = self._make_layer(out_dim,\n out_dim,\n kernel_size=3,\n padding=1)\n self.smooth2 = self._make_layer(out_dim,\n out_dim,\n kernel_size=3,\n padding=1)\n def _make_layer(self, in_dim, out_dim, kernel_size=1, padding=0):\n return nn.Sequential(\n nn.Conv2D(in_dim,\n out_dim,\n kernel_size=kernel_size,\n stride=1,\n padding=padding,\n bias_attr=False),\n nn.GroupNorm(num_groups=32, num_channels=out_dim))\n def forward(self, x_4x, x_8x, x_16x):\n \"\"\" forward function\"\"\"\n x_16x = self.toplayer(x_16x)\n x_8x = self.latlayer1(x_8x)\n x_4x = self.latlayer2(x_4x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/cfbi.py:29-54"
+ },
+ "6335": {
+ "file_id": 493,
+ "content": "The code defines a backbone model with two convolutional layers followed by GroupNorm layer. The forward function applies the defined layers to input images of size 4x, 8x, and 16x.",
+ "type": "comment"
+ },
+ "6336": {
+ "file_id": 493,
+ "content": " x_8x = x_8x + F.interpolate(\n x_16x, size=x_8x.shape[-2:], mode='bilinear', align_corners=True)\n x_4x = x_4x + F.interpolate(\n x_8x, size=x_4x.shape[-2:], mode='bilinear', align_corners=True)\n x_8x = self.smooth1(x_8x)\n x_4x = self.smooth2(x_4x)\n return F.relu(x_4x), F.relu(x_8x), F.relu(x_16x)\n@BACKBONES.register()\nclass CFBI(nn.Layer):\n \"\"\"CFBI plus backbone\"\"\"\n def __init__(self,\n backbone='resnet',\n freeze_bn=True,\n model_aspp_outdim=256,\n in_dim_8x=512,\n model_semantic_embedding_dim=256): #,epsilon=1e-05):\n super(CFBI, self).__init__()\n #self.epsilon = epsilon\n self.feature_extracter = DeepLab(backbone=backbone, freeze_bn=freeze_bn)\n self.fpn = FPN(in_dim_4x=model_aspp_outdim,\n in_dim_8x=in_dim_8x,\n in_dim_16x=model_aspp_outdim,\n out_dim=model_semantic_embedding_dim)\n def forward(self, x):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/cfbi.py:56-84"
+ },
+ "6337": {
+ "file_id": 493,
+ "content": "This code defines a class \"CFBI\" which is a backbone model for feature extraction. It utilizes DeepLab as the feature extractor and FPN (Feature Pyramid Network) to combine features from different scales. The input image x is processed through the feature extracter and the output is passed through the fpn to obtain three outputs at 4x, 8x and 16x scales. These outputs are then interpolated and smoothed before being returned after applying ReLU activation.",
+ "type": "comment"
+ },
+ "6338": {
+ "file_id": 493,
+ "content": " \"\"\"forward function\"\"\"\n x, aspp_x, low_level, mid_level = self.feature_extracter(x, True)\n x_4x, x_8x, x_16x = self.fpn(x, mid_level, aspp_x)\n return x_4x, x_8x, x_16x, low_level",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/cfbi.py:85-88"
+ },
+ "6339": {
+ "file_id": 493,
+ "content": "This code defines a forward function that takes an input image and uses the feature_extracter and fpn modules to extract features at different scales. It returns the extracted features at 4x, 8x, and 16x scales along with the low-level features.",
+ "type": "comment"
+ },
+ "6340": {
+ "file_id": 494,
+ "content": "/paddlevideo/modeling/backbones/ctrgcn.py",
+ "type": "filepath"
+ },
+ "6341": {
+ "file_id": 494,
+ "content": "The code presents a CTRGCN backbone for video models, initializes a CTRGC model with batch normalization layers and NTUGraph class, defines a neural network model with TCN_GCN_unit, and includes a final layer 10 (l10) to process input and return output.",
+ "type": "summary"
+ },
+ "6342": {
+ "file_id": 494,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\ndef conv_init(conv):\n if conv.weight is not None:\n weight_init_(conv.weight, 'kaiming_normal_', mode='fan_in')\n if conv.bias is not None:\n nn.initializer.Constant(value=0.0)(conv.bias)\ndef bn_init(bn, scale):\n nn.initializer.Constant(value=float(scale))(bn.weight)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:1-31"
+ },
+ "6343": {
+ "file_id": 494,
+ "content": "This code imports necessary libraries, defines a convolution initialization function and a batch normalization initialization function. It also sets up scale values for the batch normalization function and registers backbone models in the registry.",
+ "type": "comment"
+ },
+ "6344": {
+ "file_id": 494,
+ "content": " nn.initializer.Constant(value=0.0)(bn.bias)\ndef einsum(x1, x3):\n \"\"\"paddle.einsum only support in dynamic graph mode.\n x1 : n c u v\n x2 : n c t v\n \"\"\"\n n, c, u, v1 = x1.shape\n n, c, t, v3 = x3.shape\n assert (v1 == v3), \"Args of einsum not match!\"\n x1 = paddle.transpose(x1, perm=[0, 1, 3, 2]) # n c v u\n y = paddle.matmul(x3, x1)\n # out: n c t u\n return y\nclass CTRGC(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n rel_reduction=8,\n mid_reduction=1):\n super(CTRGC, self).__init__()\n self.in_channels = in_channels\n self.out_channels = out_channels\n if in_channels == 3 or in_channels == 9:\n self.rel_channels = 8\n self.mid_channels = 16\n else:\n self.rel_channels = in_channels // rel_reduction\n self.mid_channels = in_channels // mid_reduction\n self.conv1 = nn.Conv2D(self.in_channels,\n self.rel_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:32-66"
+ },
+ "6345": {
+ "file_id": 494,
+ "content": "Defines a CTRGC class, a type of convolutional neural network layer. It has two reductions: rel_reduction (defaults to 8) and mid_reduction (defaults to 1). Depending on the input channels, it assigns different channel numbers for rel_channels (always 8 if in_channels is 3 or 9; otherwise based on rel_reduction). It also initializes a Conv2D layer with the assigned channel numbers.",
+ "type": "comment"
+ },
+ "6346": {
+ "file_id": 494,
+ "content": " kernel_size=1)\n self.conv2 = nn.Conv2D(self.in_channels,\n self.rel_channels,\n kernel_size=1)\n self.conv3 = nn.Conv2D(self.in_channels,\n self.out_channels,\n kernel_size=1)\n self.conv4 = nn.Conv2D(self.rel_channels,\n self.out_channels,\n kernel_size=1)\n self.tanh = nn.Tanh()\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n conv_init(m)\n elif isinstance(m, nn.BatchNorm2D):\n bn_init(m, 1)\n def forward(self, x, A=None, alpha=1):\n x1, x2, x3 = self.conv1(x).mean(-2), self.conv2(x).mean(-2), self.conv3(\n x)\n x1 = self.tanh(x1.unsqueeze(-1) - x2.unsqueeze(-2))\n x1 = self.conv4(x1) * alpha + (\n A.unsqueeze(0).unsqueeze(0) if A is not None else 0) # N,C,V,V",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:67-93"
+ },
+ "6347": {
+ "file_id": 494,
+ "content": "This code defines a Convolutional Temporal Relational Graph Convolutional Network (CTRGCN) backbone for a video model. It initializes weights and performs forward pass calculations. It uses convolution layers, tanh activation function, and optionally includes an additional input A.",
+ "type": "comment"
+ },
+ "6348": {
+ "file_id": 494,
+ "content": " # We only support 'paddle.einsum()' in dynamic graph mode, if use in infer model please implement self.\n # x1 = paddle.einsum('ncuv,nctv->nctu', x1, x3)\n x1 = einsum(x1, x3)\n return x1\nclass TemporalConv(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n dilation=1):\n super(TemporalConv, self).__init__()\n pad = (kernel_size + (kernel_size - 1) * (dilation - 1) - 1) // 2\n self.conv = nn.Conv2D(in_channels,\n out_channels,\n kernel_size=(kernel_size, 1),\n padding=(pad, 0),\n stride=(stride, 1),\n dilation=(dilation, 1))\n self.bn = nn.BatchNorm2D(out_channels)\n def forward(self, x):\n x = self.conv(x)\n x = self.bn(x)\n return x\nclass MultiScale_TemporalConv(nn.Layer):\n def __init__(self,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:94-127"
+ },
+ "6349": {
+ "file_id": 494,
+ "content": "Code snippet defines a class TemporalConv, which is a 2D convolutional layer for temporal data. It inherits from the paddle.nn.Layer and includes an instance of nn.Conv2D and nn.BatchNorm2D layers. The MultiScale_TemporalConv class is also defined but its implementation is missing, suggesting it extends TemporalConv with multiple temporal convolution blocks for multi-scale processing.",
+ "type": "comment"
+ },
+ "6350": {
+ "file_id": 494,
+ "content": " in_channels,\n out_channels,\n kernel_size=3,\n stride=1,\n dilations=[1, 2, 3, 4],\n residual=True,\n residual_kernel_size=1):\n super(MultiScale_TemporalConv, self).__init__()\n assert out_channels % (\n len(dilations) +\n 2) == 0, '# out channels should be multiples of # branches'\n # Multiple branches of temporal convolution\n self.num_branches = len(dilations) + 2\n branch_channels = out_channels // self.num_branches\n if type(kernel_size) == list:\n assert len(kernel_size) == len(dilations)\n else:\n kernel_size = [kernel_size] * len(dilations)\n # Temporal Convolution branches\n self.branches = nn.LayerList([\n nn.Sequential(\n nn.Conv2D(in_channels,\n branch_channels,\n kernel_size=1,\n padding=0),\n nn.BatchNorm2D(branch_channels),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:128-155"
+ },
+ "6351": {
+ "file_id": 494,
+ "content": "This code defines a MultiScale_TemporalConv layer with multiple branches of temporal convolution. The number of branches is determined by the dilations, and out channels should be multiples of the number of branches for correct operation. Each branch has its own kernel size, and there are Conv2D layers followed by BatchNorm2D for each branch.",
+ "type": "comment"
+ },
+ "6352": {
+ "file_id": 494,
+ "content": " nn.ReLU(),\n TemporalConv(branch_channels,\n branch_channels,\n kernel_size=ks,\n stride=stride,\n dilation=dilation),\n ) for ks, dilation in zip(kernel_size, dilations)\n ])\n # Additional Max & 1x1 branch\n self.branches.append(\n nn.Sequential(\n nn.Conv2D(in_channels,\n branch_channels,\n kernel_size=1,\n padding=0), nn.BatchNorm2D(branch_channels),\n nn.ReLU(),\n nn.MaxPool2D(kernel_size=(3, 1),\n stride=(stride, 1),\n padding=(1, 0)), nn.BatchNorm2D(branch_channels)))\n self.branches.append(\n nn.Sequential(\n nn.Conv2D(in_channels,\n branch_channels,\n kernel_size=1,\n padding=0,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:156-182"
+ },
+ "6353": {
+ "file_id": 494,
+ "content": "This code defines a Conv-Temporal RGN backbone model for video analysis. It consists of multiple branches with various convolutional and pooling layers, including TemporalConv and MaxPool2D operations. The branches are appended to the model and initialized with respective settings such as kernel size, dilation rate, etc.",
+ "type": "comment"
+ },
+ "6354": {
+ "file_id": 494,
+ "content": " stride=(stride, 1)), nn.BatchNorm2D(branch_channels)))\n # Residual connection\n if not residual:\n self.residual = lambda x: 0\n elif (in_channels == out_channels) and (stride == 1):\n self.residual = lambda x: x\n else:\n self.residual = TemporalConv(in_channels,\n out_channels,\n kernel_size=residual_kernel_size,\n stride=stride)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n # initialize\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n conv_init(m)\n elif isinstance(m, nn.BatchNorm2D):\n weight_init_(m.weight, 'Normal', std=0.02, mean=1.0)\n nn.initializer.Constant(value=0.0)(m.bias)\n def forward(self, x):\n # Input dim: (N,C,T,V)\n res = self.residual(x)\n branch_outs = []\n for tempconv in self.branches:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:183-211"
+ },
+ "6355": {
+ "file_id": 494,
+ "content": "This code defines a class for a Conv-Temporal Residual Group Convolutional Network (CTRGCN) backbone. The class contains a constructor that sets up the architecture, an initialization function to set the weights, and a forward pass function for feeding data into the model. It performs residual connections using temporal convolutions and has batch normalization layers.",
+ "type": "comment"
+ },
+ "6356": {
+ "file_id": 494,
+ "content": " out = tempconv(x)\n branch_outs.append(out)\n out = paddle.concat(branch_outs, axis=1)\n out += res\n return out\nclass unit_tcn(nn.Layer):\n def __init__(self, in_channels, out_channels, kernel_size=9, stride=1):\n super(unit_tcn, self).__init__()\n pad = int((kernel_size - 1) / 2)\n self.conv = nn.Conv2D(in_channels,\n out_channels,\n kernel_size=(kernel_size, 1),\n padding=(pad, 0),\n stride=(stride, 1))\n self.bn = nn.BatchNorm2D(out_channels)\n self.relu = nn.ReLU()\n conv_init(self.conv)\n bn_init(self.bn, 1)\n def forward(self, x):\n x = self.bn(self.conv(x))\n return x\nclass unit_gcn(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n A,\n coff_embedding=4,\n adaptive=True,\n residual=True):\n super(unit_gcn, self).__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:212-250"
+ },
+ "6357": {
+ "file_id": 494,
+ "content": "This code defines two classes: \"unit_tcn\" and \"unit_gcn\". The \"unit_tcn\" class is a Temporal Convolutional Network unit that performs temporal convolution with batch normalization and ReLU activation. The \"unit_gcn\" class is a Graph Convolutional Network unit that takes input channels, output channels, adjacency matrix A, coefficient embedding, adaptive flag, and residual flag as parameters.",
+ "type": "comment"
+ },
+ "6358": {
+ "file_id": 494,
+ "content": " inter_channels = out_channels // coff_embedding\n self.inter_c = inter_channels\n self.out_c = out_channels\n self.in_c = in_channels\n self.adaptive = adaptive\n self.num_subset = A.shape[0]\n self.convs = nn.LayerList()\n for i in range(self.num_subset):\n self.convs.append(CTRGC(in_channels, out_channels))\n if residual:\n if in_channels != out_channels:\n self.down = nn.Sequential(\n nn.Conv2D(in_channels, out_channels, 1),\n nn.BatchNorm2D(out_channels))\n else:\n self.down = lambda x: x\n else:\n self.down = lambda x: 0\n if self.adaptive:\n pa_param = paddle.ParamAttr(\n initializer=paddle.nn.initializer.Assign(A.astype(np.float32)))\n self.PA = paddle.create_parameter(shape=A.shape,\n dtype='float32',\n attr=pa_param)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:251-276"
+ },
+ "6359": {
+ "file_id": 494,
+ "content": "This code initializes a CTRGC model with specified input and output channels. It also includes optional residual connection, batch normalization, and adaptive parameterization. The number of subsets is determined by the shape of A. If adaptive is set to True, it creates a trainable parameter for the subset of weights.",
+ "type": "comment"
+ },
+ "6360": {
+ "file_id": 494,
+ "content": " else:\n A_tensor = paddle.to_tensor(A, dtype=\"float32\")\n self.A = paddle.create_parameter(\n shape=A_tensor.shape,\n dtype='float32',\n default_initializer=paddle.nn.initializer.Assign(A_tensor))\n self.A.stop_gradient = True\n alpha_tensor = paddle.to_tensor(np.zeros(1), dtype=\"float32\")\n self.alpha = paddle.create_parameter(\n shape=alpha_tensor.shape,\n dtype='float32',\n default_initializer=paddle.nn.initializer.Assign(alpha_tensor))\n self.bn = nn.BatchNorm2D(out_channels)\n self.soft = nn.Softmax(-2)\n self.relu = nn.ReLU()\n def init_weights(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n conv_init(m)\n elif isinstance(m, nn.BatchNorm2D):\n bn_init(m, 1)\n bn_init(self.bn, 1e-6)\n def forward(self, x):\n y = None\n if self.adaptive:\n A = self.PA\n else:\n A = self.A.cuda(x.get_device())",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:277-306"
+ },
+ "6361": {
+ "file_id": 494,
+ "content": "This code initializes the parameters A and alpha, sets up batch normalization (bn) layers with Softmax and ReLU activation functions, initializes weights using conv_init and bn_init functions, and defines a forward pass that adapts A based on the adaptive flag.",
+ "type": "comment"
+ },
+ "6362": {
+ "file_id": 494,
+ "content": " for i in range(self.num_subset):\n z = self.convs[i](x, A[i], self.alpha)\n y = z + y if y is not None else z\n y = self.bn(y)\n y += self.down(x)\n y = self.relu(y)\n return y\nclass TCN_GCN_unit(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n A,\n stride=1,\n residual=True,\n adaptive=True,\n kernel_size=5,\n dilations=[1, 2]):\n super(TCN_GCN_unit, self).__init__()\n self.gcn1 = unit_gcn(in_channels, out_channels, A, adaptive=adaptive)\n self.tcn1 = MultiScale_TemporalConv(out_channels,\n out_channels,\n kernel_size=kernel_size,\n stride=stride,\n dilations=dilations,\n residual=False)\n self.relu = nn.ReLU()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:307-335"
+ },
+ "6363": {
+ "file_id": 494,
+ "content": "This code defines a TCN_GCN_unit class, which is a combination of Graph Convolutional Network (GCN) and Temporal Convolution units. The unit takes input channels, output channels, adjacency matrix A, stride, residual connection, adaptive flag, kernel size, and dilations as parameters. It initializes the GCN and TemporalConv layers, followed by a ReLU activation function.",
+ "type": "comment"
+ },
+ "6364": {
+ "file_id": 494,
+ "content": " if not residual:\n self.residual = lambda x: 0\n elif (in_channels == out_channels) and (stride == 1):\n self.residual = lambda x: x\n else:\n self.residual = unit_tcn(in_channels,\n out_channels,\n kernel_size=1,\n stride=stride)\n def forward(self, x):\n y = self.relu(self.tcn1(self.gcn1(x)) + self.residual(x))\n return y\nclass NTUDGraph:\n def __init__(self, labeling_mode='spatial'):\n num_node = 25\n self_link = [(i, i) for i in range(num_node)]\n inward_ori_index = [(1, 2), (2, 21), (3, 21), (4, 3), (5, 21), (6, 5),\n (7, 6), (8, 7), (9, 21), (10, 9), (11, 10),\n (12, 11), (13, 1), (14, 13), (15, 14), (16, 15),\n (17, 1), (18, 17), (19, 18), (20, 19), (22, 23),\n (23, 8), (24, 25), (25, 12)]\n inward = [(i - 1, j - 1) for (i, j) in inward_ori_index]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:336-363"
+ },
+ "6365": {
+ "file_id": 494,
+ "content": "The code defines a `CTRGCN` class with a `forward` method and an `NTUDGraph` class. The `forward` method takes input `x`, applies `relu` activation and adds the residual output of a `unit_tcn` layer or simply passes through if specified conditions are met. The `NTUDGraph` initializes with a fixed number of nodes, self-links, and inward connections.",
+ "type": "comment"
+ },
+ "6366": {
+ "file_id": 494,
+ "content": " outward = [(j, i) for (i, j) in inward]\n neighbor = inward + outward\n self.num_node = num_node\n self.self_link = self_link\n self.inward = inward\n self.outward = outward\n self.neighbor = neighbor\n self.A = self.get_adjacency_matrix(labeling_mode)\n def edge2mat(self, link, num_node):\n A = np.zeros((num_node, num_node))\n for i, j in link:\n A[j, i] = 1\n return A\n def normalize_digraph(self, A):\n Dl = np.sum(A, 0)\n h, w = A.shape\n Dn = np.zeros((w, w))\n for i in range(w):\n if Dl[i] > 0:\n Dn[i, i] = Dl[i]**(-1)\n AD = np.dot(A, Dn)\n return AD\n def get_spatial_graph(self, num_node, self_link, inward, outward):\n I = self.edge2mat(self_link, num_node)\n In = self.normalize_digraph(self.edge2mat(inward, num_node))\n Out = self.normalize_digraph(self.edge2mat(outward, num_node))\n A = np.stack((I, In, Out))\n return A\n def get_adjacency_matrix(self, labeling_mode=None):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:364-397"
+ },
+ "6367": {
+ "file_id": 494,
+ "content": "Function `get_adjacency_matrix` generates adjacency matrices for the model. The function takes a parameter `labeling_mode`, which is optional. It initializes a set of variables: `inward`, `outward`, and `neighbor`. These variables store the connections between nodes in both directions. Then, it calls other helper functions to generate normalized adjacency matrices for self-links, inward edges, outward edges, and finally returns an array containing all these matrices. This is useful for inputting into a model that requires specific formatted input data.",
+ "type": "comment"
+ },
+ "6368": {
+ "file_id": 494,
+ "content": " if labeling_mode is None:\n return self.A\n if labeling_mode == 'spatial':\n A = self.get_spatial_graph(self.num_node, self.self_link,\n self.inward, self.outward)\n else:\n raise ValueError()\n return A\n@BACKBONES.register()\nclass CTRGCN(nn.Layer):\n \"\"\"\n CTR-GCN model from:\n `\"Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition\" `_\n Args:\n num_point: int, numbers of sketeton point.\n num_person: int, numbers of person.\n base_channel: int, model's hidden dim.\n graph: str, sketeton adjacency matrix name.\n graph_args: dict, sketeton adjacency graph class args.\n in_channels: int, channels of vertex coordinate. 2 for (x,y), 3 for (x,y,z). Default 3.\n adaptive: bool, if adjacency matrix can adaptive.\n \"\"\"\n def __init__(self,\n num_point=25,\n num_person=2,\n base_channel=64,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:398-426"
+ },
+ "6369": {
+ "file_id": 494,
+ "content": "This code is part of the CTRGCN class in the PaddleVideo library, which represents a specific type of model for skeleton-based action recognition. The function within this code block is used to return an adjacency matrix (A) based on a given labeling mode. If no labeling mode is specified, it returns the adjacency matrix from the instance variables. If the labeling mode is set to 'spatial', it calls another function to generate a spatial adjacency graph. Otherwise, if an invalid labeling mode is provided, it raises a ValueError exception.",
+ "type": "comment"
+ },
+ "6370": {
+ "file_id": 494,
+ "content": " graph='ntu_rgb_d',\n graph_args=dict(),\n in_channels=3,\n adaptive=True):\n super(CTRGCN, self).__init__()\n if graph == 'ntu_rgb_d':\n self.graph = NTUDGraph(**graph_args)\n else:\n raise ValueError()\n A = self.graph.A # 3,25,25\n self.num_point = num_point\n self.data_bn = nn.BatchNorm1D(num_person * in_channels * num_point)\n self.base_channel = base_channel\n self.l1 = TCN_GCN_unit(in_channels,\n self.base_channel,\n A,\n residual=False,\n adaptive=adaptive)\n self.l2 = TCN_GCN_unit(self.base_channel,\n self.base_channel,\n A,\n adaptive=adaptive)\n self.l3 = TCN_GCN_unit(self.base_channel,\n self.base_channel,\n A,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:427-455"
+ },
+ "6371": {
+ "file_id": 494,
+ "content": "This code defines the CTRGCN class, which initializes its graph and layers based on input parameters. It includes a batch normalization layer (data_bn) and three TCN_GCN_unit layers (l1, l2, l3). The graph is determined by the 'graph' parameter, with NTUDGraph used if 'ntu_rgb_d'. If another graph is provided, it raises a ValueError.",
+ "type": "comment"
+ },
+ "6372": {
+ "file_id": 494,
+ "content": " adaptive=adaptive)\n self.l4 = TCN_GCN_unit(self.base_channel,\n self.base_channel,\n A,\n adaptive=adaptive)\n self.l5 = TCN_GCN_unit(self.base_channel,\n self.base_channel * 2,\n A,\n stride=2,\n adaptive=adaptive)\n self.l6 = TCN_GCN_unit(self.base_channel * 2,\n self.base_channel * 2,\n A,\n adaptive=adaptive)\n self.l7 = TCN_GCN_unit(self.base_channel * 2,\n self.base_channel * 2,\n A,\n adaptive=adaptive)\n self.l8 = TCN_GCN_unit(self.base_channel * 2,\n self.base_channel * 4,\n A,\n stride=2,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:456-477"
+ },
+ "6373": {
+ "file_id": 494,
+ "content": "The code initializes six TCN_GCN_unit layers, each with different configurations, for a CTRGCN model. The first layer (l4) has the base channel as input and output. Following layers (l5 to l8) increase the number of channels or apply strides. This represents a deep TCN-GCN architecture with progressively increasing depth and downsampling.",
+ "type": "comment"
+ },
+ "6374": {
+ "file_id": 494,
+ "content": " adaptive=adaptive)\n self.l9 = TCN_GCN_unit(self.base_channel * 4,\n self.base_channel * 4,\n A,\n adaptive=adaptive)\n self.l10 = TCN_GCN_unit(self.base_channel * 4,\n self.base_channel * 4,\n A,\n adaptive=adaptive)\n def init_weights(self):\n bn_init(self.data_bn, 1)\n def forward(self, x):\n N, C, T, V, M = x.shape\n x = paddle.transpose(x, perm=[0, 4, 3, 1, 2])\n x = paddle.reshape(x, (N, M * V * C, T))\n x = self.data_bn(x)\n x = paddle.reshape(x, (N, M, V, C, T))\n x = paddle.transpose(x, perm=(0, 1, 3, 4, 2))\n x = paddle.reshape(x, (N * M, C, T, V))\n x = self.l1(x)\n x = self.l2(x)\n x = self.l3(x)\n x = self.l4(x)\n x = self.l5(x)\n x = self.l6(x)\n x = self.l7(x)\n x = self.l8(x)\n x = self.l9(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:478-511"
+ },
+ "6375": {
+ "file_id": 494,
+ "content": "This code defines a neural network model with multiple layers. It uses Paddle's TCN_GCN_unit in the last two layers. The init_weights function initializes batch normalization for the data_bn layer, and the forward function processes input through multiple layers before returning the final output.",
+ "type": "comment"
+ },
+ "6376": {
+ "file_id": 494,
+ "content": " x = self.l10(x)\n return x, N, M",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ctrgcn.py:512-514"
+ },
+ "6377": {
+ "file_id": 494,
+ "content": "This code represents the final step of a neural network function. It applies layer 10 (l10) to input x, and returns both the updated x and the original N, M values. This function seems to be part of a larger model, as it references previous layers.",
+ "type": "comment"
+ },
+ "6378": {
+ "file_id": 495,
+ "content": "/paddlevideo/modeling/backbones/darknet.py",
+ "type": "filepath"
+ },
+ "6379": {
+ "file_id": 495,
+ "content": "This code defines a ConvBNLayer class and Darknet backbone, performing convolutions, pooling, and reorganization in a neural network. It concatenates results from two branches, applies more convolutions, and returns final output.",
+ "type": "summary"
+ },
+ "6380": {
+ "file_id": 495,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle import ParamAttr\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n input_channels,\n output_channels,\n filter_size,\n stride,\n padding,\n name=None):\n super(ConvBNLayer, self).__init__()\n self._conv = nn.Conv2D(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:1-32"
+ },
+ "6381": {
+ "file_id": 495,
+ "content": "This code defines a ConvBNLayer class that inherits from nn.Layer and includes a Conv2D layer, Batch Normalization, and other parameters like input_channels, output_channels, filter_size, stride, padding, and name.",
+ "type": "comment"
+ },
+ "6382": {
+ "file_id": 495,
+ "content": " in_channels=input_channels,\n out_channels=output_channels,\n kernel_size=filter_size,\n stride=stride,\n padding=padding,\n weight_attr=ParamAttr(name=name + \".conv.weights\"),\n bias_attr=False)\n bn_name = name + \".bn\"\n self._bn = nn.BatchNorm(\n num_channels=output_channels,\n act=\"leaky_relu\",\n param_attr=ParamAttr(name=bn_name + \".scale\"),\n bias_attr=ParamAttr(name=bn_name + \".offset\"),\n moving_mean_name=bn_name + \".mean\",\n moving_variance_name=bn_name + \".var\")\n def forward(self, inputs):\n x = self._conv(inputs)\n x = self._bn(x)\n return x\nclass BasicBlock(nn.Layer):\n def __init__(self, input_channels, output_channels, name=None):\n super(BasicBlock, self).__init__()\n self._conv1 = ConvBNLayer(input_channels=input_channels, output_channels=output_channels, filter_size=[\n 3, 3], stride=1, padding=1, name=name+'.0')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:33-61"
+ },
+ "6383": {
+ "file_id": 495,
+ "content": "This code defines a convolutional neural network block with batch normalization and leaky ReLU activation. The forward function applies the convolution followed by batch normalization, and BasicBlock is a subclass of nn.Layer representing a single block in the model architecture.",
+ "type": "comment"
+ },
+ "6384": {
+ "file_id": 495,
+ "content": " self._max_pool = nn.MaxPool2D(kernel_size=2, stride=2, padding=0)\n self._conv2 = ConvBNLayer(input_channels=output_channels, output_channels=output_channels *\n 2, filter_size=[3, 3], stride=1, padding=1, name=name+'.1')\n self._conv3 = ConvBNLayer(input_channels=output_channels*2, output_channels=output_channels,\n filter_size=[1, 1], stride=1, padding=0, name=name+'.2')\n def forward(self, x):\n x = self._conv1(x)\n x = self._max_pool(x)\n x = self._conv2(x)\n x = self._conv3(x)\n return x\nclass Reorg(nn.Layer):\n def __init__(self, stride=2):\n super(Reorg, self).__init__()\n self.stride = stride\n def forward(self, x):\n stride = self.stride\n assert (x.dim() == 4)\n B = x.shape[0]\n C = x.shape[1]\n H = x.shape[2]\n W = x.shape[3]\n assert (H % stride == 0)\n assert (W % stride == 0)\n ws = stride\n hs = stride\n x = x.reshape([B, C, H // hs, hs, W // ws, ws]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:62-92"
+ },
+ "6385": {
+ "file_id": 495,
+ "content": "Code defines a Darknet backbone with ConvBNLayer and MaxPooling layers, followed by Reorg layer for spatial downsampling.",
+ "type": "comment"
+ },
+ "6386": {
+ "file_id": 495,
+ "content": " ).transpose([0, 1, 2, 4, 3, 5])\n x = x.reshape([B, C, H // hs * W // ws, hs * ws]\n ).transpose([0, 1, 3, 2])\n x = x.reshape([B, C, hs * ws, H // hs, W // ws]\n ).transpose([0, 2, 1, 3, 4])\n x = x.reshape([B, hs * ws * C, H // hs, W // ws])\n return x\nclass Darknet(nn.Layer):\n def __init__(self, pretrained=None):\n super(Darknet, self).__init__()\n self.pretrained = pretrained\n self._conv1 = ConvBNLayer(\n input_channels=3, output_channels=32, filter_size=3, stride=1, padding=1, name='input')\n self._max_pool1 = nn.MaxPool2D(kernel_size=2, stride=2, padding=0)\n self._basic_block_11 = BasicBlock(\n input_channels=32, output_channels=64, name='1.1')\n self._basic_block_12 = BasicBlock(\n input_channels=64, output_channels=128, name='1.2')\n self._basic_block_13 = BasicBlock(\n input_channels=128, output_channels=256, name='1.3')\n self._conv2 = ConvBNLayer(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:93-115"
+ },
+ "6387": {
+ "file_id": 495,
+ "content": "This code reshapes the input tensor and performs a sequence of transpose operations to rearrange dimensions. The code is part of a Darknet class, which inherits from nn.Layer and contains various ConvBNLayer and BasicBlock instances for building a convolutional neural network.",
+ "type": "comment"
+ },
+ "6388": {
+ "file_id": 495,
+ "content": " input_channels=256, output_channels=512, filter_size=3, stride=1, padding=1, name='up1')\n self._conv3 = ConvBNLayer(\n input_channels=512, output_channels=256, filter_size=1, stride=1, padding=0, name='down1')\n self._conv4 = ConvBNLayer(\n input_channels=256, output_channels=512, filter_size=3, stride=1, padding=1, name='2.1')\n self._max_pool2 = nn.MaxPool2D(kernel_size=2, stride=2, padding=0)\n self._conv5 = ConvBNLayer(\n input_channels=512, output_channels=1024, filter_size=3, stride=1, padding=1, name='2.2')\n self._conv6 = ConvBNLayer(input_channels=1024, output_channels=512,\n filter_size=1, stride=1, padding=0, name='2.3') # ori\n self._conv7 = ConvBNLayer(\n input_channels=512, output_channels=1024, filter_size=3, stride=1, padding=1, name='up2')\n self._conv8 = ConvBNLayer(input_channels=1024, output_channels=512,\n filter_size=1, stride=1, padding=0, name='down2')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:116-129"
+ },
+ "6389": {
+ "file_id": 495,
+ "content": "The code defines a series of ConvBNLayer objects for the Darknet backbone. These layers include upsampling, downsampling, and convolution operations with different filter sizes and strides. The ConvBNLayer class is used to perform convolutions followed by batch normalization.",
+ "type": "comment"
+ },
+ "6390": {
+ "file_id": 495,
+ "content": " self._conv9 = ConvBNLayer(\n input_channels=512, output_channels=1024, filter_size=3, stride=1, padding=1, name='3.1')\n self._conv10 = ConvBNLayer(\n input_channels=1024, output_channels=1024, filter_size=3, stride=1, padding=1, name='3.2')\n self._conv11 = ConvBNLayer(\n input_channels=1024, output_channels=1024, filter_size=3, stride=1, padding=1, name='3.3')\n self._conv12 = ConvBNLayer(\n input_channels=512, output_channels=64, filter_size=1, stride=1, padding=0, name='4.1')\n self._reorg = Reorg()\n self._conv13 = ConvBNLayer(\n input_channels=1280, output_channels=1024, filter_size=3, stride=1, padding=1, name='5.1')\n self._conv14 = nn.Conv2D(1024, 425, kernel_size=1)\n def forward(self, inputs):\n x = self._conv1(inputs)\n x = self._max_pool1(x)\n x = self._basic_block_11(x)\n x = self._basic_block_12(x)\n x = self._basic_block_13(x)\n x = self._conv2(x)\n x = self._conv3(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:130-150"
+ },
+ "6391": {
+ "file_id": 495,
+ "content": "This code defines a neural network backbone with multiple convolutional layers, batch normalization, and pooling operations. The forward method implements the network's processing flow for input images.",
+ "type": "comment"
+ },
+ "6392": {
+ "file_id": 495,
+ "content": " ori = self._conv4(x)\n x = self._max_pool2(ori)\n x = self._conv5(x)\n x = self._conv6(x)\n x = self._conv7(x)\n x = self._conv8(x)\n x = self._conv9(x)\n x = self._conv10(x)\n x1 = self._conv11(x)\n x2 = self._conv12(ori)\n x2 = self._reorg(x2)\n x = paddle.concat([x2, x1], 1)\n x = self._conv13(x)\n x = self._conv14(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/darknet.py:151-165"
+ },
+ "6393": {
+ "file_id": 495,
+ "content": "The code performs multiple convolutional operations, followed by pooling and reorganization. It concatenates the results of two separate branches, then applies further convolutions before returning the final output.",
+ "type": "comment"
+ },
+ "6394": {
+ "file_id": 496,
+ "content": "/paddlevideo/modeling/backbones/deeplab.py",
+ "type": "filepath"
+ },
+ "6395": {
+ "file_id": 496,
+ "content": "This code constructs a PaddlePaddle DeepLab network with convolution layers, batch normalization and activation functions in Bottleneck and ResNet classes. It includes additional layers for better performance, initializes ASPP modules in the DeepLab model for feature extraction, defines a segmentation model with ResNet backbone, adaptive pooling, and Decoder modules, and performs inference using forward function.",
+ "type": "summary"
+ },
+ "6396": {
+ "file_id": 496,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport copy\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nclass FrozenBatchNorm2D(nn.Layer):\n \"\"\"\n BatchNorm2D where the batch statistics and the affine parameters\n are fixed\n \"\"\"\n def __init__(self, n, epsilon=1e-5):\n super(FrozenBatchNorm2D, self).__init__()\n x1 = paddle.ones([n])\n x2 = paddle.zeros([n])\n weight = self.create_parameter(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:1-33"
+ },
+ "6397": {
+ "file_id": 496,
+ "content": "This code defines a class `FrozenBatchNorm2D` which is a type of batch normalization layer where the batch statistics and affine parameters are fixed. It inherits from `nn.Layer` and initializes `paddle.ones` and `paddle.zeros` tensors as its parameters, representing fixed batch statistics and affine transformation.",
+ "type": "comment"
+ },
+ "6398": {
+ "file_id": 496,
+ "content": " shape=x1.shape, default_initializer=nn.initializer.Assign(x1))\n bias = self.create_parameter(\n shape=x2.shape, default_initializer=nn.initializer.Assign(x2))\n running_mean = self.create_parameter(\n shape=x2.shape, default_initializer=nn.initializer.Assign(x2))\n running_var = self.create_parameter(\n shape=x1.shape, default_initializer=nn.initializer.Assign(x1))\n self.add_parameter('weight', weight)\n self.add_parameter('bias', bias)\n self.add_parameter('running_mean', running_mean)\n self.add_parameter('running_var', running_var)\n self.epsilon = epsilon\n def forward(self, x):\n scale = self.weight * paddle.rsqrt((self.running_var + self.epsilon))\n bias = self.bias - self.running_mean * scale\n scale = paddle.reshape(scale, [1, -1, 1, 1])\n bias = paddle.reshape(bias, [1, -1, 1, 1])\n return x * scale + bias\nclass Bottleneck(nn.Layer):\n expansion = 4\n def __init__(self,\n inplanes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:34-59"
+ },
+ "6399": {
+ "file_id": 496,
+ "content": "The code defines a DeepLab class, initializes its parameters, and creates a Bottleneck layer. The DeepLab class contains a weight parameter for the convolution operation, a bias parameter to adjust output, and running_mean and running_var parameters used in normalization. The Bottleneck layer has an expansion factor of 4, implying it will increase the number of channels by this factor. This code is part of a neural network backbone implementation using PaddlePaddle framework.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/64.json b/docs/data/64.json
new file mode 100644
index 000000000..1ad6633ba
--- /dev/null
+++ b/docs/data/64.json
@@ -0,0 +1,549 @@
+{
+ "6400": {
+ "file_id": 496,
+ "content": " planes,\n stride=1,\n dilation=1,\n downsample=None,\n BatchNorm=None):\n super(Bottleneck, self).__init__()\n self.conv1 = nn.Conv2D(inplanes, planes, kernel_size=1, bias_attr=False)\n self.bn1 = BatchNorm(planes)\n self.conv2 = nn.Conv2D(planes,\n planes,\n kernel_size=3,\n stride=stride,\n dilation=dilation,\n padding=dilation,\n bias_attr=False)\n self.bn2 = BatchNorm(planes)\n self.conv3 = nn.Conv2D(planes,\n planes * 4,\n kernel_size=1,\n bias_attr=False)\n self.bn3 = BatchNorm(planes * 4)\n self.relu = nn.ReLU()\n self.downsample = downsample\n self.stride = stride\n self.dilation = dilation\n def forward(self, x):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:60-86"
+ },
+ "6401": {
+ "file_id": 496,
+ "content": "Bottleneck class is a convolution neural network layer with batch normalization, designed for DeepLab model. It consists of 3 consecutive convolutions with varying kernel sizes and stride. BatchNorm layers are used after each convolution to normalize the activations, followed by ReLU activation function. The output channels are scaled by 4 in the final convolution.",
+ "type": "comment"
+ },
+ "6402": {
+ "file_id": 496,
+ "content": " residual = x\n out = self.conv1(x)\n out = self.bn1(out)\n out = self.relu(out)\n out = self.conv2(out)\n out = self.bn2(out)\n out = self.relu(out)\n out = self.conv3(out)\n out = self.bn3(out)\n if self.downsample is not None:\n residual = self.downsample(x)\n out += residual\n out = self.relu(out)\n return out\nclass ResNet(nn.Layer):\n def __init__(self,\n block,\n layers,\n output_stride,\n BatchNorm,\n pretrained=False):\n self.inplanes = 64\n super(ResNet, self).__init__()\n blocks = [1, 2, 4]\n if output_stride == 16:\n strides = [1, 2, 2, 1]\n dilations = [1, 1, 1, 2]\n elif output_stride == 8:\n strides = [1, 2, 1, 1]\n dilations = [1, 1, 2, 4]\n else:\n raise NotImplementedError\n # Modules\n self.conv1 = nn.Conv2D(3,\n 64,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:87-130"
+ },
+ "6403": {
+ "file_id": 496,
+ "content": "Code snippet performs residual block operations using convolutional layers and batch normalization with ReLU activation. It also includes downsampling if specified, and returns the output after applying final ReLU. The class ResNet initializes a ResNet network with given number of blocks, output stride, BatchNorm type, and pretrained option.",
+ "type": "comment"
+ },
+ "6404": {
+ "file_id": 496,
+ "content": " kernel_size=7,\n stride=2,\n padding=3,\n bias_attr=False)\n self.bn1 = BatchNorm(64)\n self.relu = nn.ReLU()\n self.maxpool = nn.MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.layer1 = self._make_layer(block,\n 64,\n layers[0],\n stride=strides[0],\n dilation=dilations[0],\n BatchNorm=BatchNorm)\n self.layer2 = self._make_layer(block,\n 128,\n layers[1],\n stride=strides[1],\n dilation=dilations[1],\n BatchNorm=BatchNorm)\n self.layer3 = self._make_layer(block,\n 256,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:131-152"
+ },
+ "6405": {
+ "file_id": 496,
+ "content": "This code is defining a deep learning model with convolutional layers, batch normalization, and activation functions. It uses the DeepLab backbone architecture and specifies parameters such as kernel sizes, strides, padding, and dilation rates for each layer. The BatchNorm parameter allows for optional batch normalization between layers, improving model performance by reducing internal covariate shift.",
+ "type": "comment"
+ },
+ "6406": {
+ "file_id": 496,
+ "content": " layers[2],\n stride=strides[2],\n dilation=dilations[2],\n BatchNorm=BatchNorm)\n self.layer4 = self._make_MG_unit(block,\n 512,\n blocks=blocks,\n stride=strides[3],\n dilation=dilations[3],\n BatchNorm=BatchNorm)\n self._init_weight()\n def _make_layer(self,\n block,\n planes,\n blocks,\n stride=1,\n dilation=1,\n BatchNorm=None):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n downsample = nn.Sequential(\n nn.Conv2D(self.inplanes,\n planes * block.expansion,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:153-176"
+ },
+ "6407": {
+ "file_id": 496,
+ "content": "This code defines a class with two layers, layer3 and layer4. Layer3 is created using the _make_MG_unit function with specific parameters like block, planes, blocks, stride, dilation, and BatchNorm. Layer4 is also created by calling _make_layer function. Downsampling is done if stride is not 1 or inplanes are not equal to planes*block.expansion.",
+ "type": "comment"
+ },
+ "6408": {
+ "file_id": 496,
+ "content": " kernel_size=1,\n stride=stride,\n bias_attr=False),\n BatchNorm(planes * block.expansion),\n )\n layers = []\n layers.append(\n block(self.inplanes, planes, stride, dilation, downsample,\n BatchNorm))\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(\n block(self.inplanes,\n planes,\n dilation=dilation,\n BatchNorm=BatchNorm))\n return nn.Sequential(*layers)\n def _make_MG_unit(self,\n block,\n planes,\n blocks,\n stride=1,\n dilation=1,\n BatchNorm=None):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n downsample = nn.Sequential(\n nn.Conv2D(self.inplanes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:177-207"
+ },
+ "6409": {
+ "file_id": 496,
+ "content": "This code defines a function _make_MG_unit that creates a module for the DeepLab model, which includes multiple layers of a specified block. It takes in parameters such as block, planes, blocks, stride, dilation, and BatchNorm (optional). The function first checks if downsampling is needed based on stride and inplanes. If so, it creates a Conv2D layer for downsampling. Then, it appends the initial layer with the specified parameters and expands the number of layers as required. Finally, it returns the created sequence of layers as a nn.Sequential module.",
+ "type": "comment"
+ },
+ "6410": {
+ "file_id": 496,
+ "content": " planes * block.expansion,\n kernel_size=1,\n stride=stride,\n bias_attr=False),\n BatchNorm(planes * block.expansion),\n )\n layers = []\n layers.append(\n block(self.inplanes,\n planes,\n stride,\n dilation=blocks[0] * dilation,\n downsample=downsample,\n BatchNorm=BatchNorm))\n self.inplanes = planes * block.expansion\n for i in range(1, len(blocks)):\n layers.append(\n block(self.inplanes,\n planes,\n stride=1,\n dilation=blocks[i] * dilation,\n BatchNorm=BatchNorm))\n return nn.Sequential(*layers)\n def forward(self, input, return_mid_level=False):\n x = self.conv1(input)\n x = self.bn1(x)\n x = self.relu(x)\n x = self.maxpool(x)\n x = self.layer1(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:208-240"
+ },
+ "6411": {
+ "file_id": 496,
+ "content": "This code defines a function that creates a convolutional neural network for the DeepLab model. It takes input, creates layers with specified parameters, and returns a Sequential object representing the model.",
+ "type": "comment"
+ },
+ "6412": {
+ "file_id": 496,
+ "content": " low_level_feat = x\n x = self.layer2(x)\n mid_level_feat = x\n x = self.layer3(x)\n x = self.layer4(x)\n if return_mid_level:\n return x, low_level_feat, mid_level_feat\n else:\n return x, low_level_feat\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass _ASPPModule(nn.Layer):\n def __init__(self, inplanes, planes, kernel_size, padding, dilation,\n BatchNorm):\n super(_ASPPModule, self).__init__()\n self.atrous_conv = nn.Conv2D(inplanes,\n planes,\n kernel_size=kernel_size,\n stride=1,\n padding=padding,\n dilation=dilation,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:241-269"
+ },
+ "6413": {
+ "file_id": 496,
+ "content": "This code defines a DeepLab model that utilizes an ASPP module. It extracts low and mid-level features from the input, has multiple layers of convolutions, and initializes weights using specific initializers. The ASPP module applies atrous convolutions with different dilation rates for feature extraction.",
+ "type": "comment"
+ },
+ "6414": {
+ "file_id": 496,
+ "content": " bias_attr=False)\n self.bn = BatchNorm(planes)\n self.relu = nn.ReLU()\n self._init_weight()\n def forward(self, x):\n x = self.atrous_conv(x)\n x = self.bn(x)\n return self.relu(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n m.weight_attr = nn.initializer.KaimingNormal()\n elif isinstance(m, nn.BatchNorm2D):\n m.weight.data.fill_(1)\n m.bias.data.zero_()\nclass ASPP(nn.Layer):\n def __init__(self, backbone, output_stride, BatchNorm):\n super(ASPP, self).__init__()\n if backbone == 'drn':\n inplanes = 512\n elif backbone == 'mobilenet':\n inplanes = 320\n else:\n inplanes = 2048\n if output_stride == 16:\n dilations = [1, 6, 12, 18]\n elif output_stride == 8:\n dilations = [1, 12, 24, 36]\n else:\n raise NotImplementedError\n self.aspp1 = _ASPPModule(inplanes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:270-307"
+ },
+ "6415": {
+ "file_id": 496,
+ "content": "The code defines a DeepLab class with an ASPP module for feature extraction. It initializes the layers and sets their weights using Kaiming normal initialization or fills BatchNorm2D weight with 1 and biases with 0. The ASPP class accepts backbone and output_stride as parameters to determine dilations for the ASPP modules.",
+ "type": "comment"
+ },
+ "6416": {
+ "file_id": 496,
+ "content": " 256,\n 1,\n padding=0,\n dilation=dilations[0],\n BatchNorm=BatchNorm)\n self.aspp2 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[1],\n dilation=dilations[1],\n BatchNorm=BatchNorm)\n self.aspp3 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[2],\n dilation=dilations[2],\n BatchNorm=BatchNorm)\n self.aspp4 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[3],\n dilation=dilations[3],\n BatchNorm=BatchNorm)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:308-330"
+ },
+ "6417": {
+ "file_id": 496,
+ "content": "This code initializes four instances of the _ASPPModule class, each with different dilation rates and padding values for the DeepLab model's ASPP feature extraction module. The inplanes parameter is consistent across all four modules, indicating the number of input feature planes. BatchNorm specifies whether to apply batch normalization or not.",
+ "type": "comment"
+ },
+ "6418": {
+ "file_id": 496,
+ "content": " self.global_avg_pool = nn.Sequential(\n nn.AdaptiveAvgPool2D((1, 1)),\n nn.Conv2D(inplanes, 256, 1, stride=1, bias_attr=False),\n BatchNorm(256), nn.ReLU())\n self.conv1 = nn.Conv2D(1280, 256, 1, bias_attr=False)\n self.bn1 = BatchNorm(256)\n self.relu = nn.ReLU()\n self.dropout = nn.Dropout(0.1)\n self._init_weight()\n def forward(self, x):\n x1 = self.aspp1(x)\n x2 = self.aspp2(x)\n x3 = self.aspp3(x)\n x4 = self.aspp4(x)\n x5 = self.global_avg_pool(x)\n x5 = F.interpolate(x5,\n size=x4.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat(x=[x1, x2, x3, x4, x5], axis=1)\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n return self.dropout(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:332-363"
+ },
+ "6419": {
+ "file_id": 496,
+ "content": "This code defines a DeepLab backbone model for image segmentation. It has adaptive global average pooling, multiple ASPP modules, and convolutional layers with batch normalization, ReLU activation, and dropout regularization. The constructor initializes the model's sublayers with Kaiming Normal initialization.",
+ "type": "comment"
+ },
+ "6420": {
+ "file_id": 496,
+ "content": " elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass Decoder(nn.Layer):\n def __init__(self, backbone, BatchNorm):\n super(Decoder, self).__init__()\n if backbone == 'resnet':\n low_level_inplanes = 256\n elif backbone == 'mobilenet':\n raise NotImplementedError\n else:\n raise NotImplementedError\n self.conv1 = nn.Conv2D(low_level_inplanes, 48, 1, bias_attr=False)\n self.bn1 = BatchNorm(48)\n self.relu = nn.ReLU()\n self.last_conv = nn.Sequential(\n nn.Conv2D(304,\n 256,\n kernel_size=3,\n stride=1,\n padding=1,\n bias_attr=False), BatchNorm(256), nn.ReLU(),\n nn.Sequential(),\n nn.Conv2D(256,\n 256,\n kernel_size=3,\n stride=1,\n padding=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:364-395"
+ },
+ "6421": {
+ "file_id": 496,
+ "content": "This code is defining a Decoder class that takes in a backbone and BatchNorm as arguments. It initializes a convolution layer, batch normalization layer, and ReLU activation function. The last convolution sequence includes two convolutional layers with BatchNorm between them, followed by an optional second sequence of layers.",
+ "type": "comment"
+ },
+ "6422": {
+ "file_id": 496,
+ "content": " bias_attr=False), BatchNorm(256), nn.ReLU(),\n nn.Sequential())\n self._init_weight()\n def forward(self, x, low_level_feat):\n low_level_feat = self.conv1(low_level_feat)\n low_level_feat = self.bn1(low_level_feat)\n low_level_feat = self.relu(low_level_feat)\n x = F.interpolate(x,\n size=low_level_feat.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat(x=[x, low_level_feat], axis=1)\n x = self.last_conv(x)\n return x\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass DeepLab(nn.Layer):\n \"\"\"DeepLab model for segmentation\"\"\"\n def __init__(self, backbone='resnet', output_stride=16, freeze_bn=True):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:396-426"
+ },
+ "6423": {
+ "file_id": 496,
+ "content": "The provided code defines a DeepLab model for segmentation. It includes a convolution layer, batch normalization, ReLU activation function, and interpolation operation. The forward method processes input features and returns output features. The _init_weight method initializes the weight of each sublayer. The DeepLab class takes parameters like backbone type, output stride, and freeze batch normalization flag for model initialization.",
+ "type": "comment"
+ },
+ "6424": {
+ "file_id": 496,
+ "content": " super(DeepLab, self).__init__()\n if freeze_bn == True:\n print(\"Use frozen BN in DeepLab!\")\n BatchNorm = FrozenBatchNorm2D\n else:\n BatchNorm = nn.BatchNorm2D\n self.backbone = ResNet(Bottleneck, [3, 4, 23, 3],\n output_stride,\n BatchNorm,\n pretrained=True)\n self.aspp = ASPP(backbone, output_stride, BatchNorm)\n self.decoder = Decoder(backbone, BatchNorm)\n def forward(self, input, return_aspp=False):\n \"\"\"forward function\"\"\"\n if return_aspp:\n x, low_level_feat, mid_level_feat = self.backbone(input, True)\n else:\n x, low_level_feat = self.backbone(input)\n aspp_x = self.aspp(x)\n x = self.decoder(aspp_x, low_level_feat)\n if return_aspp:\n return x, aspp_x, low_level_feat, mid_level_feat\n else:\n return x, low_level_feat",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/deeplab.py:427-454"
+ },
+ "6425": {
+ "file_id": 496,
+ "content": "The code defines a DeepLab class with an optional frozen Batch Normalization layer. It initializes the backbone network (ResNet) and adds ASPP and Decoder modules. The forward function performs inference, returning either the final output or additional intermediate features depending on the return_aspp flag.",
+ "type": "comment"
+ },
+ "6426": {
+ "file_id": 497,
+ "content": "/paddlevideo/modeling/backbones/movinet.py",
+ "type": "filepath"
+ },
+ "6427": {
+ "file_id": 497,
+ "content": "The code defines a MoViNet model configuration with MobileNetV2 layers and parameters, constructing CNN layers and a MoViNet backbone class for video analysis. The model is configurable and can be causal or non-causal based on the 'causal' parameter.",
+ "type": "summary"
+ },
+ "6428": {
+ "file_id": 497,
+ "content": "import collections.abc\nfrom itertools import repeat\nfrom typing import Any, Callable, Optional, Tuple, Union\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn.layer import Identity\nfrom ..registry import BACKBONES\nfrom collections import OrderedDict\ncontainer_abcs = collections.abc\n\"\"\"Model Config\n\"\"\"\nA0 = {'block_num': [0, 1, 3, 3, 4, 4]}\nA0['conv1'] = [3, 8, (1, 3, 3), (1, 2, 2), (0, 1, 1)]\nA0['b2_l0'] = [8, 8, 24, (1, 5, 5), (1, 2, 2), (0, 2, 2), (0, 1, 1)]\nA0['b3_l0'] = [8, 32, 80, (3, 3, 3), (1, 2, 2), (1, 0, 0), (0, 0, 0)]\nA0['b3_l1'] = [32, 32, 80, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]\nA0['b3_l2'] = [32, 32, 80, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]\nA0['b4_l0'] = [32, 56, 184, (5, 3, 3), (1, 2, 2), (2, 0, 0), (0, 0, 0)]\nA0['b4_l1'] = [56, 56, 112, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]\nA0['b4_l2'] = [56, 56, 184, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]\nA0['b5_l0'] = [56, 56, 184, (5, 3, 3), (1, 1, 1), (2, 1, 1), (0, 1, 1)]\nA0['b5_l1'] = [56, 56, 184, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:1-27"
+ },
+ "6429": {
+ "file_id": 497,
+ "content": "This code contains the configuration for a MOViNet model. It specifies the number of blocks, convolutional layers, and filter sizes for each stage of the network. The configuration is stored in a dictionary format with keys like 'A0', 'b2_l0', and so on, representing different parts of the model architecture.",
+ "type": "comment"
+ },
+ "6430": {
+ "file_id": 497,
+ "content": "A0['b5_l2'] = [56, 56, 184, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]\nA0['b5_l3'] = [56, 56, 184, (3, 3, 3), (1, 1, 1), (1, 1, 1), (0, 1, 1)]\nA0['b6_l0'] = [56, 104, 384, (5, 3, 3), (1, 2, 2), (2, 1, 1), (0, 1, 1)]\nA0['b6_l1'] = [104, 104, 280, (1, 5, 5), (1, 1, 1), (0, 2, 2), (0, 1, 1)]\nA0['b6_l2'] = [104, 104, 280, (1, 5, 5), (1, 1, 1), (0, 2, 2), (0, 1, 1)]\nA0['b6_l3'] = [104, 104, 344, (1, 5, 5), (1, 1, 1), (0, 2, 2), (0, 1, 1)]\nA0['conv7'] = [104, 480, (1, 1, 1), (1, 1, 1), (0, 0, 0)]\nMODEL_CONFIG = {'A0': A0}\ndef _ntuple(n):\n def parse(x):\n if isinstance(x, container_abcs.Iterable):\n return x\n return tuple(repeat(x, n))\n return parse\ndef _make_divisible(v: float,\n divisor: int,\n min_value: Optional[int] = None) -> int:\n \"\"\"\n This function is taken from the original tf repo.\n It ensures that all layers have a channel number that is divisible by 8.\n It can be seen here:\n https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:28-55"
+ },
+ "6431": {
+ "file_id": 497,
+ "content": "This code defines a model architecture for the MobileNetV2 network, specifying the number of filters, kernel sizes, and stride values at each layer. The `_ntuple` function parses layer configurations, and `_make_divisible` ensures all layers have a divisible channel number. The dictionary `MODEL_CONFIG` stores these configuration parameters for the model.",
+ "type": "comment"
+ },
+ "6432": {
+ "file_id": 497,
+ "content": " \"\"\"\n if min_value is None:\n min_value = divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n # Make sure that round down does not go down by more than 10%.\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\n_single = _ntuple(1)\n_pair = _ntuple(2)\n_triple = _ntuple(3)\n_quadruple = _ntuple(4)\nclass CausalModule(nn.Layer):\n def __init__(self) -> None:\n super().__init__()\n self.activation = None\n def reset_activation(self) -> None:\n self.activation = None\nclass Conv2dBNActivation(nn.Sequential):\n def __init__(\n self,\n in_planes: int,\n out_planes: int,\n kernel_size: Union[int, Tuple[int, int]],\n padding: Union[int, Tuple[int, int]],\n stride: Union[int, Tuple[int, int]] = 1,\n groups: int = 1,\n norm_layer: Optional[Callable[..., nn.Layer]] = None,\n activation_layer: Optional[Callable[..., nn.Layer]] = None,\n **kwargs: Any,\n ) -> None:\n kernel_size = _pair(kernel_size)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:56-94"
+ },
+ "6433": {
+ "file_id": 497,
+ "content": "The code defines a CausalModule class that contains an activation layer and resets it when needed. Conv2dBNActivation is a Sequential module with optional normalization and activation layers, used in the construction of the MoviNet backbone model.",
+ "type": "comment"
+ },
+ "6434": {
+ "file_id": 497,
+ "content": " stride = _pair(stride)\n padding = _pair(padding)\n if norm_layer is None:\n norm_layer = Identity\n if activation_layer is None:\n activation_layer = Identity\n self.kernel_size = kernel_size\n self.stride = stride\n dict_layers = (nn.Conv2D(in_planes,\n out_planes,\n kernel_size=kernel_size,\n stride=stride,\n padding=padding,\n groups=groups,\n **kwargs), norm_layer(out_planes,\n momentum=0.1),\n activation_layer())\n self.out_channels = out_planes\n super(Conv2dBNActivation, self).__init__(dict_layers[0], dict_layers[1],\n dict_layers[2])\nclass Conv3DBNActivation(nn.Sequential):\n def __init__(\n self,\n in_planes: int,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:95-121"
+ },
+ "6435": {
+ "file_id": 497,
+ "content": "This code defines two classes, `Conv2dBNActivation` and `Conv3DBNActivation`, which are convolutional neural network layers with batch normalization and activation functions. The layers have adjustable input (in_planes), output (out_planes), kernel size, stride, padding, and groups parameters. The batch normalization layer uses a momentum of 0.1, and the activation function is an Identity function by default but can be overridden with another specified activation function.",
+ "type": "comment"
+ },
+ "6436": {
+ "file_id": 497,
+ "content": " out_planes: int,\n kernel_size: Union[int, Tuple[int, int, int]],\n padding: Union[int, Tuple[int, int, int]],\n stride: Union[int, Tuple[int, int, int]] = 1,\n groups: int = 1,\n norm_layer: Optional[Callable[..., nn.Layer]] = None,\n activation_layer: Optional[Callable[..., nn.Layer]] = None,\n **kwargs: Any,\n ) -> None:\n kernel_size = _triple(kernel_size)\n stride = _triple(stride)\n padding = _triple(padding)\n if norm_layer is None:\n norm_layer = Identity\n if activation_layer is None:\n activation_layer = Identity\n self.kernel_size = kernel_size\n self.stride = stride\n dict_layers = (nn.Conv3D(in_planes,\n out_planes,\n kernel_size=kernel_size,\n stride=stride,\n padding=padding,\n groups=groups,\n **kwargs), norm_layer(out_planes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:122-147"
+ },
+ "6437": {
+ "file_id": 497,
+ "content": "This function is creating a Conv3D layer with specified parameters, including the number of input and output planes, kernel size, padding, stride, groups, and optional norm and activation layers. The function also ensures that the input values for kernel_size, stride, and padding are correctly formatted as triples.",
+ "type": "comment"
+ },
+ "6438": {
+ "file_id": 497,
+ "content": " momentum=0.1),\n activation_layer())\n self.out_channels = out_planes\n super(Conv3DBNActivation, self).__init__(dict_layers[0], dict_layers[1],\n dict_layers[2])\nclass ConvBlock3D(CausalModule):\n def __init__(\n self,\n in_planes: int,\n out_planes: int,\n kernel_size: Union[int, Tuple[int, int, int]],\n causal: bool,\n conv_type: str,\n padding: Union[int, Tuple[int, int, int]] = 0,\n stride: Union[int, Tuple[int, int, int]] = 1,\n norm_layer: Optional[Callable[..., nn.Layer]] = None,\n activation_layer: Optional[Callable[..., nn.Layer]] = None,\n bias_attr: bool = False,\n **kwargs: Any,\n ) -> None:\n super().__init__()\n kernel_size = _triple(kernel_size)\n stride = _triple(stride)\n padding = _triple(padding)\n self.conv_2 = None\n if causal is True:\n padding = (0, padding[1], padding[2])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:148-177"
+ },
+ "6439": {
+ "file_id": 497,
+ "content": "The code defines a class named `ConvBlock3D` as a subclass of `CausalModule`. It takes inputs such as the number of input and output planes, kernel size, causality status, convolution type, padding, stride, normalization layer, activation layer, bias attribute, and optional keyword arguments. It initializes the class variables and creates an instance of the `Conv3DBNActivation` class.",
+ "type": "comment"
+ },
+ "6440": {
+ "file_id": 497,
+ "content": " if conv_type != \"2plus1d\" and conv_type != \"3d\":\n raise ValueError(\"only 2plus2d or 3d are \" +\n \"allowed as 3d convolutions\")\n if conv_type == \"2plus1d\":\n self.conv_1 = Conv2dBNActivation(in_planes,\n out_planes,\n kernel_size=(kernel_size[1],\n kernel_size[2]),\n padding=(padding[1], padding[2]),\n stride=(stride[1], stride[2]),\n activation_layer=activation_layer,\n norm_layer=norm_layer,\n bias_attr=bias_attr,\n **kwargs)\n if kernel_size[0] > 1:\n self.conv_2 = Conv2dBNActivation(\n in_planes,\n out_planes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:178-196"
+ },
+ "6441": {
+ "file_id": 497,
+ "content": "This code is checking the convolution type and raising a ValueError if it's neither \"2plus1d\" nor \"3d\". If the type is \"2plus1d\", it initializes two Conv2dBNActivation layers with appropriate parameters.",
+ "type": "comment"
+ },
+ "6442": {
+ "file_id": 497,
+ "content": " kernel_size=(kernel_size[0], 1),\n padding=(padding[0], 0),\n stride=(stride[0], 1),\n activation_layer=activation_layer,\n norm_layer=norm_layer,\n bias_attr=bias_attr,\n **kwargs)\n elif conv_type == \"3d\":\n self.conv_1 = Conv3DBNActivation(in_planes,\n out_planes,\n kernel_size=kernel_size,\n padding=padding,\n activation_layer=activation_layer,\n norm_layer=norm_layer,\n stride=stride,\n bias_attr=bias_attr,\n **kwargs)\n self.padding = padding\n self.kernel_size = kernel_size\n self.dim_pad = self.kernel_size[0] - 1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:197-216"
+ },
+ "6443": {
+ "file_id": 497,
+ "content": "The code defines a layer with different convolution types (\"2d\" or \"3d\") and initializes the corresponding Conv2D or Conv3D layers with specified parameters such as input/output planes, kernel size, padding, activation layer, norm layer, stride, bias attribute and other keyword arguments. It also stores the padding and kernel size for future use.",
+ "type": "comment"
+ },
+ "6444": {
+ "file_id": 497,
+ "content": " self.stride = stride\n self.causal = causal\n self.conv_type = conv_type\n def _forward(self, x: paddle.Tensor) -> paddle.Tensor:\n if self.dim_pad > 0 and self.conv_2 is None and self.causal is True:\n x = self._cat_stream_buffer(x)\n b, c, t, h, w = x.shape\n if self.conv_type == \"2plus1d\":\n x = paddle.transpose(x, (0, 2, 1, 3, 4)) # bcthw --> btchw\n x = paddle.reshape_(x, (-1, c, h, w)) # btchw --> bt,c,h,w\n x = self.conv_1(x)\n if self.conv_type == \"2plus1d\":\n b, c, h, w = x.shape\n x = paddle.reshape_(x, (-1, t, c, h, w)) # bt,c,h,w --> b,t,c,h,w\n x = paddle.transpose(x, (0, 2, 1, 3, 4)) # b,t,c,h,w --> b,c,t,h,w\n if self.conv_2 is not None:\n if self.dim_pad > 0 and self.causal is True:\n x = self._cat_stream_buffer(x)\n b, c, t, h, w = x.shape\n x = paddle.reshape_(x, (b, c, t, h * w))\n x = self.conv_2(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:217-238"
+ },
+ "6445": {
+ "file_id": 497,
+ "content": "This code defines a class with an attribute `_forward` method. The constructor takes stride, causal, and conv_type as parameters. If causal is True, stream buffer is concatenated to the input tensor. Depending on conv_type, the tensor shape may be reshaped for proper processing. Finally, if conv_2 is not None, it applies a convolution operation to the tensor.",
+ "type": "comment"
+ },
+ "6446": {
+ "file_id": 497,
+ "content": " b, c, t, _ = x.shape\n x = paddle.reshape_(x, (b, c, t, h, w))\n return x\n def forward(self, x: paddle.Tensor) -> paddle.Tensor:\n x = self._forward(x)\n return x\n def _cat_stream_buffer(self, x: paddle.Tensor) -> paddle.Tensor:\n if self.activation is None:\n self._setup_activation(x.shape)\n x = paddle.concat((self.activation, x), 2)\n self._save_in_activation(x)\n return x\n def _save_in_activation(self, x: paddle.Tensor) -> None:\n assert self.dim_pad > 0\n self.activation = paddle.to_tensor(x.numpy()[:, :, -self.dim_pad:,\n ...]).clone().detach()\n def _setup_activation(self, input_shape: Tuple[float, ...]) -> None:\n assert self.dim_pad > 0\n self.activation = paddle.zeros(shape=[\n *input_shape[:2], # type: ignore\n self.dim_pad,\n *input_shape[3:]\n ])\nclass TemporalCGAvgPool3D(CausalModule):\n def __init__(self, ) -> None:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:239-269"
+ },
+ "6447": {
+ "file_id": 497,
+ "content": "1. Reshapes input tensor to (b, c, t, h, w).\n2. Defines a forward function that applies the _forward function and returns the result.\n3. Concatenates the activation tensor with the input along dimension 2.\n4. Saves the last self.dim_pad rows of the input in the activation tensor.\n5. Sets up the activation tensor with zeros and self.dim_pad rows for future use.\n6. TemporalCGAvgPool3D is a CausalModule class.",
+ "type": "comment"
+ },
+ "6448": {
+ "file_id": 497,
+ "content": " super().__init__()\n self.n_cumulated_values = 0\n self.register_forward_post_hook(self._detach_activation)\n def forward(self, x: paddle.Tensor) -> paddle.Tensor:\n input_shape = x.shape\n cumulative_sum = paddle.cumsum(x, axis=2)\n if self.activation is None:\n self.activation = cumulative_sum[:, :, -1:].clone()\n else:\n cumulative_sum += self.activation\n self.activation = cumulative_sum[:, :, -1:].clone()\n noe = paddle.arange(1, input_shape[2] + 1)\n axis = paddle.to_tensor([0, 1, 3, 4])\n noe = paddle.unsqueeze(noe, axis=axis)\n divisor = noe.expand(x.shape)\n x = cumulative_sum / (self.n_cumulated_values + divisor)\n self.n_cumulated_values += input_shape[2]\n return x\n @staticmethod\n def _detach_activation(module: CausalModule, inputs: paddle.Tensor,\n output: paddle.Tensor) -> None:\n module.activation.detach()\n def reset_activation(self) -> None:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:270-296"
+ },
+ "6449": {
+ "file_id": 497,
+ "content": "The code defines a forward function for a CausalModule that performs cumulative sum operation on input tensor. It also includes methods to detach and reset the activation tensor.",
+ "type": "comment"
+ },
+ "6450": {
+ "file_id": 497,
+ "content": " super().reset_activation()\n self.n_cumulated_values = 0\nclass SqueezeExcitation(nn.Layer):\n def __init__(self,\n input_channels: int,\n activation_2: nn.Layer,\n activation_1: nn.Layer,\n conv_type: str,\n causal: bool,\n squeeze_factor: int = 4,\n bias_attr: bool = True) -> None:\n super().__init__()\n self.causal = causal\n se_multiplier = 2 if causal else 1\n squeeze_channels = _make_divisible(\n input_channels // squeeze_factor * se_multiplier, 8)\n self.temporal_cumualtive_GAvg3D = TemporalCGAvgPool3D()\n self.fc1 = ConvBlock3D(input_channels * se_multiplier,\n squeeze_channels,\n kernel_size=(1, 1, 1),\n padding=0,\n causal=causal,\n conv_type=conv_type,\n bias_attr=bias_attr)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:297-322"
+ },
+ "6451": {
+ "file_id": 497,
+ "content": "This code defines a SqueezeExcitation layer class with input channels, activation functions, convolution type, causality flag, and squeeze factor as parameters. It initializes the layer by setting the causal flag's multiplier, dividing the input channel count by the squeeze factor, rounding up to 8 using make_divisible function, and adding temporal cumulative average pooling and convolution blocks with specified parameters.",
+ "type": "comment"
+ },
+ "6452": {
+ "file_id": 497,
+ "content": " self.activation_1 = activation_1()\n self.activation_2 = activation_2()\n self.fc2 = ConvBlock3D(squeeze_channels,\n input_channels,\n kernel_size=(1, 1, 1),\n padding=0,\n causal=causal,\n conv_type=conv_type,\n bias_attr=bias_attr)\n def _scale(self, inputs: paddle.Tensor) -> paddle.Tensor:\n if self.causal:\n x_space = paddle.mean(inputs, axis=[3, 4], keepdim=True)\n scale = self.temporal_cumualtive_GAvg3D(x_space)\n scale = paddle.concat((scale, x_space), axis=1)\n else:\n scale = F.adaptive_avg_pool3d(inputs, 1)\n scale = self.fc1(scale)\n scale = self.activation_1(scale)\n scale = self.fc2(scale)\n return self.activation_2(scale)\n def forward(self, inputs: paddle.Tensor) -> paddle.Tensor:\n scale = self._scale(inputs)\n return scale * inputs",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:323-347"
+ },
+ "6453": {
+ "file_id": 497,
+ "content": "The code defines a class with two activation functions, and a _scale method that scales the input tensor based on temporal average or average pooling. The forward method applies the scale to the input for spatial pyramid pooling.",
+ "type": "comment"
+ },
+ "6454": {
+ "file_id": 497,
+ "content": "class BasicBneck(nn.Layer):\n def __init__(\n self,\n input_channels,\n out_channels,\n expanded_channels,\n kernel_size,\n stride,\n padding,\n padding_avg,\n causal: bool,\n conv_type: str,\n norm_layer: Optional[Callable[..., nn.Layer]] = None,\n activation_layer: Optional[Callable[..., nn.Layer]] = None,\n ) -> None:\n super().__init__()\n assert type(stride) is tuple\n if (not stride[0] == 1 or not (1 <= stride[1] <= 2)\n or not (1 <= stride[2] <= 2)):\n raise ValueError('illegal stride value')\n self.res = None\n layers = []\n if expanded_channels != out_channels:\n # expand\n self.expand = ConvBlock3D(in_planes=input_channels,\n out_planes=expanded_channels,\n kernel_size=(1, 1, 1),\n padding=(0, 0, 0),\n causal=causal,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:350-382"
+ },
+ "6455": {
+ "file_id": 497,
+ "content": "This code defines the BasicBneck class which is a neural network layer. It has multiple parameters such as input_channels, out_channels, expanded_channels, kernel_size, stride, padding, padding_avg, causal, conv_type, norm_layer, and activation_layer. If expanded_channels is not equal to out_channels, it will first expand the channels using ConvBlock3D. The class also checks for illegal stride values to prevent unexpected behavior.",
+ "type": "comment"
+ },
+ "6456": {
+ "file_id": 497,
+ "content": " conv_type=conv_type,\n norm_layer=norm_layer,\n activation_layer=activation_layer)\n # deepwise\n self.deep = ConvBlock3D(in_planes=expanded_channels,\n out_planes=expanded_channels,\n kernel_size=kernel_size,\n padding=padding,\n stride=stride,\n groups=expanded_channels,\n causal=causal,\n conv_type=conv_type,\n norm_layer=norm_layer,\n activation_layer=activation_layer)\n # SE\n self.se = SqueezeExcitation(\n expanded_channels,\n causal=causal,\n activation_1=activation_layer,\n activation_2=(nn.Sigmoid if conv_type == \"3d\" else nn.Hardsigmoid),\n conv_type=conv_type)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:383-404"
+ },
+ "6457": {
+ "file_id": 497,
+ "content": "This code defines a ConvBlock3D for MoviNet backbone, includes deepwise convolution and SE (Squeeze Excitation) layers. These components process 3D feature maps with various configurations depending on the input planes, kernel size, stride, padding, etc., applying different activation functions based on the conv_type.",
+ "type": "comment"
+ },
+ "6458": {
+ "file_id": 497,
+ "content": " # project\n self.project = ConvBlock3D(expanded_channels,\n out_channels,\n kernel_size=(1, 1, 1),\n padding=(0, 0, 0),\n causal=causal,\n conv_type=conv_type,\n norm_layer=norm_layer,\n activation_layer=Identity)\n if not (stride == (1, 1, 1) and input_channels == out_channels):\n if stride != (1, 1, 1):\n layers.append(\n nn.AvgPool3D((1, 3, 3), stride=stride, padding=padding_avg))\n layers.append(\n ConvBlock3D(\n in_planes=input_channels,\n out_planes=out_channels,\n kernel_size=(1, 1, 1),\n padding=(0, 0, 0),\n norm_layer=norm_layer,\n activation_layer=Identity,\n causal=causal,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:405-427"
+ },
+ "6459": {
+ "file_id": 497,
+ "content": "This code defines a ConvBlock3D for projecting the input channels to the desired output channels. If the stride is not (1, 1, 1) or input and output channels are different, it adds an average pooling layer and another ConvBlock3D with appropriate parameters. The causal parameter determines if causal convolution should be used, and Identity activation layer is applied without any transformation.",
+ "type": "comment"
+ },
+ "6460": {
+ "file_id": 497,
+ "content": " conv_type=conv_type,\n ))\n self.res = nn.Sequential(*layers)\n self.alpha = self.create_parameter(shape=[1], dtype=\"float32\")\n def forward(self, inputs: paddle.Tensor) -> paddle.Tensor:\n if self.res is not None:\n residual = self.res(inputs)\n else:\n residual = inputs\n if self.expand is not None:\n x = self.expand(inputs)\n else:\n x = inputs\n x = self.deep(x)\n x = self.se(x)\n x = self.project(x)\n result = residual + self.alpha * x\n return result\n@BACKBONES.register()\nclass MoViNet(nn.Layer):\n def __init__(\n self,\n model_type: str = 'A0',\n hidden_dim: int = 2048,\n causal: bool = True,\n num_classes: int = 400,\n conv_type: str = \"3d\",\n ) -> None:\n super().__init__()\n \"\"\"\n causal: causal mode\n num_classes: number of classes for classifcation\n conv_type: type of convolution either 3d or 2plus1d",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:428-464"
+ },
+ "6461": {
+ "file_id": 497,
+ "content": "The code defines the MoViNet class, which is a backbone model for video analysis. It initializes layers based on input parameters and then performs feature extraction using the defined layers. The forward method applies residual connections and a scale factor to combine the extracted features with the input.",
+ "type": "comment"
+ },
+ "6462": {
+ "file_id": 497,
+ "content": " \"\"\"\n blocks_dic = OrderedDict()\n cfg = MODEL_CONFIG[model_type]\n norm_layer = nn.BatchNorm3D if conv_type == \"3d\" else nn.BatchNorm2D\n activation_layer = nn.Swish if conv_type == \"3d\" else nn.Hardswish\n # conv1\n self.conv1 = ConvBlock3D(in_planes=cfg['conv1'][0],\n out_planes=cfg['conv1'][1],\n kernel_size=cfg['conv1'][2],\n stride=cfg['conv1'][3],\n padding=cfg['conv1'][4],\n causal=causal,\n conv_type=conv_type,\n norm_layer=norm_layer,\n activation_layer=activation_layer)\n # blocks\n for i in range(2, len(cfg['block_num']) + 1):\n for j in range(cfg['block_num'][i - 1]):\n blocks_dic[f'b{i}_l{j}'] = BasicBneck(\n cfg[f'b{i}_l{j}'][0],\n cfg[f'b{i}_l{j}'][1],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:465-487"
+ },
+ "6463": {
+ "file_id": 497,
+ "content": "The code defines a MOViNet model, which consists of a ConvBlock3D (conv1) and multiple BasicBneck blocks. It takes in parameters such as the number of input and output planes, kernel size, stride, padding, causal flag, conv type, norm layer, and activation layer. These parameters are extracted from the MODEL_CONFIG dictionary based on the model type. The blocks are organized in an OrderedDict called blocks_dic for future reference.",
+ "type": "comment"
+ },
+ "6464": {
+ "file_id": 497,
+ "content": " cfg[f'b{i}_l{j}'][2],\n cfg[f'b{i}_l{j}'][3],\n cfg[f'b{i}_l{j}'][4],\n cfg[f'b{i}_l{j}'][5],\n cfg[f'b{i}_l{j}'][6],\n causal=causal,\n conv_type=conv_type,\n norm_layer=norm_layer,\n activation_layer=activation_layer)\n self.blocks = nn.Sequential(*(blocks_dic.values()))\n # conv7\n self.conv7 = ConvBlock3D(in_planes=cfg['conv7'][0],\n out_planes=cfg['conv7'][1],\n kernel_size=cfg['conv7'][2],\n stride=cfg['conv7'][3],\n padding=cfg['conv7'][4],\n causal=causal,\n conv_type=conv_type,\n norm_layer=norm_layer,\n activation_layer=activation_layer)\n # pool\n self.classifier = nn.Sequential(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:488-510"
+ },
+ "6465": {
+ "file_id": 497,
+ "content": "This code is defining a MOViNet model with specific configurations for blocks, convolutional layers, and pooling operations. It initializes the blocks as sequential layers and adds an additional 3D ConvBlock layer ('conv7') followed by a classifier.",
+ "type": "comment"
+ },
+ "6466": {
+ "file_id": 497,
+ "content": " # dense9\n ConvBlock3D(in_planes=cfg['conv7'][1],\n out_planes=hidden_dim,\n kernel_size=(1, 1, 1),\n causal=causal,\n conv_type=conv_type,\n bias_attr=True),\n nn.Swish(),\n nn.Dropout(p=0.2),\n # dense10d\n ConvBlock3D(in_planes=hidden_dim,\n out_planes=num_classes,\n kernel_size=(1, 1, 1),\n causal=causal,\n conv_type=conv_type,\n bias_attr=True),\n )\n if causal:\n self.cgap = TemporalCGAvgPool3D()\n self.apply(self._weight_init)\n self.causal = causal\n def avg(self, x: paddle.Tensor) -> paddle.Tensor:\n if self.causal:\n avg = F.adaptive_avg_pool3d(x, (x.shape[2], 1, 1))\n avg = self.cgap(avg)[:, :, -1:]\n else:\n avg = F.adaptive_avg_pool3d(x, 1)\n return avg",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:511-539"
+ },
+ "6467": {
+ "file_id": 497,
+ "content": "This code defines a 3D Convolutional Neural Network (CNN) backbone for MoviNet. It includes dense layers, convolution blocks, and optional temporal pooling. The model architecture can be causal or non-causal depending on the `causal` parameter.",
+ "type": "comment"
+ },
+ "6468": {
+ "file_id": 497,
+ "content": " @staticmethod\n def _weight_init(m):\n if isinstance(m, nn.Conv3D):\n nn.initializer.KaimingNormal(m.weight)\n if m.bias is not None:\n nn.initializer.Constant(0.0)(m.bias)\n elif isinstance(m, (nn.BatchNorm3D, nn.BatchNorm2D, nn.GroupNorm)):\n nn.initializer.Constant(1.0)(m.weight)\n nn.initializer.Constant(0.0)(m.bias)\n elif isinstance(m, nn.Linear):\n nn.initializer.Normal(m.weight, 0, 0.01)\n nn.initializer.Constant(0.0)(m.bias)\n def forward(self, x: paddle.Tensor) -> paddle.Tensor:\n x = self.conv1(x)\n x = self.blocks(x)\n x = self.conv7(x)\n x = self.avg(x)\n x = self.classifier(x)\n x = x.flatten(1)\n return x\n @staticmethod\n def _clean_activation_buffers(m):\n if issubclass(type(m), CausalModule):\n m.reset_activation()\n def clean_activation_buffers(self) -> None:\n self.apply(self._clean_activation_buffers)\nif __name__ == '__main__':",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:541-572"
+ },
+ "6469": {
+ "file_id": 497,
+ "content": "The code defines a class for a MoviNet backbone, which performs convolutions and has block layers. The forward function passes the input through these layers and then flattens the result before returning it. A static method initializes the network weights based on the layer type. Another static method cleans activation buffers in CausalModule subclasses.",
+ "type": "comment"
+ },
+ "6470": {
+ "file_id": 497,
+ "content": " net = MoViNet(causal=False, conv_type='3d')\n paddle.summary(net, input_size=(1, 3, 8, 224, 224))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/movinet.py:573-574"
+ },
+ "6471": {
+ "file_id": 497,
+ "content": "Creating a MoViNet network instance with causal set to False and 3D convolution type, then generating summary using Paddle's summary function with input size (1, 3, 8, 224, 224).",
+ "type": "comment"
+ },
+ "6472": {
+ "file_id": 498,
+ "content": "/paddlevideo/modeling/backbones/ms_tcn.py",
+ "type": "filepath"
+ },
+ "6473": {
+ "file_id": 498,
+ "content": "The code imports modules, defines Kaiming uniform initialization and SingleStageModel class. It initializes MSTCN backbone with DilatedResidualLayer stages and applies softmax to previous outputs, concatenating them together while initializing weights for convolutional layers with KaimingUniform_like_torch.",
+ "type": "summary"
+ },
+ "6474": {
+ "file_id": 498,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nimport copy\nimport random\nimport math\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = len(tensor.shape)\n if dimensions < 2:\n raise ValueError(\"Fan in and fan out can not be computed \\\n for tensor with fewer than 2 dimensions\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ms_tcn.py:1-32"
+ },
+ "6475": {
+ "file_id": 498,
+ "content": "This code snippet appears to be part of a larger file and sets up some initial definitions, imports, and checks for necessary conditions. It includes license information, imports various modules, and defines a function to calculate fan-in and fan-out for tensor dimensions.",
+ "type": "comment"
+ },
+ "6476": {
+ "file_id": 498,
+ "content": " if dimensions == 2: # Linear\n fan_in = tensor.shape[1]\n fan_out = tensor.shape[0]\n else:\n num_input_fmaps = tensor.shape[1]\n num_output_fmaps = tensor.shape[0]\n receptive_field_size = 1\n if tensor.dim() > 2:\n receptive_field_size = tensor[0][0].numel()\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef calculate_gain(nonlinearity=None, a=None):\n if nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if a != None:\n return math.sqrt(2.0 / (1 + a**2))\n else:\n return math.sqrt(2.0 / (1 + 0.01**2))\n elif nonlinearity == 'selu':\n return 3.0 / 4\n else:\n return 1\ndef KaimingUniform_like_torch(weight_npy,\n mode='fan_in',\n nonlinearity='leaky_relu'):\n fan_in, fan_out = _calculate_fan_in_and_fan_out(weight_npy)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ms_tcn.py:34-68"
+ },
+ "6477": {
+ "file_id": 498,
+ "content": "This code defines three functions: `_calculate_fan_in_and_fan_out`, `calculate_gain`, and `KaimingUniform_like_torch`. The first function calculates the fan-in and fan-out values based on the input tensor's dimensions. The second function determines the gain value depending on the nonlinearity used. The third function applies the Kaiming uniform initialization to the weight_npy parameter, utilizing the previous two functions.",
+ "type": "comment"
+ },
+ "6478": {
+ "file_id": 498,
+ "content": " if mode == 'fan_in':\n fan_mode = fan_in\n else:\n fan_mode = fan_out\n a = math.sqrt(5.0)\n gain = calculate_gain(nonlinearity=nonlinearity, a=a)\n std = gain / math.sqrt(fan_mode)\n bound = math.sqrt(3.0) * std\n return np.random.uniform(-bound, bound, weight_npy.shape)\ndef init_bias(weight_npy, bias_npy):\n # attention this weight is not bias\n fan_in, fan_out = _calculate_fan_in_and_fan_out(weight_npy)\n bound = 1.0 / math.sqrt(fan_in)\n return np.random.uniform(-bound, bound, bias_npy.shape)\nclass SingleStageModel(nn.Layer):\n def __init__(self, num_layers, num_f_maps, dim, num_classes):\n super(SingleStageModel, self).__init__()\n self.conv_in = nn.Conv1D(dim, num_f_maps, 1)\n self.layers = nn.LayerList([\n copy.deepcopy(DilatedResidualLayer(2**i, num_f_maps, num_f_maps))\n for i in range(num_layers)\n ])\n self.conv_out = nn.Conv1D(num_f_maps, num_classes, 1)\n def forward(self, x):\n out = self.conv_in(x)\n for layer in self.layers:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ms_tcn.py:69-100"
+ },
+ "6479": {
+ "file_id": 498,
+ "content": "This code defines a SingleStageModel class that inherits from nn.Layer and consists of a convolutional layer, multiple DilatedResidualLayers, and another convolutional layer. The model is initialized with specified parameters: number of layers, number of feature maps, input dimension, and number of output classes.",
+ "type": "comment"
+ },
+ "6480": {
+ "file_id": 498,
+ "content": " out = layer(out)\n out = self.conv_out(out)\n return out\nclass DilatedResidualLayer(nn.Layer):\n def __init__(self, dilation, in_channels, out_channels):\n super(DilatedResidualLayer, self).__init__()\n self.conv_dilated = nn.Conv1D(in_channels,\n out_channels,\n 3,\n padding=dilation,\n dilation=dilation)\n self.conv_in = nn.Conv1D(out_channels, out_channels, 1)\n self.dropout = nn.Dropout()\n def forward(self, x):\n out = F.relu(self.conv_dilated(x))\n out = self.conv_in(out)\n out = self.dropout(out)\n return (x + out)\n@BACKBONES.register()\nclass MSTCN(nn.Layer):\n def __init__(self, num_stages, num_layers, num_f_maps, dim, num_classes):\n super().__init__()\n self.stage1 = SingleStageModel(num_layers, num_f_maps, dim, num_classes)\n self.stages = nn.LayerList([\n copy.deepcopy(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ms_tcn.py:101-132"
+ },
+ "6481": {
+ "file_id": 498,
+ "content": "The code defines a DilatedResidualLayer, which is a type of residual layer used in the MSTCN backbone. The MSTCN class initializes a SingleStageModel and a list of stages using the provided parameters. Each stage within the model is an instance of the DilatedResidualLayer.",
+ "type": "comment"
+ },
+ "6482": {
+ "file_id": 498,
+ "content": " SingleStageModel(num_layers, num_f_maps, num_classes,\n num_classes)) for s in range(num_stages - 1)\n ])\n def forward(self, x):\n \"\"\" MSTCN forward\n \"\"\"\n out = self.stage1(x)\n outputs = out.unsqueeze(0)\n for s in self.stages:\n out = s(F.softmax(out, axis=1))\n outputs = paddle.concat((outputs, out.unsqueeze(0)), axis=0)\n return outputs\n def init_weights(self):\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv1D):\n layer.weight.set_value(\n KaimingUniform_like_torch(layer.weight).astype('float32'))\n if layer.bias is not None:\n layer.bias.set_value(\n init_bias(layer.weight, layer.bias).astype('float32'))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/ms_tcn.py:133-154"
+ },
+ "6483": {
+ "file_id": 498,
+ "content": "The code defines a forward function for MSTCN model and initializes the weights for convolutional layers. It iterates over stages, applying softmax to previous output and concatenating it to previous outputs. Weights are initialized with KaimingUniform_like_torch for conv1D layers and bias is set according to the layer's weight.",
+ "type": "comment"
+ },
+ "6484": {
+ "file_id": 499,
+ "content": "/paddlevideo/modeling/backbones/pptsm_mv2.py",
+ "type": "filepath"
+ },
+ "6485": {
+ "file_id": 499,
+ "content": "The ConvBNLayer class introduces PaddlePaddle's MobileNetV2 backbone model for image/video processing with pretrained weights and inverted residual units. It initializes and returns three models (PPTSM_MobileNetV2_x0_75, PPTSM_MobileNetV2_x1_5, PPTSM_MobileNetV2_x2_0).",
+ "type": "summary"
+ },
+ "6486": {
+ "file_id": 499,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nfrom paddle import ParamAttr\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn import Conv2D, BatchNorm, Linear, Dropout\nfrom paddle.nn import AdaptiveAvgPool2D, MaxPool2D, AvgPool2D\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt\n# Download URL of pretrained model\n# {\n# \"MobileNetV2\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_ssld_pretrained.pdparams\",",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:1-30"
+ },
+ "6487": {
+ "file_id": 499,
+ "content": "This code is part of the PaddlePaddle deep learning framework, specifically for the MobileNetV2 backbone model. It imports necessary libraries and defines functions for the architecture, weight initialization, and pre-trained model downloading. The commented sections provide licensing information and download URLs for pretrained models.",
+ "type": "comment"
+ },
+ "6488": {
+ "file_id": 499,
+ "content": "# \"MobileNetV2_x0_25\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x0_25_pretrained.pdparams\",\n# \"MobileNetV2_x0_5\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x0_5_pretrained.pdparams\",\n# \"MobileNetV2_x0_75\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x0_75_pretrained.pdparams\",\n# \"MobileNetV2_x1_5\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x1_5_pretrained.pdparams\",\n# \"MobileNetV2_x2_0\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x2_0_pretrained.pdparams\"\n# }\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n num_channels,\n filter_size,\n num_filters,\n stride,\n padding,\n channels=None,\n num_groups=1,\n name=None,\n use_cudnn=True):\n super(ConvBNLayer, self).__init__()\n self._conv = Conv2D(in_channels=num_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:32-58"
+ },
+ "6489": {
+ "file_id": 499,
+ "content": "This code defines the ConvBNLayer class, which inherits from nn.Layer and contains a convolutional layer followed by a batch normalization layer. The constructor takes several parameters such as number of channels, filter size, etc., to define the specifics of the convolutional layer. The URLs provided indicate that pretrained models are available for MobileNetV2 with various scaling factors.",
+ "type": "comment"
+ },
+ "6490": {
+ "file_id": 499,
+ "content": " out_channels=num_filters,\n kernel_size=filter_size,\n stride=stride,\n padding=padding,\n groups=num_groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False)\n self._batch_norm = BatchNorm(\n num_filters,\n param_attr=ParamAttr(name=name + \"_bn_scale\"),\n bias_attr=ParamAttr(name=name + \"_bn_offset\"),\n moving_mean_name=name + \"_bn_mean\",\n moving_variance_name=name + \"_bn_variance\")\n def forward(self, inputs, if_act=True):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if if_act:\n y = F.relu6(y)\n return y\nclass InvertedResidualUnit(nn.Layer):\n def __init__(self, num_channels, num_in_filter, num_filters, stride,\n filter_size, padding, expansion_factor, name, num_seg):\n super(InvertedResidualUnit, self).__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:59-85"
+ },
+ "6491": {
+ "file_id": 499,
+ "content": "The code defines a class for an inverted residual unit with batch normalization. The unit takes input, performs convolution, applies batch normalization, and optionally applies activation if specified.",
+ "type": "comment"
+ },
+ "6492": {
+ "file_id": 499,
+ "content": " self.num_seg = num_seg\n num_expfilter = int(round(num_in_filter * expansion_factor))\n self._expand_conv = ConvBNLayer(num_channels=num_channels,\n num_filters=num_expfilter,\n filter_size=1,\n stride=1,\n padding=0,\n num_groups=1,\n name=name + \"_expand\")\n self._bottleneck_conv = ConvBNLayer(num_channels=num_expfilter,\n num_filters=num_expfilter,\n filter_size=filter_size,\n stride=stride,\n padding=padding,\n num_groups=num_expfilter,\n use_cudnn=False,\n name=name + \"_dwise\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:86-103"
+ },
+ "6493": {
+ "file_id": 499,
+ "content": "This code initializes and assigns class attributes for a backbone model. It defines two convolutional layers, one for expansion (num_channels to num_expfilter) and another for bottleneck (num_expfilter to num_expfilter), both followed by BN operations. The layers are named with the prefix \"name\" for future reference or identification.",
+ "type": "comment"
+ },
+ "6494": {
+ "file_id": 499,
+ "content": " self._linear_conv = ConvBNLayer(num_channels=num_expfilter,\n num_filters=num_filters,\n filter_size=1,\n stride=1,\n padding=0,\n num_groups=1,\n name=name + \"_linear\")\n def forward(self, inputs, ifshortcut):\n # add temporal shift module\n y = inputs\n if ifshortcut:\n y = F.temporal_shift(y, self.num_seg, 1.0 / self.num_seg)\n y = self._expand_conv(y, if_act=True)\n y = self._bottleneck_conv(y, if_act=True)\n y = self._linear_conv(y, if_act=False)\n if ifshortcut:\n y = paddle.add(inputs, y)\n return y\nclass InvresiBlocks(nn.Layer):\n def __init__(self, in_c, t, c, n, s, name, num_seg):\n super(InvresiBlocks, self).__init__()\n self._first_block = InvertedResidualUnit(num_channels=in_c,\n num_in_filter=in_c,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:105-132"
+ },
+ "6495": {
+ "file_id": 499,
+ "content": "This code defines a neural network layer, likely for image or video processing. It contains a series of convolutional layers and activation functions. The \"forward\" function applies temporal shift to the input based on the number of segments and performs convolutions in different stages. The \"InvresiBlocks\" class defines an Inverted Residual block with initial parameters.",
+ "type": "comment"
+ },
+ "6496": {
+ "file_id": 499,
+ "content": " num_filters=c,\n stride=s,\n filter_size=3,\n padding=1,\n expansion_factor=t,\n name=name + \"_1\",\n num_seg=num_seg)\n self._block_list = []\n for i in range(1, n):\n block = self.add_sublayer(name + \"_\" + str(i + 1),\n sublayer=InvertedResidualUnit(\n num_channels=c,\n num_in_filter=c,\n num_filters=c,\n stride=1,\n filter_size=3,\n padding=1,\n expansion_factor=t,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:133-151"
+ },
+ "6497": {
+ "file_id": 499,
+ "content": "The code defines a function for the PPTSM_MV2 model, creating an InvertedResidualUnit with specified parameters and adding it to a list. The loop iterates from 1 to n-1, building multiple residual units with increasing indexes.",
+ "type": "comment"
+ },
+ "6498": {
+ "file_id": 499,
+ "content": " name=name + \"_\" + str(i + 1),\n num_seg=num_seg))\n self._block_list.append(block)\n def forward(self, inputs):\n y = self._first_block(inputs, ifshortcut=False)\n for block in self._block_list:\n y = block(y, ifshortcut=True)\n return y\nclass MobileNet(nn.Layer):\n def __init__(self,\n class_num=400,\n scale=1.0,\n pretrained=None,\n prefix_name=\"\",\n num_seg=8):\n super(MobileNet, self).__init__()\n self.scale = scale\n self.class_num = class_num\n self.pretrained = pretrained\n self.num_seg = num_seg\n bottleneck_params_list = [\n (1, 16, 1, 1),\n (6, 24, 2, 2),\n (6, 32, 3, 2),\n (6, 64, 4, 2),\n (6, 96, 3, 1),\n (6, 160, 3, 2),\n (6, 320, 1, 1),\n ]\n self.conv1 = ConvBNLayer(num_channels=3,\n num_filters=int(32 * scale),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:152-187"
+ },
+ "6499": {
+ "file_id": 499,
+ "content": "This code defines a PPTSM-MV2 backbone and MobileNet model for image processing. The `__init__` function initializes the model with class number, scaling factor, pretrained weights, prefix name, and number of segments. The `forward` function passes inputs through each block in sequence. The `MobileNet` class defines a convolutional neural network (CNN) architecture with specific parameters for each stage, including the number of filters and stride sizes.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/65.json b/docs/data/65.json
new file mode 100644
index 000000000..56458d739
--- /dev/null
+++ b/docs/data/65.json
@@ -0,0 +1,548 @@
+{
+ "6500": {
+ "file_id": 499,
+ "content": " filter_size=3,\n stride=2,\n padding=1,\n name=prefix_name + \"conv1_1\")\n self.block_list = []\n i = 1\n in_c = int(32 * scale)\n for layer_setting in bottleneck_params_list:\n t, c, n, s = layer_setting\n i += 1\n block = self.add_sublayer(prefix_name + \"conv\" + str(i),\n sublayer=InvresiBlocks(in_c=in_c,\n t=t,\n c=int(c * scale),\n n=n,\n s=s,\n name=prefix_name +\n \"conv\" + str(i),\n num_seg=num_seg))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:188-207"
+ },
+ "6501": {
+ "file_id": 499,
+ "content": "This code initializes a PPTSM_MV2 backbone model. It adds a convolution layer with specific parameters, and creates a list of block layers using InvresiBlocks with varying settings. The scale value affects the number of input channels in each layer.",
+ "type": "comment"
+ },
+ "6502": {
+ "file_id": 499,
+ "content": " self.block_list.append(block)\n in_c = int(c * scale)\n self.out_c = int(1280 * scale) if scale > 1.0 else 1280\n self.conv9 = ConvBNLayer(num_channels=in_c,\n num_filters=self.out_c,\n filter_size=1,\n stride=1,\n padding=0,\n name=prefix_name + \"conv9\")\n self.pool2d_avg = AdaptiveAvgPool2D(1)\n self.out = Linear(self.out_c,\n class_num,\n weight_attr=ParamAttr(name=prefix_name +\n \"fc10_weights\"),\n bias_attr=ParamAttr(name=prefix_name + \"fc10_offset\"))\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:208-232"
+ },
+ "6503": {
+ "file_id": 499,
+ "content": "This code defines a class, appends blocks to block_list, sets output channels based on scale factor, initializes convolution and pooling layers, and defines an initialization function for the weights. It seems to be part of a deep learning model backbone implementation.",
+ "type": "comment"
+ },
+ "6504": {
+ "file_id": 499,
+ "content": " for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n y = self.conv1(inputs, if_act=True)\n for block in self.block_list:\n y = block(y)\n y = self.conv9(y, if_act=True)\n y = self.pool2d_avg(y)\n y = paddle.reshape(y, [-1, self.num_seg, y.shape[1]])\n y = paddle.mean(y, axis=1)\n y = paddle.reshape(y, shape=[-1, self.out_c])\n y = self.out(y)\n return y\n@BACKBONES.register()\ndef PPTSM_MobileNetV2(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=1.0, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x0_25(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=0.25, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x0_5(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=0.5, **kwargs)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:233-266"
+ },
+ "6505": {
+ "file_id": 499,
+ "content": "Initializes a PPTSM MobileNetV2 model with optional pretrained weights and customizable scale. Initializes the underlying MobileNet model and applies custom modifications. Iterates through sublayers, applying Kaiming Normal initialization for Conv2D layers and constant initialization for BatchNorm2D layers. Defines the forward pass of the PPTSM MobileNetV2 model. Registers multiple PPTSM MobileNetV2 variants with different scales.",
+ "type": "comment"
+ },
+ "6506": {
+ "file_id": 499,
+ "content": " return model\ndef PPTSM_MobileNetV2_x0_75(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=0.75, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x1_5(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=1.5, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x2_0(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=2.0, **kwargs)\n return model",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv2.py:267-282"
+ },
+ "6507": {
+ "file_id": 499,
+ "content": "The code defines three functions, PPTSM_MobileNetV2_x0_75, PPTSM_MobileNetV2_x1_5, and PPTSM_MobileNetV2_x2_0. Each function creates a MobileNet model with different scales (0.75, 1.5, and 2.0) using the MobileNet class. The pretrained option allows for loading pre-trained weights if set to True. The functions return the created models.",
+ "type": "comment"
+ },
+ "6508": {
+ "file_id": 500,
+ "content": "/paddlevideo/modeling/backbones/pptsm_mv3.py",
+ "type": "filepath"
+ },
+ "6509": {
+ "file_id": 500,
+ "content": "The code introduces PPTSM-Mv3 backbone networks and MobileNetV3 models in PaddleVideo using PyTorch, with diverse parameters, weight initialization, pretrained model URLs, network configuration dictionaries. It also constructs CNN layers with Batch Normalization and builds the PPTSM-MV3 backbone model using temporal shifting, convolutions, SE modules, and implements Hardsigmoid function separately.",
+ "type": "summary"
+ },
+ "6510": {
+ "file_id": 500,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# reference: https://arxiv.org/abs/1905.02244\nfrom __future__ import absolute_import, division, print_function\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle import ParamAttr\nfrom paddle.nn import AdaptiveAvgPool2D, BatchNorm, Conv2D, Dropout, Linear\nfrom paddle.regularizer import L2Decay\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:1-28"
+ },
+ "6511": {
+ "file_id": 500,
+ "content": "Copyright notice, license information, and reference to the associated research paper. The code imports necessary libraries and registers the backbone model within the PaddleVideo module registry. It also includes a function for weight initialization.",
+ "type": "comment"
+ },
+ "6512": {
+ "file_id": 500,
+ "content": "# Download URL of pretrained model\n# MODEL_URLS = {\n# \"MobileNetV3_small_x1_0\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/MobileNetV3_small_x1_0_ssld_pretrained.pdparams\",\n# \"MobileNetV3_large_x1_0\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/MobileNetV3_large_x1_0_ssld_pretrained.pdparams\",\n# }\nMODEL_STAGES_PATTERN = {\n \"MobileNetV3_small\": [\"blocks[0]\", \"blocks[2]\", \"blocks[7]\", \"blocks[10]\"],\n \"MobileNetV3_large\":\n [\"blocks[0]\", \"blocks[2]\", \"blocks[5]\", \"blocks[11]\", \"blocks[14]\"]\n}\n# \"large\", \"small\" is just for MobinetV3_large, MobileNetV3_small respectively.\n# The type of \"large\" or \"small\" config is a list. Each element(list) represents a depthwise block, which is composed of k, exp, se, act, s.\n# k: kernel_size\n# exp: middle channel number in depthwise block\n# c: output channel number in depthwise block\n# se: whether to use SE block\n# act: which activation to use\n# s: stride in depthwise block\nNET_CONFIG = {",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:30-52"
+ },
+ "6513": {
+ "file_id": 500,
+ "content": "The code defines pretrained model URLs for MobileNetV3_small_x1_0 and MobileNetV3_large_x1_0, as well as lists of stages for each model. The MODEL_STAGES_PATTERN contains different depthwise blocks' parameters such as kernel size, channel numbers, activation function, and stride. NET_CONFIG is a dictionary containing configurations for specific network architectures with different parameters.",
+ "type": "comment"
+ },
+ "6514": {
+ "file_id": 500,
+ "content": " \"large\": [\n # k, exp, c, se, act, s\n [3, 16, 16, False, \"relu\", 1],\n [3, 64, 24, False, \"relu\", 2],\n [3, 72, 24, False, \"relu\", 1],\n [5, 72, 40, True, \"relu\", 2],\n [5, 120, 40, True, \"relu\", 1],\n [5, 120, 40, True, \"relu\", 1],\n [3, 240, 80, False, \"hardswish\", 2],\n [3, 200, 80, False, \"hardswish\", 1],\n [3, 184, 80, False, \"hardswish\", 1],\n [3, 184, 80, False, \"hardswish\", 1],\n [3, 480, 112, True, \"hardswish\", 1],\n [3, 672, 112, True, \"hardswish\", 1],\n [5, 672, 160, True, \"hardswish\", 2],\n [5, 960, 160, True, \"hardswish\", 1],\n [5, 960, 160, True, \"hardswish\", 1],\n ],\n \"small\": [\n # k, exp, c, se, act, s\n [3, 16, 16, True, \"relu\", 2],\n [3, 72, 24, False, \"relu\", 2],\n [3, 88, 24, False, \"relu\", 1],\n [5, 96, 40, True, \"hardswish\", 2],\n [5, 240, 40, True, \"hardswish\", 1],\n [5, 240, 40, True, \"hardswish\", 1],\n [5, 120, 48, True, \"hardswish\", 1],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:53-79"
+ },
+ "6515": {
+ "file_id": 500,
+ "content": "This code defines two versions of the PPTSM-Mv3 backbone network architecture for the PaddleVideo library: \"large\" and \"small\". The backbone is a series of convolutional layers, with different configurations specified by parameters k (kernel size), exp (expansion factor), c (number of channels), se (if using squeeze-and-excitation), act (activation function), and s (strides). The large version has more layers and higher capacities for learning, while the small version is optimized for inference speed. Each layer's configuration is defined in a list of lists.",
+ "type": "comment"
+ },
+ "6516": {
+ "file_id": 500,
+ "content": " [5, 144, 48, True, \"hardswish\", 1],\n [5, 288, 96, True, \"hardswish\", 2],\n [5, 576, 96, True, \"hardswish\", 1],\n [5, 576, 96, True, \"hardswish\", 1],\n ]\n}\n# first conv output channel number in MobileNetV3\nSTEM_CONV_NUMBER = 16\n# last second conv output channel for \"small\"\nLAST_SECOND_CONV_SMALL = 576\n# last second conv output channel for \"large\"\nLAST_SECOND_CONV_LARGE = 960\n# last conv output channel number for \"large\" and \"small\"\nLAST_CONV = 1280\ndef _make_divisible(v, divisor=8, min_value=None):\n if min_value is None:\n min_value = divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\ndef _create_act(act):\n if act == \"hardswish\":\n return nn.Hardswish()\n elif act == \"relu\":\n return nn.ReLU()\n elif act is None:\n return None\n else:\n raise RuntimeError(\n \"The activation function is not supported: {}\".format(act))\nclass MobileNetV3(nn.Layer):\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:80-118"
+ },
+ "6517": {
+ "file_id": 500,
+ "content": "This code defines the MobileNetV3 model with various parameters such as channel numbers, activation functions, and division rules for each layer. The class \"MobileNetV3\" is a custom PyTorch Layer that represents the network architecture, utilizing convolutional layers and activation functions like Hardswish or ReLU. The function \"_make_divisible\" ensures proper alignment of channel numbers with hardware considerations, while \"_create_act\" creates instances of the specified activation functions.",
+ "type": "comment"
+ },
+ "6518": {
+ "file_id": 500,
+ "content": " MobileNetV3\n Args:\n config: list. MobileNetV3 depthwise blocks config.\n scale: float=1.0. The coefficient that controls the size of network parameters.\n class_num: int=1000. The number of classes.\n inplanes: int=16. The output channel number of first convolution layer.\n class_squeeze: int=960. The output channel number of penultimate convolution layer.\n class_expand: int=1280. The output channel number of last convolution layer.\n dropout_prob: float=0.2. Probability of setting units to zero.\n Returns:\n model: nn.Layer. Specific MobileNetV3 model depends on args.\n \"\"\"\n def __init__(self,\n config,\n stages_pattern,\n scale=1.0,\n class_num=400,\n inplanes=STEM_CONV_NUMBER,\n class_squeeze=LAST_SECOND_CONV_LARGE,\n class_expand=LAST_CONV,\n dropout_prob=0.2,\n num_seg=8,\n pretrained=None,\n return_patterns=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:119-142"
+ },
+ "6519": {
+ "file_id": 500,
+ "content": "The function defines a MobileNetV3 model with configurable parameters like depthwise blocks, scale, class number, inplanes, class_squeeze, class_expand, dropout probability, and number of segments. It takes these parameters as inputs and returns the specific MobileNetV3 model based on the arguments provided.",
+ "type": "comment"
+ },
+ "6520": {
+ "file_id": 500,
+ "content": " return_stages=None):\n super().__init__()\n self.cfg = config\n self.scale = scale\n self.inplanes = inplanes\n self.class_squeeze = class_squeeze\n self.class_expand = class_expand\n self.class_num = class_num\n self.num_seg = num_seg\n self.pretrained = pretrained\n self.conv = ConvBNLayer(in_c=3,\n out_c=_make_divisible(self.inplanes *\n self.scale),\n filter_size=3,\n stride=2,\n padding=1,\n num_groups=1,\n if_act=True,\n act=\"hardswish\")\n self.blocks = nn.Sequential(*[\n ResidualUnit(in_c=_make_divisible(self.inplanes * self.scale if i ==\n 0 else self.cfg[i - 1][2] *\n self.scale),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:143-168"
+ },
+ "6521": {
+ "file_id": 500,
+ "content": "This code defines a PPTSM-MV3 backbone model with specified configurations, including input planes, scale factor, class parameters, and number of segments. It uses convolutional layers and residual units for feature extraction and processing.",
+ "type": "comment"
+ },
+ "6522": {
+ "file_id": 500,
+ "content": " mid_c=_make_divisible(self.scale * exp),\n out_c=_make_divisible(self.scale * c),\n filter_size=k,\n stride=s,\n use_se=se,\n num_seg=self.num_seg,\n act=act)\n for i, (k, exp, c, se, act, s) in enumerate(self.cfg)\n ])\n self.last_second_conv = ConvBNLayer(\n in_c=_make_divisible(self.cfg[-1][2] * self.scale),\n out_c=_make_divisible(self.scale * self.class_squeeze),\n filter_size=1,\n stride=1,\n padding=0,\n num_groups=1,\n if_act=True,\n act=\"hardswish\")\n self.avg_pool = AdaptiveAvgPool2D(1)\n self.last_conv = Conv2D(in_channels=_make_divisible(self.scale *\n self.class_squeeze),\n out_channels=self.class_expand,\n kernel_size=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:169-194"
+ },
+ "6523": {
+ "file_id": 500,
+ "content": "The code initializes a PPTSM-MV3 model, which consists of several convolutional blocks and a final classification layer. The convolutional blocks are defined by the `self.cfg` list, where each element contains the kernel size, expansion factor, output channels, whether to use SE module, and activation function, along with the stride. The last convolutional block is followed by an average pooling layer and a final convolution layer for classification.",
+ "type": "comment"
+ },
+ "6524": {
+ "file_id": 500,
+ "content": " stride=1,\n padding=0,\n bias_attr=False)\n self.hardswish = nn.Hardswish()\n if dropout_prob is not None:\n self.dropout = Dropout(p=dropout_prob, mode=\"downscale_in_infer\")\n else:\n self.dropout = None\n self.fc = Linear(self.class_expand, class_num)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n #XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, x):\n x = self.conv(x)\n x = self.blocks(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:195-222"
+ },
+ "6525": {
+ "file_id": 500,
+ "content": "This code defines a neural network model for the PPTSM_MV3 backbone. It includes convolutional layers, blocks, a Hardswish activation function, optional dropout, and fully connected layers for classification. The `init_weights` method initializes the network's weights, and the `forward` method passes input through the model layers to generate output.",
+ "type": "comment"
+ },
+ "6526": {
+ "file_id": 500,
+ "content": " x = self.last_second_conv(x)\n x = self.avg_pool(x)\n x = self.last_conv(x)\n x = self.hardswish(x)\n if self.dropout is not None:\n x = self.dropout(x)\n # feature aggregation for video\n x = paddle.reshape(x, [-1, self.num_seg, x.shape[1]])\n x = paddle.mean(x, axis=1)\n x = paddle.reshape(x, shape=[-1, self.class_expand])\n x = self.fc(x)\n return x\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n in_c,\n out_c,\n filter_size,\n stride,\n padding,\n num_groups=1,\n if_act=True,\n act=None):\n super().__init__()\n self.conv = Conv2D(in_channels=in_c,\n out_channels=out_c,\n kernel_size=filter_size,\n stride=stride,\n padding=padding,\n groups=num_groups,\n bias_attr=False)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:223-258"
+ },
+ "6527": {
+ "file_id": 500,
+ "content": "This code defines a ConvBNLayer class that takes input and output channels, filter size, stride, padding, number of groups, activation function flag, and activation type as parameters. It initializes the layers for convolutional neural network and applies Batch Normalization and activation functions if specified. The class also returns the last layer of the model after feature aggregation for video classification.",
+ "type": "comment"
+ },
+ "6528": {
+ "file_id": 500,
+ "content": " self.bn = BatchNorm(num_channels=out_c,\n act=None,\n param_attr=ParamAttr(regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(regularizer=L2Decay(0.0)))\n self.if_act = if_act\n self.act = _create_act(act)\n def forward(self, x):\n x = self.conv(x)\n x = self.bn(x)\n if self.if_act:\n x = self.act(x)\n return x\nclass ResidualUnit(nn.Layer):\n def __init__(self,\n in_c,\n mid_c,\n out_c,\n filter_size,\n stride,\n use_se,\n num_seg=8,\n act=None):\n super().__init__()\n self.if_shortcut = stride == 1 and in_c == out_c\n self.if_se = use_se\n self.num_seg = num_seg\n self.expand_conv = ConvBNLayer(in_c=in_c,\n out_c=mid_c,\n filter_size=1,\n stride=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:259-292"
+ },
+ "6529": {
+ "file_id": 500,
+ "content": "The code defines a ResidualUnit class with an expand_conv layer containing ConvBNLayer, used for building the residual unit in PPTSM-MV3 model. It also includes optional BatchNorm (bn) and activation (act) layers based on provided parameters.",
+ "type": "comment"
+ },
+ "6530": {
+ "file_id": 500,
+ "content": " padding=0,\n if_act=True,\n act=act)\n self.bottleneck_conv = ConvBNLayer(in_c=mid_c,\n out_c=mid_c,\n filter_size=filter_size,\n stride=stride,\n padding=int((filter_size - 1) // 2),\n num_groups=mid_c,\n if_act=True,\n act=act)\n if self.if_se:\n self.mid_se = SEModule(mid_c)\n self.linear_conv = ConvBNLayer(in_c=mid_c,\n out_c=out_c,\n filter_size=1,\n stride=1,\n padding=0,\n if_act=False,\n act=None)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:293-312"
+ },
+ "6531": {
+ "file_id": 500,
+ "content": "Defines a PPTSM_MV3 block with ConvBNLayer, bottleneck convolution layer, optional SEModule for spatial attention, and a linear convolution layer.",
+ "type": "comment"
+ },
+ "6532": {
+ "file_id": 500,
+ "content": " def forward(self, x):\n identity = x\n if self.if_shortcut:\n x = F.temporal_shift(x, self.num_seg, 1.0 / self.num_seg)\n x = self.expand_conv(x)\n x = self.bottleneck_conv(x)\n if self.if_se:\n x = self.mid_se(x)\n x = self.linear_conv(x)\n if self.if_shortcut:\n x = paddle.add(identity, x)\n return x\n# nn.Hardsigmoid can't transfer \"slope\" and \"offset\" in nn.functional.hardsigmoid\nclass Hardsigmoid(nn.Layer):\n def __init__(self, slope=0.2, offset=0.5):\n super().__init__()\n self.slope = slope\n self.offset = offset\n def forward(self, x):\n return nn.functional.hardsigmoid(x,\n slope=self.slope,\n offset=self.offset)\nclass SEModule(nn.Layer):\n def __init__(self, channel, reduction=4):\n super().__init__()\n self.avg_pool = AdaptiveAvgPool2D(1)\n self.conv1 = Conv2D(in_channels=channel,\n out_channels=channel // reduction,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:314-348"
+ },
+ "6533": {
+ "file_id": 500,
+ "content": "This code defines a PPTSM-MV3 backbone model for video analysis. It uses temporal shifting, convolutions, and SE module (if specified) in its forward pass. The Hardsigmoid function is implemented as a separate class to apply hard sigmoid activation with customizable slope and offset parameters.",
+ "type": "comment"
+ },
+ "6534": {
+ "file_id": 500,
+ "content": " kernel_size=1,\n stride=1,\n padding=0)\n self.relu = nn.ReLU()\n self.conv2 = Conv2D(in_channels=channel // reduction,\n out_channels=channel,\n kernel_size=1,\n stride=1,\n padding=0)\n self.hardsigmoid = Hardsigmoid(slope=0.2, offset=0.5)\n def forward(self, x):\n identity = x\n x = self.avg_pool(x)\n x = self.conv1(x)\n x = self.relu(x)\n x = self.conv2(x)\n x = self.hardsigmoid(x)\n return paddle.multiply(x=identity, y=x)\ndef PPTSM_MobileNetV3_small_x1_0(pretrained=None, **kwargs):\n \"\"\"\n MobileNetV3_small_x1_0\n Args:\n pretrained: bool=False or str. If `True` load pretrained parameters, `False` otherwise.\n If str, means the path of the pretrained model.\n use_ssld: bool=False. Whether using distillation pretrained model when pretrained=True.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:349-376"
+ },
+ "6535": {
+ "file_id": 500,
+ "content": "The code defines a Convolutional Neural Network layer for the PPTSM-MobileNetV3_small_x1_0 model. It consists of an average pooling layer, two 1x1 convolution layers with ReLU and hard sigmoid activations. The forward function performs element-wise multiplication between input and output to implement residual learning.",
+ "type": "comment"
+ },
+ "6536": {
+ "file_id": 500,
+ "content": " Returns:\n model: nn.Layer. Specific `MobileNetV3_small_x1_0` model depends on args.\n \"\"\"\n model = MobileNetV3(\n config=NET_CONFIG[\"small\"],\n scale=1.0,\n stages_pattern=MODEL_STAGES_PATTERN[\"MobileNetV3_small\"],\n class_squeeze=LAST_SECOND_CONV_SMALL,\n pretrained=pretrained,\n **kwargs)\n return model\n@BACKBONES.register()\ndef PPTSM_MobileNetV3(pretrained=None, **kwargs):\n \"\"\"\n MobileNetV3_large_x1_0\n Args:\n pretrained: bool=False or str. If `True` load pretrained parameters, `False` otherwise.\n If str, means the path of the pretrained model.\n use_ssld: bool=False. Whether using distillation pretrained model when pretrained=True.\n Returns:\n model: nn.Layer. Specific `MobileNetV3_large_x1_0` model depends on args.\n \"\"\"\n model = MobileNetV3(\n config=NET_CONFIG[\"large\"],\n scale=1.0,\n stages_pattern=MODEL_STAGES_PATTERN[\"MobileNetV3_large\"],\n class_squeeze=LAST_SECOND_CONV_LARGE,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:377-405"
+ },
+ "6537": {
+ "file_id": 500,
+ "content": "This code defines a function that returns specific MobileNetV3 models based on given arguments. The \"MobileNetV3\" class is used to create the models, and parameters such as config, scale, stages_pattern, class_squeeze, pretrained, and other optional keyword arguments are passed to the constructor of the class. The function is then registered with BACKBONES for future use.",
+ "type": "comment"
+ },
+ "6538": {
+ "file_id": 500,
+ "content": " pretrained=pretrained,\n **kwargs)\n return model",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_mv3.py:406-408"
+ },
+ "6539": {
+ "file_id": 500,
+ "content": "This code is creating an instance of the PPTSM-MV3 backbone model with specified pretrained weights and returning it.",
+ "type": "comment"
+ },
+ "6540": {
+ "file_id": 501,
+ "content": "/paddlevideo/modeling/backbones/pptsm_v2.py",
+ "type": "filepath"
+ },
+ "6541": {
+ "file_id": 501,
+ "content": "This Python module provides video processing layers and functions, including Depthwise Separable Convolution layers initialization, PPTSMV2 model with convolutional layers, and batch normalization. It defines a PPTSM_v2 backbone model for video analysis with customizable options like pretrained models, scaling, depths, dropout probability, and additional arguments.",
+ "type": "summary"
+ },
+ "6542": {
+ "file_id": 501,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import absolute_import, division, print_function\nimport paddle\nimport paddle.nn as nn\nfrom paddle import ParamAttr\nfrom paddle.nn import AdaptiveAvgPool2D, BatchNorm, Conv2D, Dropout, Linear, BatchNorm2D\nfrom paddle.regularizer import L2Decay\nfrom paddle.nn.initializer import KaimingNormal\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:1-27"
+ },
+ "6543": {
+ "file_id": 501,
+ "content": "This code is a Python module for the PaddlePaddle framework. It contains definitions of various layers and functions used in neural network backbones, including convolutional layers, pooling layers, batch normalization, linear layers, and more. The code also includes comments about copyright and licensing information, as well as imports necessary modules for these operations. Additionally, it references utility functions for weight initialization and model loading from checkpoints.",
+ "type": "comment"
+ },
+ "6544": {
+ "file_id": 501,
+ "content": "# MODEL_URLS = {\n# \"PPLCNetV2\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/PPLCNetV2_base_ssld_pretrained.pdparams\",\n# }\nMODEL_STAGES_PATTERN = {\n \"PPLCNet\": [\"blocks2\", \"blocks3\", \"blocks4\", \"blocks5\", \"blocks6\"]\n}\nNET_CONFIG = {\n # in_channels, kernel_size, split_pw, use_rep, use_se, use_shortcut\n \"stage1\": [64, 3, False, False, False, False],\n \"stage2\": [128, 3, False, False, False, False],\n \"stage3\": [256, 5, True, True, True, False],\n \"stage4\": [512, 5, False, True, False, True],\n}\ndef make_divisible(v, divisor=8, min_value=None):\n if min_value is None:\n min_value = divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\nclass GlobalAttention(nn.Layer):\n \"\"\"\n Lightweight temporal attention module.\n \"\"\"\n def __init__(self, num_seg=8):\n super().__init__()\n self.fc = nn.Linear(in_features=num_seg,\n out_features=num_seg,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:29-64"
+ },
+ "6545": {
+ "file_id": 501,
+ "content": "This code defines the PPLCNetV2 backbone model for video processing tasks. It includes the URL to download a pretrained model, stages of the network (PPLCNet), and network configurations. The make_divisible function is used to round up numbers for better performance. The GlobalAttention class is a lightweight temporal attention module used in the model.",
+ "type": "comment"
+ },
+ "6546": {
+ "file_id": 501,
+ "content": " weight_attr=ParamAttr(learning_rate=5.0,\n regularizer=L2Decay(1e-4)),\n bias_attr=ParamAttr(learning_rate=10.0,\n regularizer=L2Decay(0.0)))\n self.num_seg = num_seg\n def forward(self, x):\n _, C, H, W = x.shape\n x0 = x\n x = x.reshape([-1, self.num_seg, C * H * W])\n x = paddle.mean(x, axis=2) # efficient way of avg_pool\n x = x.squeeze(axis=-1)\n x = self.fc(x)\n attention = F.sigmoid(x)\n attention = attention.reshape(\n (-1, self.num_seg, 1, 1, 1)) #for broadcast\n x0 = x0.reshape([-1, self.num_seg, C, H, W])\n y = paddle.multiply(x0, attention)\n y = y.reshape_([-1, C, H, W])\n return y\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride,\n groups=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:65-96"
+ },
+ "6547": {
+ "file_id": 501,
+ "content": "The code defines a ConvBNLayer class and a PPTSMV2 class. The ConvBNLayer class is a convolution layer followed by batch normalization, and the PPTSMV2 class is an encoder model that takes input of shape (-1, 3, H, W) where H and W are height and width respectively, and returns output of the same shape after processing. It first resizes the input, applies convolution with specified parameters, calculates attention maps, and performs element-wise multiplication between original input and attention maps to extract relevant features for each segmented region.",
+ "type": "comment"
+ },
+ "6548": {
+ "file_id": 501,
+ "content": " use_act=True):\n super().__init__()\n self.use_act = use_act\n self.conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(initializer=KaimingNormal()),\n bias_attr=False)\n self.bn = BatchNorm2D(out_channels,\n weight_attr=ParamAttr(regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(regularizer=L2Decay(0.0)))\n if self.use_act:\n self.act = nn.ReLU()\n def forward(self, x):\n x = self.conv(x)\n x = self.bn(x)\n if self.use_act:\n x = self.act(x)\n return x\nclass SEModule(nn.Layer):\n def __init__(self, channel, reduction=4):\n super().__init__()\n self.avg_pool = AdaptiveAvgPool2D(1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:97-127"
+ },
+ "6549": {
+ "file_id": 501,
+ "content": "The code defines a Conv2D layer followed by a BatchNorm2D layer and an optional ReLU activation function. The SEModule class inherits from nn.Layer and contains an AdaptiveAvgPool2D layer for average pooling operations.",
+ "type": "comment"
+ },
+ "6550": {
+ "file_id": 501,
+ "content": " self.conv1 = Conv2D(in_channels=channel,\n out_channels=channel // reduction,\n kernel_size=1,\n stride=1,\n padding=0)\n self.relu = nn.ReLU()\n self.conv2 = Conv2D(in_channels=channel // reduction,\n out_channels=channel,\n kernel_size=1,\n stride=1,\n padding=0)\n self.hardsigmoid = nn.Sigmoid()\n def forward(self, x):\n identity = x\n x = self.avg_pool(x)\n x = self.conv1(x)\n x = self.relu(x)\n x = self.conv2(x)\n x = self.hardsigmoid(x)\n x = paddle.multiply(x=identity, y=x)\n return x\nclass RepDepthwiseSeparable(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n dw_size=3,\n split_pw=False,\n use_rep=False,\n use_se=False,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:128-161"
+ },
+ "6551": {
+ "file_id": 501,
+ "content": "This code initializes a depthwise separable convolution layer with optional parameters like in_channels, out_channels, stride, dw_size, split_pw, use_rep, and use_se. It contains Conv2D layers for convolution operations, ReLU activation, Sigmoid activation, and element-wise multiplication for identity shortcut connection.",
+ "type": "comment"
+ },
+ "6552": {
+ "file_id": 501,
+ "content": " use_shortcut=False):\n super().__init__()\n self.is_repped = False\n self.dw_size = dw_size\n self.split_pw = split_pw\n self.use_rep = use_rep\n self.use_se = use_se\n self.use_shortcut = True if use_shortcut and stride == 1 and in_channels == out_channels else False\n if self.use_rep:\n self.dw_conv_list = nn.LayerList()\n for kernel_size in range(self.dw_size, 0, -2):\n if kernel_size == 1 and stride != 1:\n continue\n dw_conv = ConvBNLayer(in_channels=in_channels,\n out_channels=in_channels,\n kernel_size=kernel_size,\n stride=stride,\n groups=in_channels,\n use_act=False)\n self.dw_conv_list.append(dw_conv)\n self.dw_conv = nn.Conv2D(in_channels=in_channels,\n out_channels=in_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:162-185"
+ },
+ "6553": {
+ "file_id": 501,
+ "content": "This code initializes a PPTSM backbone model. It creates a ConvBNLayer for each kernel size in the dw_size range, skipping 1x1 if stride is not 1. The layers are stored in the dw_conv_list. An additional Conv2D layer with the same number of input and output channels is also created. This model can be used for image classification tasks.",
+ "type": "comment"
+ },
+ "6554": {
+ "file_id": 501,
+ "content": " kernel_size=dw_size,\n stride=stride,\n padding=(dw_size - 1) // 2,\n groups=in_channels)\n else:\n self.dw_conv = ConvBNLayer(in_channels=in_channels,\n out_channels=in_channels,\n kernel_size=dw_size,\n stride=stride,\n groups=in_channels)\n self.act = nn.ReLU()\n if use_se:\n self.se = SEModule(in_channels)\n if self.split_pw:\n pw_ratio = 0.5\n self.pw_conv_1 = ConvBNLayer(in_channels=in_channels,\n kernel_size=1,\n out_channels=int(out_channels *\n pw_ratio),\n stride=1)\n self.pw_conv_2 = ConvBNLayer(in_channels=int(out_channels *",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:186-209"
+ },
+ "6555": {
+ "file_id": 501,
+ "content": "Code creates a ConvBNLayer object for downsample convolution with optional SE module and split point-wise convolution. It handles different configurations based on dw_size, stride, and use_se parameters.",
+ "type": "comment"
+ },
+ "6556": {
+ "file_id": 501,
+ "content": " pw_ratio),\n kernel_size=1,\n out_channels=out_channels,\n stride=1)\n else:\n self.pw_conv = ConvBNLayer(in_channels=in_channels,\n kernel_size=1,\n out_channels=out_channels,\n stride=1)\n def forward(self, x):\n if self.use_rep:\n input_x = x\n if self.is_repped:\n x = self.act(self.dw_conv(x))\n else:\n y = self.dw_conv_list[0](x)\n for dw_conv in self.dw_conv_list[1:]:\n y += dw_conv(x)\n x = self.act(y)\n else:\n x = self.dw_conv(x)\n if self.use_se:\n x = self.se(x)\n if self.split_pw:\n x = self.pw_conv_1(x)\n x = self.pw_conv_2(x)\n else:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:210-238"
+ },
+ "6557": {
+ "file_id": 501,
+ "content": "This code defines a backbone for a deep learning model, specifically the PPTSM_v2 architecture. It uses convolutional layers and Batch Normalization to process input data. The use of Point-wise Convolution (pw_conv) or Depth-wise Separable Convolutions (dw_conv) depends on certain conditions. If \"use_rep\" is True, it applies repeated depth-wise convolutions if the current layer has been repped. It also includes optional Squeeze and Excitation blocks for feature enhancement.",
+ "type": "comment"
+ },
+ "6558": {
+ "file_id": 501,
+ "content": " x = self.pw_conv(x)\n if self.use_shortcut:\n x = x + input_x\n return x\n def rep(self):\n if self.use_rep:\n self.is_repped = True\n kernel, bias = self._get_equivalent_kernel_bias()\n self.dw_conv.weight.set_value(kernel)\n self.dw_conv.bias.set_value(bias)\n def _get_equivalent_kernel_bias(self):\n kernel_sum = 0\n bias_sum = 0\n for dw_conv in self.dw_conv_list:\n kernel, bias = self._fuse_bn_tensor(dw_conv)\n kernel = self._pad_tensor(kernel, to_size=self.dw_size)\n kernel_sum += kernel\n bias_sum += bias\n return kernel_sum, bias_sum\n def _fuse_bn_tensor(self, branch):\n kernel = branch.conv.weight\n running_mean = branch.bn._mean\n running_var = branch.bn._variance\n gamma = branch.bn.weight\n beta = branch.bn.bias\n eps = branch.bn._epsilon\n std = (running_var + eps).sqrt()\n t = (gamma / std).reshape((-1, 1, 1, 1))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:239-269"
+ },
+ "6559": {
+ "file_id": 501,
+ "content": "This code implements a backbone for PPTSM_V2 model. It performs pointwise convolution, adds shortcut connection if enabled, and includes functions for representation fusion and fusing batch normalization tensor.",
+ "type": "comment"
+ },
+ "6560": {
+ "file_id": 501,
+ "content": " return kernel * t, beta - running_mean * gamma / std\n def _pad_tensor(self, tensor, to_size):\n from_size = tensor.shape[-1]\n if from_size == to_size:\n return tensor\n pad = (to_size - from_size) // 2\n return F.pad(tensor, [pad, pad, pad, pad])\nclass PPTSM_v2_LCNet(nn.Layer):\n def __init__(self,\n scale,\n depths,\n class_num=400,\n dropout_prob=0,\n num_seg=8,\n use_temporal_att=False,\n pretrained=None,\n use_last_conv=True,\n class_expand=1280):\n super().__init__()\n self.scale = scale\n self.use_last_conv = use_last_conv\n self.class_expand = class_expand\n self.num_seg = num_seg\n self.use_temporal_att = use_temporal_att\n self.pretrained = pretrained\n self.stem = nn.Sequential(*[\n ConvBNLayer(in_channels=3,\n kernel_size=3,\n out_channels=make_divisible(32 * scale),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:270-303"
+ },
+ "6561": {
+ "file_id": 501,
+ "content": "The code defines a PPTSM_v2_LCNet class, which is a type of backbone neural network. It includes initialization parameters such as scale, depths, and class_num. The class also has methods for kernel multiplication, tensor padding, and other operations related to image processing and neural network layers.",
+ "type": "comment"
+ },
+ "6562": {
+ "file_id": 501,
+ "content": " stride=2),\n RepDepthwiseSeparable(in_channels=make_divisible(32 * scale),\n out_channels=make_divisible(64 * scale),\n stride=1,\n dw_size=3)\n ])\n # stages\n self.stages = nn.LayerList()\n for depth_idx, k in enumerate(NET_CONFIG):\n in_channels, kernel_size, split_pw, use_rep, use_se, use_shortcut = NET_CONFIG[\n k]\n self.stages.append(\n nn.Sequential(*[\n RepDepthwiseSeparable(in_channels=make_divisible(\n (in_channels if i == 0 else in_channels * 2) * scale),\n out_channels=make_divisible(\n in_channels * 2 * scale),\n stride=2 if i == 0 else 1,\n dw_size=kernel_size,\n split_pw=split_pw,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:304-324"
+ },
+ "6563": {
+ "file_id": 501,
+ "content": "This code defines a PPTSM-v2 backbone model, using DepthwiseSeparable blocks with varying configurations for different stages. It utilizes `make_divisible()` function to adjust the number of channels and kernel sizes, and a LayerList to create a sequence of layers for each stage. The NET_CONFIG determines the specifics of each stage's parameters like in_channels, kernel_size, split_pw, use_rep, use_se, and use_shortcut.",
+ "type": "comment"
+ },
+ "6564": {
+ "file_id": 501,
+ "content": " use_rep=use_rep,\n use_se=use_se,\n use_shortcut=use_shortcut)\n for i in range(depths[depth_idx])\n ]))\n self.avg_pool = AdaptiveAvgPool2D(1)\n if self.use_last_conv:\n self.last_conv = Conv2D(in_channels=make_divisible(\n NET_CONFIG[\"stage4\"][0] * 2 * scale),\n out_channels=self.class_expand,\n kernel_size=1,\n stride=1,\n padding=0,\n bias_attr=False)\n self.act = nn.ReLU()\n self.dropout = Dropout(p=dropout_prob, mode=\"downscale_in_infer\")\n self.flatten = nn.Flatten(start_axis=1, stop_axis=-1)\n in_features = self.class_expand if self.use_last_conv else NET_CONFIG[\n \"stage4\"][0] * 2 * scale\n self.fc = Linear(in_features, class_num)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:325-347"
+ },
+ "6565": {
+ "file_id": 501,
+ "content": "This code defines a PPTSM_V2 backbone for the PaddleVideo model. It includes multiple Conv2D layers, BatchNorm layers, AdaptiveAvgPool2D, and optional final convolutional layer, flatten layer, linear layer. The use of these components depends on certain conditions such as `use_rep`, `use_se`, `use_shortcut`, `use_last_conv`, and other parameters.",
+ "type": "comment"
+ },
+ "6566": {
+ "file_id": 501,
+ "content": " if self.use_temporal_att:\n self.global_attention = GlobalAttention(num_seg=self.num_seg)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, x):\n x = self.stem(x)\n count = 0\n for stage in self.stages:\n # only add temporal attention and tsm in stage3 for efficiency\n if count == 2:\n # add temporal attention\n if self.use_temporal_att:\n x = self.global_attention(x)\n x = F.temporal_shift(x, self.num_seg, 1.0 / self.num_seg)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:348-372"
+ },
+ "6567": {
+ "file_id": 501,
+ "content": "Code initializes weights for a PPTSM_v2 backbone model. It first checks if the pretrained weights are provided and then initializes the layers with specified methods. Stage 3 adds temporal attention and Temporal Shift Module (TSM) operations for efficiency.",
+ "type": "comment"
+ },
+ "6568": {
+ "file_id": 501,
+ "content": " count += 1\n x = stage(x)\n x = self.avg_pool(x)\n if self.use_last_conv:\n x = self.last_conv(x)\n x = self.act(x)\n x = self.dropout(x)\n # Feature aggregation\n x = paddle.reshape(x, [-1, self.num_seg, x.shape[1]])\n x = paddle.mean(x, axis=1)\n x = paddle.reshape(x, shape=[-1, self.class_expand])\n x = self.fc(x)\n return x\n@BACKBONES.register()\ndef PPTSM_v2(pretrained=None, use_ssld=False, **kwargs):\n \"\"\"\n PP-TSM_v2 model.\n Args:\n pretrained: str, means the path of the pretrained model.\n Returns:\n model: nn.Layer.\n \"\"\"\n model = PPTSM_v2_LCNet(pretrained=pretrained,\n scale=1.0,\n depths=[2, 2, 6, 2],\n dropout_prob=0.2,\n **kwargs)\n return model",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/pptsm_v2.py:373-405"
+ },
+ "6569": {
+ "file_id": 501,
+ "content": "Code snippet defines a PPTSM_v2 backbone model for video analysis. It consists of stages, an average pooling layer, and a convolution layer. The function also includes feature aggregation and reshaping operations before feeding the data to a fully connected layer. The pretrained model can be loaded from a given path, and it supports custom scaling, depths, dropout probability, and additional keyword arguments.",
+ "type": "comment"
+ },
+ "6570": {
+ "file_id": 502,
+ "content": "/paddlevideo/modeling/backbones/resnet.py",
+ "type": "filepath"
+ },
+ "6571": {
+ "file_id": 502,
+ "content": "This code defines a ResNet backbone model, utilizing ConvBNLayer and ReLU activation. It can dynamically add bottleneck blocks for models like ResNet-101 and ResNet-152, includes forward function and supports pretrained models.",
+ "type": "summary"
+ },
+ "6572": {
+ "file_id": 502,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport math\nimport paddle\nimport paddle.nn as nn\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout, MaxPool2D,\n AvgPool2D)\nfrom paddle import ParamAttr\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt\nclass ConvBNLayer(nn.Layer):\n \"\"\"Conv2D and BatchNorm2D layer.\n Args:\n in_channels (int): Number of channels for the input.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:1-34"
+ },
+ "6573": {
+ "file_id": 502,
+ "content": "This code is importing necessary libraries and defining the ConvBNLayer class, which combines a convolutional layer with batch normalization. This class takes in the number of input channels as an argument. The copyright notice at the beginning indicates this code is licensed under the Apache License 2.0.",
+ "type": "comment"
+ },
+ "6574": {
+ "file_id": 502,
+ "content": " out_channels (int): Number of channels for the output.\n kernel_size (int): Kernel size.\n stride (int): Stride in the Conv2D layer. Default: 1.\n groups (int): Groups in the Conv2D, Default: 1.\n act (str): Indicate activation after BatchNorm2D layer.\n name (str): the name of an instance of ConvBNLayer.\n Note: weight and bias initialization include initialize values and name the restored parameters, values initialization are explicit declared in the ```init_weights``` method.\n \"\"\"\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n act=None,\n name=None):\n super(ConvBNLayer, self).__init__()\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:35-58"
+ },
+ "6575": {
+ "file_id": 502,
+ "content": "The ConvBNLayer class is a custom layer that initializes the Conv2D layer with BatchNorm2D and optional activation function. It takes in_channels, out_channels, kernel_size, stride (default: 1), groups (default: 1), act (optional activation function), and name as parameters. Weight and bias initialization are defined in the init_weights method.",
+ "type": "comment"
+ },
+ "6576": {
+ "file_id": 502,
+ "content": " groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._act = act\n self._batch_norm = BatchNorm2D(out_channels,\n weight_attr=ParamAttr(name=bn_name +\n \"_scale\"),\n bias_attr=ParamAttr(bn_name + \"_offset\"))\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if self._act:\n y = getattr(paddle.nn.functional, self._act)(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n name=None):\n super(BottleneckBlock, self).__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:59-89"
+ },
+ "6577": {
+ "file_id": 502,
+ "content": "ResNet module with batch normalization and optional activation function. BottleneckBlock class for ResNet blocks.",
+ "type": "comment"
+ },
+ "6578": {
+ "file_id": 502,
+ "content": " self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act=\"relu\",\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act=\"relu\",\n name=name + \"_branch2b\")\n self.conv2 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n act=None,\n name=name + \"_branch2c\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:90-111"
+ },
+ "6579": {
+ "file_id": 502,
+ "content": "This code defines the ResNet backbone structure. It creates three ConvBNLayer instances for the first block of the network, with different parameters for each layer. The `self.conv0` layer has 1x1 kernel and performs a relu activation. `self.conv1` has a 3x3 kernel and also applies a relu activation after a stride operation. Lastly, `self.conv2` has a 1x1 kernel, no activation function, and increases the number of output channels by 4 times. The shortcut connection is created if `shortcut` is not set to `True`.",
+ "type": "comment"
+ },
+ "6580": {
+ "file_id": 502,
+ "content": " stride=stride,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n return F.relu(y)\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n name=None):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n filter_size=3,\n stride=stride,\n act=\"relu\",\n name=name + \"_branch2a\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:112-143"
+ },
+ "6581": {
+ "file_id": 502,
+ "content": "The code defines a ResNet block with optional shortcut connection, containing ConvBNLayer and ReLU activation. The BasicBlock class initializes the parameters for the ResNet block including stride, number of channels, convolution layer, and optional shortcut.",
+ "type": "comment"
+ },
+ "6582": {
+ "file_id": 502,
+ "content": " self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n filter_size=3,\n act=None,\n name=name + \"_branch2b\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n filter_size=1,\n stride=stride,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(short, conv1)\n y = F.relu(y)\n return y\n@BACKBONES.register()\nclass ResNet(nn.Layer):\n \"\"\"ResNet backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:144-178"
+ },
+ "6583": {
+ "file_id": 502,
+ "content": "This code defines a ResNet backbone model. It includes a ConvBNLayer for feature extraction and optionally applies shortcut connections based on the input and output channel count. The forward function performs addition, followed by ReLU activation for each input. The ResNet class is registered with BACKBONES and takes arguments for depth and pretrained model.",
+ "type": "comment"
+ },
+ "6584": {
+ "file_id": 502,
+ "content": " \"\"\"\n def __init__(self, depth, pretrained=None):\n super(ResNet, self).__init__()\n self.pretrained = pretrained\n self.layers = depth\n supported_layers = [18, 34, 50, 101, 152]\n assert self.layers in supported_layers, \\\n \"supported layers are {} but input layer is {}\".format(\n supported_layers, self.layers)\n if self.layers == 18:\n depth = [2, 2, 2, 2]\n elif self.layers == 34 or self.layers == 50:\n depth = [3, 4, 6, 3]\n elif self.layers == 101:\n depth = [3, 4, 23, 3]\n elif self.layers == 152:\n depth = [3, 8, 36, 3]\n in_channels = [64, 256, 512, 1024]\n out_channels = [64, 128, 256, 512]\n self.conv = ConvBNLayer(in_channels=3,\n out_channels=64,\n kernel_size=7,\n stride=2,\n act=\"relu\",\n name=\"conv1\")\n self.pool2D_max = MaxPool2D(kernel_size=3, stride=2, padding=1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:179-208"
+ },
+ "6585": {
+ "file_id": 502,
+ "content": "This code defines a ResNet class with different layers and their corresponding depths. It also initializes the ConvBNLayer for the first convolution operation and MaxPool2D layer for pooling. The supported layers are 18, 34, 50, 101, and 152.",
+ "type": "comment"
+ },
+ "6586": {
+ "file_id": 502,
+ "content": " self.block_list = []\n if self.layers >= 50:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n if self.layers in [101, 152] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n conv_name,\n BottleneckBlock(\n # NOTE: Be careful! Here is different from TSM model.\n in_channels=in_channels[block]\n if i == 0 else out_channels[block] * 4,\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:210-229"
+ },
+ "6587": {
+ "file_id": 502,
+ "content": "This code adds bottleneck blocks to a ResNet backbone model, dynamically creating sublayers based on the input parameters. The block type and number of layers are determined by the given depth configuration. It also handles specific cases for ResNet-101 and ResNet-152 models.",
+ "type": "comment"
+ },
+ "6588": {
+ "file_id": 502,
+ "content": " shortcut=shortcut,\n name=conv_name))\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n conv_name,\n BasicBlock(in_channels=in_channels[block]\n if i == 0 else out_channels[block],\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n name=conv_name))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):\n \"\"\"Initiate the parameters.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:230-252"
+ },
+ "6589": {
+ "file_id": 502,
+ "content": "The code is defining a ResNet model by creating layers and blocks based on the given depth configuration. It alternates between BottleneckBlock and BasicBlock depending on the current block number and depth. It also initializes weights for the parameters in the model.",
+ "type": "comment"
+ },
+ "6590": {
+ "file_id": 502,
+ "content": " Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n #XXX: check bias!!! check pretrained!!!\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n #XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:253-268"
+ },
+ "6591": {
+ "file_id": 502,
+ "content": "If a pretrained loading path is specified, the code will load the model with that path. If no pretrained path is provided or it's set to an empty string, it initializes Conv2D layers with KaimingNormal function and BatchNorm2D layers with Constant function (value=1).",
+ "type": "comment"
+ },
+ "6592": {
+ "file_id": 502,
+ "content": " def forward(self, inputs):\n \"\"\"Define how the backbone is going to run.\n \"\"\"\n #NOTE: Already merge axis 0(batches) and axis 1(channels) before extracting feature phase,\n # please refer to paddlevideo/modeling/framework/recognizers/recognizer2d.py#L27\n #y = paddle.reshape(\n # inputs, [-1, inputs.shape[2], inputs.shape[3], inputs.shape[4]])\n y = self.conv(inputs)\n y = self.pool2D_max(y)\n for block in self.block_list:\n y = block(y)\n return y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet.py:270-283"
+ },
+ "6593": {
+ "file_id": 502,
+ "content": "This code defines the forward function for a ResNet backbone. It reshapes and passes the input through a convolutional layer, max pooling, and a series of blocks. The output is returned after processing all blocks.",
+ "type": "comment"
+ },
+ "6594": {
+ "file_id": 503,
+ "content": "/paddlevideo/modeling/backbones/resnet3d.py",
+ "type": "filepath"
+ },
+ "6595": {
+ "file_id": 503,
+ "content": "The code introduces a simplified 3D ResNet model in PaddleVideo, allowing for configurable parameters and options for non-local blocks and dilation values. The model is initialized with inflated 2D params, constructs layers, and can utilize pretrained weights.",
+ "type": "summary"
+ },
+ "6596": {
+ "file_id": 503,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport warnings\nimport collections\nfrom itertools import repeat\nimport paddle\nfrom paddle import nn\ndef _ntuple(n):\n def parse(x):\n if isinstance(x, collections.abc.Iterable):\n return tuple(x)\n return tuple(repeat(x, n))\n return parse\n_triple = _ntuple(3)\nclass ConvBNLayer(nn.Layer):\n \"\"\"A conv block that bundles conv/norm/activation layers.\n This block simplifies the usage of convolution layers, which are commonly",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:1-37"
+ },
+ "6597": {
+ "file_id": 503,
+ "content": "The code is defining a function that creates a ConvBNLayer, which is a combination of convolution, normalization, and activation layers. It simplifies the usage of these layers in a convolutional neural network model.",
+ "type": "comment"
+ },
+ "6598": {
+ "file_id": 503,
+ "content": " used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU).\n It is based upon three build methods: `build_conv_layer()`,\n `build_norm_layer()` and `build_activation_layer()`.\n Besides, we add some additional features in this module.\n 1. Automatically set `bias` of the conv layer.\n 2. Spectral norm is supported.\n 3. More padding modes are supported. Before PyTorch 1.5, nn.Conv2d only\n supports zero and circular padding, and we add \"reflect\" padding mode.\n Args:\n in_channels (int): Number of channels in the input feature map.\n Same as that in ``nn._ConvNd``.\n out_channels (int): Number of channels produced by the convolution.\n Same as that in ``nn._ConvNd``.\n kernel_size (int | tuple[int]): Size of the convolving kernel.\n Same as that in ``nn._ConvNd``.\n stride (int | tuple[int]): Stride of the convolution.\n Same as that in ``nn._ConvNd``.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:38-56"
+ },
+ "6599": {
+ "file_id": 503,
+ "content": "This code defines a Conv2D layer with additional features including automatic bias setting, spectral norm support, and more padding modes. It is used in building convolutional layers, normalization layers, and activation layers for ResNet3D backbones in PaddleVideo.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/66.json b/docs/data/66.json
new file mode 100644
index 000000000..3ed41406b
--- /dev/null
+++ b/docs/data/66.json
@@ -0,0 +1,550 @@
+{
+ "6600": {
+ "file_id": 503,
+ "content": " padding (int | tuple[int]): Zero-padding added to both sides of\n the input. Same as that in ``nn._ConvNd``.\n dilation (int | tuple[int]): Spacing between kernel elements.\n Same as that in ``nn._ConvNd``.\n groups (int): Number of blocked connections from input channels to\n output channels. Same as that in ``nn._ConvNd``.\n \"\"\"\n def __init__(\n self,\n in_channels,\n out_channels,\n kernel_size,\n padding=0,\n stride=1,\n dilation=1,\n groups=1,\n act=None,\n bias=None,\n ):\n super(ConvBNLayer, self).__init__()\n self._conv = nn.Conv3D(\n in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=padding,\n dilation=dilation,\n groups=groups,\n bias_attr=bias)\n self._batch_norm = nn.BatchNorm3D(out_channels, momentum=0.1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:57-89"
+ },
+ "6601": {
+ "file_id": 503,
+ "content": "This code defines a ConvBNLayer class, which is a 3D convolutional layer followed by batch normalization. It takes parameters such as in_channels, out_channels, kernel_size, padding, stride, dilation, groups, act (activation function), and bias. The constructor initializes the Conv3D layer and BatchNorm3D with the specified parameters.",
+ "type": "comment"
+ },
+ "6602": {
+ "file_id": 503,
+ "content": " self.act = act\n if act is not None:\n self._act_op = nn.ReLU()\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if self.act is not None:\n y = self._act_op(y)\n return y\nclass Bottleneck3d(nn.Layer):\n \"\"\"Bottleneck 3d block for ResNet3D.\n Args:\n inplanes (int): Number of channels for the input in first conv3d layer.\n planes (int): Number of channels produced by some norm/conv3d layers.\n spatial_stride (int): Spatial stride in the conv3d layer. Default: 1.\n temporal_stride (int): Temporal stride in the conv3d layer. Default: 1.\n dilation (int): Spacing between kernel elements. Default: 1.\n downsample (nn.Module | None): Downsample layer. Default: None.\n inflate (bool): Whether to inflate kernel. Default: True.\n inflate_style (str): ``3x1x1`` or ``3x3x3``. which determines the\n kernel sizes and padding strides for conv1 and conv2 in each block.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:90-115"
+ },
+ "6603": {
+ "file_id": 503,
+ "content": "The code defines a Bottleneck3d class, which represents a bottleneck block for ResNet3D. It takes in input channels (inplanes), output channels (planes), spatial and temporal stride, dilation rate, downsample layer, inflate flag, and inflate style as arguments. The class has an act variable to store the activation function and defines a forward method that performs convolutions, batch normalization, and activation if necessary.",
+ "type": "comment"
+ },
+ "6604": {
+ "file_id": 503,
+ "content": " Default: '3x1x1'.\n non_local (bool): Determine whether to apply non-local module in this\n block. Default: False.\n non_local_cfg (dict): Config for non-local module. Default: ``dict()``.\n conv_cfg (dict): Config dict for convolution layer.\n Default: ``dict(type='Conv3d')``.\n norm_cfg (dict): Config for norm layers. required keys are ``type``,\n Default: ``dict(type='BN3d')``.\n act_cfg (dict): Config dict for activation layer.\n Default: ``dict(type='ReLU')``.\n with_cp (bool): Use checkpoint or not. Using checkpoint will save some\n memory while slowing down the training speed. Default: False.\n \"\"\"\n expansion = 4\n def __init__(self,\n inplanes,\n planes,\n spatial_stride=1,\n temporal_stride=1,\n dilation=1,\n downsample=None,\n inflate=True,\n inflate_style='3x1x1',\n non_local=False,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:116-140"
+ },
+ "6605": {
+ "file_id": 503,
+ "content": "The code defines a ResNet3D block with various configurations including the number of input and output planes, spatial and temporal stride, dilation rate, downsampling method, inflation settings, and whether to apply non-local modules or not. The default configuration includes convolution, norm, and activation layers, as well as an option for using checkpoint to save memory at the cost of training speed.",
+ "type": "comment"
+ },
+ "6606": {
+ "file_id": 503,
+ "content": " non_local_cfg=dict(),\n conv_cfg=dict(type='Conv3d'),\n norm_cfg=dict(type='BN3d'),\n act_cfg=dict(type='ReLU'),\n with_cp=False):\n super().__init__()\n assert inflate_style in ['3x1x1', '3x3x3']\n self.inplanes = inplanes\n self.planes = planes\n self.spatial_stride = spatial_stride\n self.temporal_stride = temporal_stride\n self.dilation = dilation\n self.inflate = inflate\n self.inflate_style = inflate_style\n self.norm_cfg = norm_cfg\n self.conv_cfg = conv_cfg\n self.act_cfg = act_cfg\n self.with_cp = with_cp\n self.non_local = non_local\n self.non_local_cfg = non_local_cfg\n self.conv1_stride_s = 1\n self.conv2_stride_s = spatial_stride\n self.conv1_stride_t = 1\n self.conv2_stride_t = temporal_stride\n if self.inflate:\n if inflate_style == '3x1x1':\n conv1_kernel_size = (3, 1, 1)\n conv1_padding = (1, 0, 0)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:141-171"
+ },
+ "6607": {
+ "file_id": 503,
+ "content": "This code initializes an instance of a 3D ResNet backbone model with specified parameters, including planes, spatial and temporal strides, dilation, inflate style, norm and conv configurations, whether to use non-local blocks, and more. It sets various attributes based on the input and instantiates a Conv3d layer for the first block.",
+ "type": "comment"
+ },
+ "6608": {
+ "file_id": 503,
+ "content": " conv2_kernel_size = (1, 3, 3)\n conv2_padding = (0, dilation, dilation)\n else:\n conv1_kernel_size = (1, 1, 1)\n conv1_padding = (0, 0, 0)\n conv2_kernel_size = (3, 3, 3)\n conv2_padding = (1, dilation, dilation)\n else:\n conv1_kernel_size = (1, 1, 1)\n conv1_padding = (0, 0, 0)\n conv2_kernel_size = (1, 3, 3)\n conv2_padding = (0, dilation, dilation)\n self.conv1 = ConvBNLayer(\n in_channels=inplanes,\n out_channels=planes,\n kernel_size=conv1_kernel_size,\n stride=(self.conv1_stride_t, self.conv1_stride_s,\n self.conv1_stride_s),\n padding=conv1_padding,\n bias=False,\n act='relu')\n self.conv2 = ConvBNLayer(\n in_channels=planes,\n out_channels=planes,\n kernel_size=conv2_kernel_size,\n stride=(self.conv2_stride_t, self.conv2_stride_s,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:172-198"
+ },
+ "6609": {
+ "file_id": 503,
+ "content": "Code is setting up convolutional layers for a ResNet3D model. It creates ConvBNLayer instances with different kernel sizes and padding based on the dilation value. These layers are used for temporal, spatial, and spatial dimensions depending on the dilation value provided.",
+ "type": "comment"
+ },
+ "6610": {
+ "file_id": 503,
+ "content": " self.conv2_stride_s),\n padding=conv2_padding,\n dilation=(1, dilation, dilation),\n bias=False,\n act='relu')\n self.conv3 = ConvBNLayer(\n in_channels=planes,\n out_channels=planes * self.expansion,\n kernel_size=1,\n bias=False,\n act=None,\n )\n self.downsample = downsample\n self.relu = nn.ReLU()\n def forward(self, x):\n \"\"\"Defines the computation performed at every call.\"\"\"\n def _inner_forward(x):\n \"\"\"Forward wrapper for utilizing checkpoint.\"\"\"\n identity = x\n out = self.conv1(x)\n out = self.conv2(out)\n out = self.conv3(out)\n if self.downsample is not None:\n identity = self.downsample(x)\n out = out + identity\n return out\n out = _inner_forward(x)\n out = self.relu(out)\n if self.non_local:\n out = self.non_local_block(out)\n return out",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:199-239"
+ },
+ "6611": {
+ "file_id": 503,
+ "content": "This code defines a ResNet3D block with ConvBNLayer, downsample layer, and ReLU activation. The forward method applies the layers sequentially, optionally performs downsampling, and adds the identity connection before returning the output. Non-local blocks can be applied if specified.",
+ "type": "comment"
+ },
+ "6612": {
+ "file_id": 503,
+ "content": "class ResNet3d(nn.Layer):\n \"\"\"ResNet 3d backbone.\n Args:\n depth (int): Depth of resnet, from {18, 34, 50, 101, 152}.\n pretrained (str | None): Name of pretrained model.\n stage_blocks (tuple | None): Set number of stages for each res layer.\n Default: None.\n pretrained2d (bool): Whether to load pretrained 2D model.\n Default: True.\n in_channels (int): Channel num of input features. Default: 3.\n base_channels (int): Channel num of stem output features. Default: 64.\n out_indices (Sequence[int]): Indices of output feature. Default: (3, ).\n num_stages (int): Resnet stages. Default: 4.\n spatial_strides (Sequence[int]):\n Spatial strides of residual blocks of each stage.\n Default: ``(1, 2, 2, 2)``.\n temporal_strides (Sequence[int]):\n Temporal strides of residual blocks of each stage.\n Default: ``(1, 1, 1, 1)``.\n dilations (Sequence[int]): Dilation of each stage.\n Default: ``(1, 1, 1, 1)``.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:242-263"
+ },
+ "6613": {
+ "file_id": 503,
+ "content": "The code defines a ResNet 3D backbone, with options for depth (18, 34, 50, 101, or 152), pretrained model name, number of stages for each res layer, loading of pretrained 2D model, input channel features, output feature indices, number of stages, and spatial and temporal strides.",
+ "type": "comment"
+ },
+ "6614": {
+ "file_id": 503,
+ "content": " conv1_kernel (Sequence[int]): Kernel size of the first conv layer.\n Default: ``(3, 7, 7)``.\n conv1_stride_s (int): Spatial stride of the first conv layer.\n Default: 2.\n conv1_stride_t (int): Temporal stride of the first conv layer.\n Default: 1.\n pool1_stride_s (int): Spatial stride of the first pooling layer.\n Default: 2.\n pool1_stride_t (int): Temporal stride of the first pooling layer.\n Default: 1.\n with_pool2 (bool): Whether to use pool2. Default: True.\n inflate (Sequence[int]): Inflate Dims of each block.\n Default: (1, 1, 1, 1).\n inflate_style (str): ``3x1x1`` or ``3x3x3``. which determines the\n kernel sizes and padding strides for conv1 and conv2 in each block.\n Default: '3x1x1'.\n conv_cfg (dict): Config for conv layers. required keys are ``type``\n Default: ``dict(type='Conv3d')``.\n norm_cfg (dict): Config for norm layers. required keys are ``type`` and",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:264-282"
+ },
+ "6615": {
+ "file_id": 503,
+ "content": "This code defines the parameters for ResNet3D backbone model including kernel sizes, stride values, and inflation dimensions. It also sets the default configuration for convolutional layers and normalization layers. The inflation style determines the kernel sizes for conv1 and conv2 in each block based on the given string input.",
+ "type": "comment"
+ },
+ "6616": {
+ "file_id": 503,
+ "content": " ``requires_grad``.\n Default: ``dict(type='BN3d', requires_grad=True)``.\n act_cfg (dict): Config dict for activation layer.\n Default: ``dict(type='ReLU', inplace=True)``.\n norm_eval (bool): Whether to set BN layers to eval mode, namely, freeze\n running stats (mean and var). Default: False.\n with_cp (bool): Use checkpoint or not. Using checkpoint will save some\n memory while slowing down the training speed. Default: False.\n non_local (Sequence[int]): Determine whether to apply non-local module\n in the corresponding block of each stages. Default: (0, 0, 0, 0).\n non_local_cfg (dict): Config for non-local module. Default: ``dict()``.\n zero_init_residual (bool):\n Whether to use zero initialization for residual block,\n Default: True.\n kwargs (dict, optional): Key arguments for \"make_res_layer\".\n \"\"\"\n arch_settings = {\n 50: (Bottleneck3d, (3, 4, 6, 3)),\n 101: (Bottleneck3d, (3, 4, 23, 3)),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:283-302"
+ },
+ "6617": {
+ "file_id": 503,
+ "content": "This code defines the parameters and architecture settings for ResNet3D model in PaddleVideo. It includes options such as backbone type, stages, activation layer, normalization mode, checkpoint usage, non-local module application, and residual block initialization.",
+ "type": "comment"
+ },
+ "6618": {
+ "file_id": 503,
+ "content": " 152: (Bottleneck3d, (3, 8, 36, 3))\n }\n def __init__(self,\n depth,\n stage_blocks=None,\n pretrained2d=True,\n in_channels=3,\n num_stages=4,\n base_channels=64,\n out_indices=(3, ),\n spatial_strides=(1, 2, 2, 2),\n temporal_strides=(1, 1, 1, 1),\n dilations=(1, 1, 1, 1),\n conv1_kernel=(3, 7, 7),\n conv1_stride_s=2,\n conv1_stride_t=1,\n pool1_stride_s=2,\n pool1_stride_t=1,\n with_pool1=True,\n with_pool2=True,\n inflate=(1, 1, 1, 1),\n inflate_style='3x1x1',\n conv_cfg=dict(type='Conv3d'),\n norm_cfg=dict(type='BN3d', requires_grad=True),\n act_cfg=dict(type='ReLU', inplace=True),\n norm_eval=False,\n with_cp=False,\n non_local=(0, 0, 0, 0),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:303-331"
+ },
+ "6619": {
+ "file_id": 503,
+ "content": "This code defines a ResNet3D backbone model with customizable parameters such as depth, stage blocks, and more. It uses Bottleneck3d layers and allows for pre-trained 2D weights usage. The model is designed for processing 4D data (spatial and temporal dimensions).",
+ "type": "comment"
+ },
+ "6620": {
+ "file_id": 503,
+ "content": " non_local_cfg=dict(),\n zero_init_residual=True,\n **kwargs):\n super().__init__()\n if depth not in self.arch_settings:\n raise KeyError(f'invalid depth {depth} for resnet')\n self.depth = depth\n self.pretrained2d = pretrained2d\n self.in_channels = in_channels\n self.base_channels = base_channels\n self.num_stages = num_stages\n assert 1 <= num_stages <= 4\n self.stage_blocks = stage_blocks\n self.out_indices = out_indices\n assert max(out_indices) < num_stages\n self.spatial_strides = spatial_strides\n self.temporal_strides = temporal_strides\n self.dilations = dilations\n assert len(spatial_strides) == len(temporal_strides) == len(\n dilations) == num_stages\n if self.stage_blocks is not None:\n assert len(self.stage_blocks) == num_stages\n self.conv1_kernel = conv1_kernel\n self.conv1_stride_s = conv1_stride_s\n self.conv1_stride_t = conv1_stride_t",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:332-357"
+ },
+ "6621": {
+ "file_id": 503,
+ "content": "This function is initializing a ResNet3D model with specified depth, input channels, base channels, number of stages, stage blocks, output indices, spatial and temporal strides, dilations, and convolution kernel parameters. It raises an error if the provided depth does not match any of the known configurations or if the output indices exceed the number of stages. If the stage_blocks are specified, it also checks that their length matches the number of stages. The class inherits from a superclass.",
+ "type": "comment"
+ },
+ "6622": {
+ "file_id": 503,
+ "content": " self.pool1_stride_s = pool1_stride_s\n self.pool1_stride_t = pool1_stride_t\n self.with_pool1 = with_pool1\n self.with_pool2 = with_pool2\n self.stage_inflations = _ntuple(num_stages)(inflate)\n self.non_local_stages = _ntuple(num_stages)(non_local)\n self.inflate_style = inflate_style\n self.conv_cfg = conv_cfg\n self.norm_cfg = norm_cfg\n self.act_cfg = act_cfg\n self.norm_eval = norm_eval\n self.with_cp = with_cp\n self.zero_init_residual = zero_init_residual\n self.block, stage_blocks = self.arch_settings[depth]\n if self.stage_blocks is None:\n self.stage_blocks = stage_blocks[:num_stages]\n self.inplanes = self.base_channels\n self.non_local_cfg = non_local_cfg\n self._make_stem_layer()\n self.res_layers = []\n for i, num_blocks in enumerate(self.stage_blocks):\n spatial_stride = spatial_strides[i]\n temporal_stride = temporal_strides[i]\n dilation = dilations[i]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:358-387"
+ },
+ "6623": {
+ "file_id": 503,
+ "content": "This code sets various attributes for a ResNet3D model. It initializes strides, determines if pooling layers are used in certain stages, inflates stages based on input, and configures convolutional, normalization, and activation settings. It also defines the block architecture and stage blocks according to the provided depth. Finally, it creates stem and residual layers based on the configuration.",
+ "type": "comment"
+ },
+ "6624": {
+ "file_id": 503,
+ "content": " planes = self.base_channels * 2**i\n res_layer = self.make_res_layer(\n self.block,\n self.inplanes,\n planes,\n num_blocks,\n spatial_stride=spatial_stride,\n temporal_stride=temporal_stride,\n dilation=dilation,\n norm_cfg=self.norm_cfg,\n conv_cfg=self.conv_cfg,\n act_cfg=self.act_cfg,\n non_local=self.non_local_stages[i],\n non_local_cfg=self.non_local_cfg,\n inflate=self.stage_inflations[i],\n inflate_style=self.inflate_style,\n with_cp=with_cp,\n **kwargs)\n self.inplanes = planes * self.block.expansion\n layer_name = f'layer{i + 1}'\n self.add_sublayer(layer_name, res_layer)\n self.res_layers.append(layer_name)\n self.feat_dim = self.block.expansion * self.base_channels * 2**(\n len(self.stage_blocks) - 1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:388-412"
+ },
+ "6625": {
+ "file_id": 503,
+ "content": "This code defines a function that adds ResNet3D layers with specified block, input and output planes, number of blocks, spatial and temporal strides, dilation, norm/conv cfg, non-local stages, inflations, style, and with_cp. It updates inplanes and feat_dim accordingly.",
+ "type": "comment"
+ },
+ "6626": {
+ "file_id": 503,
+ "content": " @staticmethod\n def make_res_layer(block,\n inplanes,\n planes,\n blocks,\n spatial_stride=1,\n temporal_stride=1,\n dilation=1,\n inflate=1,\n inflate_style='3x1x1',\n non_local=0,\n non_local_cfg=dict(),\n norm_cfg=None,\n act_cfg=None,\n conv_cfg=None,\n with_cp=False,\n **kwargs):\n \"\"\"Build residual layer for ResNet3D.\n Args:\n block (nn.Module): Residual module to be built.\n inplanes (int): Number of channels for the input feature\n in each block.\n planes (int): Number of channels for the output feature\n in each block.\n blocks (int): Number of residual blocks.\n spatial_stride (int | Sequence[int]): Spatial strides in",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:414-440"
+ },
+ "6627": {
+ "file_id": 503,
+ "content": "The function \"make_res_layer\" builds a residual layer for ResNet3D. It takes parameters such as block, inplanes, planes, blocks, spatial_stride, temporal_stride, and other optional settings like non_local, norm_cfg, act_cfg, conv_cfg, with_cp to create the residual module. The function constructs the layer based on the input arguments and returns it.",
+ "type": "comment"
+ },
+ "6628": {
+ "file_id": 503,
+ "content": " residual and conv layers. Default: 1.\n temporal_stride (int | Sequence[int]): Temporal strides in\n residual and conv layers. Default: 1.\n dilation (int): Spacing between kernel elements. Default: 1.\n inflate (int | Sequence[int]): Determine whether to inflate\n for each block. Default: 1.\n inflate_style (str): ``3x1x1`` or ``3x3x3``. which determines\n the kernel sizes and padding strides for conv1 and conv2\n in each block. Default: '3x1x1'.\n non_local (int | Sequence[int]): Determine whether to apply\n non-local module in the corresponding block of each stages.\n Default: 0.\n non_local_cfg (dict): Config for non-local module.\n Default: ``dict()``.\n conv_cfg (dict | None): Config for norm layers. Default: None.\n norm_cfg (dict | None): Config for norm layers. Default: None.\n act_cfg (dict | None): Config for activate layers. Default: None.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:441-457"
+ },
+ "6629": {
+ "file_id": 503,
+ "content": "This function defines the ResNet3D backbone model, allowing customization through parameters such as residual and conv layers, temporal stride, dilation, inflate, inflate_style, non_local modules, conv_cfg, norm_cfg, and act_cfg. Default values are provided for each parameter.",
+ "type": "comment"
+ },
+ "6630": {
+ "file_id": 503,
+ "content": " with_cp (bool | None): Use checkpoint or not. Using checkpoint\n will save some memory while slowing down the training speed.\n Default: False.\n Returns:\n nn.Module: A residual layer for the given config.\n \"\"\"\n inflate = inflate if not isinstance(inflate,\n int) else (inflate, ) * blocks\n non_local = non_local if not isinstance(non_local,\n int) else (non_local, ) * blocks\n assert len(inflate) == blocks and len(non_local) == blocks\n downsample = None\n if spatial_stride != 1 or inplanes != planes * block.expansion:\n downsample = ConvBNLayer(\n in_channels=inplanes,\n out_channels=planes * block.expansion,\n kernel_size=1,\n stride=(temporal_stride, spatial_stride, spatial_stride),\n bias=False,\n act=None)\n layers = []\n layers.append(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:458-481"
+ },
+ "6631": {
+ "file_id": 503,
+ "content": "This function creates a residual layer based on the given configuration. It uses inflation and non-local blocks for the specified number of blocks, and optionally adds downsampling if there is a change in spatial or in/out planes. The output is a neural network module (nn.Module).",
+ "type": "comment"
+ },
+ "6632": {
+ "file_id": 503,
+ "content": " block(\n inplanes,\n planes,\n spatial_stride=spatial_stride,\n temporal_stride=temporal_stride,\n dilation=dilation,\n downsample=downsample,\n inflate=(inflate[0] == 1),\n inflate_style=inflate_style,\n non_local=(non_local[0] == 1),\n non_local_cfg=non_local_cfg,\n norm_cfg=norm_cfg,\n conv_cfg=conv_cfg,\n act_cfg=act_cfg,\n with_cp=with_cp,\n **kwargs))\n inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(\n block(\n inplanes,\n planes,\n spatial_stride=1,\n temporal_stride=1,\n dilation=dilation,\n inflate=(inflate[i] == 1),\n inflate_style=inflate_style,\n non_local=(non_local[i] == 1),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:482-509"
+ },
+ "6633": {
+ "file_id": 503,
+ "content": "The code defines a ResNet3D architecture with multiple blocks, each with configurable parameters such as spatial and temporal stride, dilation, downsample, inflate style, non-local operation, norm/conv configuration, activation function, and whether to include channel pruning. The inplanes are updated based on the expansion factor of the block.",
+ "type": "comment"
+ },
+ "6634": {
+ "file_id": 503,
+ "content": " non_local_cfg=non_local_cfg,\n norm_cfg=norm_cfg,\n conv_cfg=conv_cfg,\n act_cfg=act_cfg,\n with_cp=with_cp,\n **kwargs))\n return nn.Sequential(*layers)\n @staticmethod\n def _inflate_conv_params(conv3d, state_dict_2d, module_name_2d,\n inflated_param_names):\n \"\"\"Inflate a conv module from 2d to 3d.\n Args:\n conv3d (nn.Module): The destination conv3d module.\n state_dict_2d (OrderedDict): The state dict of pretrained 2d model.\n module_name_2d (str): The name of corresponding conv module in the\n 2d model.\n inflated_param_names (list[str]): List of parameters that have been\n inflated.\n \"\"\"\n weight_2d_name = module_name_2d + '.weight'\n conv2d_weight = state_dict_2d[weight_2d_name]\n kernel_t = conv3d.weight.data.shape[2]\n new_weight = conv2d_weight.data.unsqueeze(2).expand_as(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:510-537"
+ },
+ "6635": {
+ "file_id": 503,
+ "content": "This code defines a function to inflate a 3D convolutional neural network module from a pre-trained 2D model. It takes the destination conv3d module, state dict of the 2D model, name of the corresponding conv module in the 2D model, and list of inflated parameters as inputs. The function extracts the weight from the 2D model's state dict and reshapes it to fit the 3D convolution.",
+ "type": "comment"
+ },
+ "6636": {
+ "file_id": 503,
+ "content": " conv3d.weight) / kernel_t\n conv3d.weight.data.copy_(new_weight)\n inflated_param_names.append(weight_2d_name)\n if getattr(conv3d, 'bias') is not None:\n bias_2d_name = module_name_2d + '.bias'\n conv3d.bias.data.copy_(state_dict_2d[bias_2d_name])\n inflated_param_names.append(bias_2d_name)\n @staticmethod\n def _inflate_bn_params(bn3d, state_dict_2d, module_name_2d,\n inflated_param_names):\n \"\"\"Inflate a norm module from 2d to 3d.\n Args:\n bn3d (nn.Module): The destination bn3d module.\n state_dict_2d (OrderedDict): The state dict of pretrained 2d model.\n module_name_2d (str): The name of corresponding bn module in the\n 2d model.\n inflated_param_names (list[str]): List of parameters that have been\n inflated.\n \"\"\"\n for param_name, param in bn3d.named_parameters():\n param_2d_name = f'{module_name_2d}.{param_name}'",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:538-561"
+ },
+ "6637": {
+ "file_id": 503,
+ "content": "This code inflates 2D convolutional and Batch Normalization (BN) parameters to 3D for a ResNet3D backbone. It copies the weights and biases, if present, from the 2D state dictionary to their corresponding 3D modules and updates the list of inflated parameter names.",
+ "type": "comment"
+ },
+ "6638": {
+ "file_id": 503,
+ "content": " param_2d = state_dict_2d[param_2d_name]\n if param.data.shape != param_2d.shape:\n warnings.warn(f'The parameter of {module_name_2d} is not'\n 'loaded due to incompatible shapes. ')\n return\n param.data.copy_(param_2d)\n inflated_param_names.append(param_2d_name)\n for param_name, param in bn3d.named_buffers():\n param_2d_name = f'{module_name_2d}.{param_name}'\n # some buffers like num_batches_tracked may not exist in old\n # checkpoints\n if param_2d_name in state_dict_2d:\n param_2d = state_dict_2d[param_2d_name]\n param.data.copy_(param_2d)\n inflated_param_names.append(param_2d_name)\n def _make_stem_layer(self):\n \"\"\"Construct the stem layers consists of a conv+norm+act module and a\n pooling layer.\"\"\"\n self.conv1 = ConvBNLayer(\n in_channels=self.in_channels,\n out_channels=self.base_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:562-586"
+ },
+ "6639": {
+ "file_id": 503,
+ "content": "This code snippet is from the PaddleVideo library, specifically the ResNet3D backbone. It is loading and inflating parameters from a state dictionary, ensuring compatibility between 2D and 3D parameter shapes. The function _make_stem_layer constructs a stem layer consisting of a convolution, normalization, activation, and pooling module.",
+ "type": "comment"
+ },
+ "6640": {
+ "file_id": 503,
+ "content": " kernel_size=self.conv1_kernel,\n stride=(self.conv1_stride_t, self.conv1_stride_s,\n self.conv1_stride_s),\n padding=tuple([(k - 1) // 2 for k in _triple(self.conv1_kernel)]),\n bias=False,\n act=\"relu\")\n self.maxpool = nn.MaxPool3D(\n kernel_size=(1, 3, 3),\n stride=(self.pool1_stride_t, self.pool1_stride_s,\n self.pool1_stride_s),\n padding=(0, 1, 1))\n self.pool2 = nn.MaxPool3D(kernel_size=(2, 1, 1), stride=(2, 1, 1))\n @staticmethod\n def _init_weights(self, pretrained=None):\n pass\n def init_weights(self, pretrained=None):\n self._init_weights(self, pretrained)\n def forward(self, x):\n \"\"\"Defines the computation performed at every call.\n Args:\n x (torch.Tensor): The input data.\n Returns:\n torch.Tensor: The feature of the input\n samples extracted by the backbone.\n \"\"\"\n x = self.conv1(x)\n if self.with_pool1:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:587-620"
+ },
+ "6641": {
+ "file_id": 503,
+ "content": "This code is initializing a ResNet3D model with convolutional and pooling layers. The convolution layer has specified kernel size, stride, padding, and uses ReLU activation function. The max pooling layer has varying sizes for temporal, spatial dimensions. This model also includes optional pool1 and can be initialized with pretrained weights.",
+ "type": "comment"
+ },
+ "6642": {
+ "file_id": 503,
+ "content": " x = self.maxpool(x)\n outs = []\n for i, layer_name in enumerate(self.res_layers):\n res_layer = getattr(self, layer_name)\n x = res_layer(x)\n if i == 0 and self.with_pool2:\n x = self.pool2(x)\n if i in self.out_indices:\n outs.append(x)\n if len(outs) == 1:\n return outs[0]\n return tuple(outs)\n def train(self, mode=True):\n \"\"\"Set the optimization status when training.\"\"\"\n super().train()\n if mode and self.norm_eval:\n for m in self.modules():\n if isinstance(m, paddle.nn._BatchNormBase):\n m.eval()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d.py:621-641"
+ },
+ "6643": {
+ "file_id": 503,
+ "content": "This code defines a ResNet-3D backbone model with residual blocks, max pooling layers, and optionally a second pooling layer. The train function sets the model to training mode and evaluates batch normalization layers if self.norm_eval is True.",
+ "type": "comment"
+ },
+ "6644": {
+ "file_id": 504,
+ "content": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py",
+ "type": "filepath"
+ },
+ "6645": {
+ "file_id": 504,
+ "content": "The ResNet3dSlowOnly class creates a Slowfast pathway in the ResNet3d architecture, reduces channel number, and is registered under BACKBONES. The make_res_layer function builds residual layers with specified spatial_strides, temporal_strides, and dilations for 3D Resnet layers.",
+ "type": "summary"
+ },
+ "6646": {
+ "file_id": 504,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport warnings\nimport paddle\nimport paddle.nn as nn\nfrom .resnet3d import ResNet3d, ConvBNLayer\nfrom ..registry import BACKBONES\n@BACKBONES.register()\nclass ResNet3dSlowOnly(ResNet3d):\n \"\"\"A pathway of Slowfast based on ResNet3d.\n Args:\n *args (arguments): Arguments same as :class:``ResNet3d``.\n channel_ratio (int): Reduce the channel number of fast pathway\n by ``channel_ratio``, corresponding to ``beta`` in the paper.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:1-30"
+ },
+ "6647": {
+ "file_id": 504,
+ "content": "This code defines a ResNet3dSlowOnly class that extends ResNet3d for creating a Slowfast pathway. It reduces the channel number of the fast pathway by a specified 'channel_ratio'. This model is registered under BACKBONES and accepts the same arguments as ResNet3d.",
+ "type": "comment"
+ },
+ "6648": {
+ "file_id": 504,
+ "content": " Default: 8.\n **kwargs (keyword arguments): Keywords arguments for ResNet3d.\n \"\"\"\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n self.inplanes = self.base_channels\n self.lateral_connections = []\n for i in range(len(self.stage_blocks)):\n planes = self.base_channels * 2**i\n self.inplanes = planes * self.block.expansion\n def make_res_layer(self,\n block,\n inplanes,\n planes,\n blocks,\n spatial_stride=1,\n temporal_stride=1,\n dilation=1,\n inflate=1,\n inflate_style='3x1x1',\n non_local=0,\n non_local_cfg=dict(),\n conv_cfg=None,\n norm_cfg=None,\n act_cfg=None,\n with_cp=False):\n \"\"\"Build residual layer for Slowfast.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:31-60"
+ },
+ "6649": {
+ "file_id": 504,
+ "content": "Function `__init__` initializes the ResNet3d object by setting initial values for inplanes and lateral_connections based on provided args and kwargs. The function make_res_layer builds a residual layer for Slowfast, taking in various parameters including block type, input planes, output planes, number of blocks, and more.",
+ "type": "comment"
+ },
+ "6650": {
+ "file_id": 504,
+ "content": " Args:\n block (nn.Module): Residual module to be built.\n inplanes (int): Number of channels for the input\n feature in each block.\n planes (int): Number of channels for the output\n feature in each block.\n blocks (int): Number of residual blocks.\n spatial_stride (int | Sequence[int]): Spatial strides\n in residual and conv layers. Default: 1.\n temporal_stride (int | Sequence[int]): Temporal strides in\n residual and conv layers. Default: 1.\n dilation (int): Spacing between kernel elements. Default: 1.\n inflate (int | Sequence[int]): Determine whether to inflate\n for each block. Default: 1.\n inflate_style (str): ``3x1x1`` or ``3x3x3``. which determines\n the kernel sizes and padding strides for conv1 and\n conv2 in each block. Default: ``3x1x1``.\n non_local (int | Sequence[int]): Determine whether to apply",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:62-79"
+ },
+ "6651": {
+ "file_id": 504,
+ "content": "This function is defining a Residual module with specified parameters including block type, input and output planes, number of residual blocks, spatial and temporal strides, dilation rate, whether to inflate or apply non-local operations for each block.",
+ "type": "comment"
+ },
+ "6652": {
+ "file_id": 504,
+ "content": " non-local module in the corresponding block of each stages.\n Default: 0.\n non_local_cfg (dict): Config for non-local module.\n Default: ``dict()``.\n conv_cfg (dict | None): Config for conv layers. Default: None.\n norm_cfg (dict | None): Config for norm layers. Default: None.\n act_cfg (dict | None): Config for activate layers. Default: None.\n with_cp (bool): Use checkpoint or not. Using checkpoint will save\n some memory while slowing down the training speed.\n Default: False.\n Returns:\n nn.Module: A residual layer for the given config.\n \"\"\"\n inflate = inflate if not isinstance(inflate,\n int) else (inflate, ) * blocks\n non_local = non_local if not isinstance(non_local,\n int) else (non_local, ) * blocks\n assert len(inflate) == blocks and len(non_local) == blocks",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:80-98"
+ },
+ "6653": {
+ "file_id": 504,
+ "content": "This function takes in a configuration for building residual layers, including parameters like blocks (number of residual layers to create), inflate (inflation times for the conv layers), non_local (whether to use non-local modules), conv_cfg, norm_cfg, act_cfg, and with_cp (use checkpoint). It asserts that the lengths of inflate and non_local match the number of blocks specified. The function returns a residual layer for the given configuration.",
+ "type": "comment"
+ },
+ "6654": {
+ "file_id": 504,
+ "content": " lateral_inplanes = 0\n if (spatial_stride != 1\n or (inplanes + lateral_inplanes) != planes * block.expansion):\n downsample = ConvBNLayer(\n in_channels=inplanes + lateral_inplanes,\n out_channels=planes * block.expansion,\n kernel_size=1,\n stride=(temporal_stride, spatial_stride, spatial_stride),\n bias=False,\n act=None)\n else:\n downsample = None\n layers = []\n layers.append(\n block(\n inplanes + lateral_inplanes,\n planes,\n spatial_stride,\n temporal_stride,\n dilation,\n downsample,\n inflate=(inflate[0] == 1),\n inflate_style=inflate_style,\n non_local=(non_local[0] == 1),\n non_local_cfg=non_local_cfg,\n conv_cfg=conv_cfg,\n norm_cfg=norm_cfg,\n act_cfg=act_cfg,\n with_cp=with_cp))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:100-129"
+ },
+ "6655": {
+ "file_id": 504,
+ "content": "This code is creating a downsample layer and appending a block to the layers list. The downsample is created based on whether the current input planes match the expected value or not. If it doesn't match, a convolutional layer with stride is used for downsampling. The block is added to the layers list with specified parameters.",
+ "type": "comment"
+ },
+ "6656": {
+ "file_id": 504,
+ "content": " inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(\n block(\n inplanes,\n planes,\n 1,\n 1,\n dilation,\n inflate=(inflate[i] == 1),\n inflate_style=inflate_style,\n non_local=(non_local[i] == 1),\n non_local_cfg=non_local_cfg,\n conv_cfg=conv_cfg,\n norm_cfg=norm_cfg,\n act_cfg=act_cfg,\n with_cp=with_cp))\n return nn.Sequential(*layers)\n def _inflate_conv_params(self, conv3d, state_dict_2d, module_name_2d,\n inflated_param_names):\n \"\"\"Inflate a conv module from 2d to 3d.\n The differences of conv modules betweene 2d and 3d in Pathway\n mainly lie in the inplanes due to lateral connections. To fit the\n shapes of the lateral connection counterpart, it will expand",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:130-157"
+ },
+ "6657": {
+ "file_id": 504,
+ "content": "This code defines a function to create layers for a Resnet3D backbone model in PaddleVideo. It takes in parameters such as planes, blocks, dilation, inflate, inflate_style, non_local, non_local_cfg, conv_cfg, norm_cfg, act_cfg, and with_cp. The function creates layers by appending instances of a block class to a list, and returns them as a sequential model for training or inference. Additionally, there is another function that inflates a 2D conv module to a 3D one, mainly adjusting the inplanes due to lateral connections for fitting the shapes of lateral connection counterparts.",
+ "type": "comment"
+ },
+ "6658": {
+ "file_id": 504,
+ "content": " parameters by concatting conv2d parameters and extra zero paddings.\n Args:\n conv3d (nn.Module): The destination conv3d module.\n state_dict_2d (OrderedDict): The state dict of pretrained 2d model.\n module_name_2d (str): The name of corresponding conv module in the\n 2d model.\n inflated_param_names (list[str]): List of parameters that have been\n inflated.\n \"\"\"\n weight_2d_name = module_name_2d + '.weight'\n conv2d_weight = state_dict_2d[weight_2d_name]\n old_shape = conv2d_weight.shape\n new_shape = conv3d.weight.data.shape\n kernel_t = new_shape[2]\n if new_shape[1] != old_shape[1]:\n if new_shape[1] < old_shape[1]:\n warnings.warn(f'The parameter of {module_name_2d} is not'\n 'loaded due to incompatible shapes. ')\n return\n # Inplanes may be different due to lateral connections\n new_channels = new_shape[1] - old_shape[1]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:158-180"
+ },
+ "6659": {
+ "file_id": 504,
+ "content": "This function loads the 2D model's state dictionary into a 3D Conv module, concatenating conv2d parameters and adding zero paddings to match the new shape. The weight shape of the 2D model is retrieved, and if the number of input channels in the 3D model is different from the 2D model, it will raise a warning or return without loading the parameters due to incompatible shapes.",
+ "type": "comment"
+ },
+ "6660": {
+ "file_id": 504,
+ "content": " pad_shape = old_shape\n pad_shape = pad_shape[:1] + (new_channels, ) + pad_shape[2:]\n # Expand parameters by concat extra channels\n conv2d_weight = paddle.concat(\n (conv2d_weight, paddle.zeros(pad_shape)), axis=1)\n new_weight = conv2d_weight.data.unsqueeze(2).expand_as(\n conv3d.weight) / kernel_t\n conv3d.weight.data.copy_(new_weight)\n inflated_param_names.append(weight_2d_name)\n if getattr(conv3d, 'bias') is not None:\n bias_2d_name = module_name_2d + '.bias'\n conv3d.bias.data.copy_(state_dict_2d[bias_2d_name])\n inflated_param_names.append(bias_2d_name)\nif __name__ == '__main__':\n net = ResNet3dSlowOnly(\n depth=50,\n in_channels=17,\n base_channels=32,\n conv1_kernel=(1, 7, 7),\n num_stages=3,\n out_indices=[2],\n stage_blocks=[3, 4, 6],\n conv1_stride_s=1,\n pool1_stride_s=1,\n inflate=[0, 1, 1],\n with_pool2=False,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:181-210"
+ },
+ "6661": {
+ "file_id": 504,
+ "content": "The code inflates a 2D convolutional layer into a 3D convolutional layer by padding the weights and copying the bias. It does this for all layers specified in the ResNet3dSlowOnly architecture, with specified parameters.",
+ "type": "comment"
+ },
+ "6662": {
+ "file_id": 504,
+ "content": " spatial_strides=[2, 2, 2],\n temporal_strides=[1, 1, 2],\n dilations=[1, 1, 1])\n pass",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet3d_slowonly.py:211-214"
+ },
+ "6663": {
+ "file_id": 504,
+ "content": "This code sets the spatial_strides, temporal_strides, and dilations for a 3D Resnet layer, with spatial strides of [2, 2, 2], temporal strides of [1, 1, 2], and dilations of [1, 1, 1].",
+ "type": "comment"
+ },
+ "6664": {
+ "file_id": 505,
+ "content": "/paddlevideo/modeling/backbones/resnet_slowfast.py",
+ "type": "filepath"
+ },
+ "6665": {
+ "file_id": 505,
+ "content": "The code defines ResNetSlowFast and SlowFast models for video recognition and computer vision tasks, respectively, with separate pathways for slow and fast processing using 3D convolutional layers and multi-pathway models.",
+ "type": "summary"
+ },
+ "6666": {
+ "file_id": 505,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import KaimingNormal\nfrom ..registry import BACKBONES\nfrom paddlevideo.utils.multigrid import get_norm\nimport sys\nimport numpy as np\nimport paddle.distributed as dist\n# seed random seed\npaddle.framework.seed(0)\n# get init parameters for conv layer\ndef get_conv_init(fan_out):\n return KaimingNormal(fan_in=fan_out)\ndef get_bn_param_attr(bn_weight=1.0, coeff=0.0):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:1-33"
+ },
+ "6667": {
+ "file_id": 505,
+ "content": "Copyright notice and license information for the code. Imports necessary modules, defines function to get convolutional layer initialization parameters, and a function to set batch normalization layer parameters. No actual model or functionality defined yet.",
+ "type": "comment"
+ },
+ "6668": {
+ "file_id": 505,
+ "content": " param_attr = paddle.ParamAttr(\n initializer=paddle.nn.initializer.Constant(bn_weight),\n regularizer=paddle.regularizer.L2Decay(coeff))\n return param_attr\n\"\"\"Video models.\"\"\"\nclass BottleneckTransform(paddle.nn.Layer):\n \"\"\"\n Bottleneck transformation: Tx1x1, 1x3x3, 1x1x1, where T is the size of\n temporal kernel.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:34-66"
+ },
+ "6669": {
+ "file_id": 505,
+ "content": "This code defines a BottleneckTransform class in PaddleVideo for video models. It performs Tx1x1, 1x3x3, 1x1x1 transformations with variable temporal kernel sizes. The constructor takes in arguments like dim_in, dim_out, temp_kernel_size, stride, and more to configure the transformation layer.",
+ "type": "comment"
+ },
+ "6670": {
+ "file_id": 505,
+ "content": " dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): if True, calculate the relu on the original\n input without allocating new memory.\n eps (float): epsilon for batch norm.\n dilation (int): size of dilation.\n \"\"\"\n super(BottleneckTransform, self).__init__()\n self.temp_kernel_size = temp_kernel_size\n self._inplace_relu = inplace_relu\n self._eps = eps\n self._stride_1x1 = stride_1x1\n self.norm_module = norm_module\n self._construct(dim_in, dim_out, stride, dim_inner, num_groups,\n dilation)\n def _construct(self, dim_in, dim_out, stride, dim_inner, num_groups,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:67-87"
+ },
+ "6671": {
+ "file_id": 505,
+ "content": "This code defines a class called BottleneckTransform with parameters such as dim_in, dim_out, stride, dim_inner, num_groups, and dilation. It also has attributes like _inplace_relu, _eps, and norm_module for various operations and settings. The _construct method is used to initialize the class with these parameters.",
+ "type": "comment"
+ },
+ "6672": {
+ "file_id": 505,
+ "content": " dilation):\n str1x1, str3x3 = (stride, 1) if self._stride_1x1 else (1, stride)\n fan = (dim_inner) * (self.temp_kernel_size * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.a = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_inner,\n kernel_size=[self.temp_kernel_size, 1, 1],\n stride=[1, str1x1, str1x1],\n padding=[int(self.temp_kernel_size // 2), 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.a_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x3x3, BN, ReLU.\n fan = (dim_inner) * (1 * 3 * 3)\n initializer_tmp = get_conv_init(fan)\n self.b = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_inner,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:88-113"
+ },
+ "6673": {
+ "file_id": 505,
+ "content": "Defines a Conv3D layer for the ResNet_SlowFast backbone, with specified dimensions and stride. Initializes Conv3D weights using get_conv_init function and includes batch normalization (BN) and ReLU activation layers.",
+ "type": "comment"
+ },
+ "6674": {
+ "file_id": 505,
+ "content": " kernel_size=[1, 3, 3],\n stride=[1, str3x3, str3x3],\n padding=[0, dilation, dilation],\n groups=num_groups,\n dilation=[1, dilation, dilation],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.b_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x1x1, BN.\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.c = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_out,\n kernel_size=[1, 1, 1],\n stride=[1, 1, 1],\n padding=[0, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.c_bn = self.norm_module(\n num_features=dim_out,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:114-139"
+ },
+ "6675": {
+ "file_id": 505,
+ "content": "This code defines a 3D convolutional layer with specific kernel sizes, strides, padding, and grouping. It also includes batch normalization layers for the intermediate and output features. The initializer functions are used to set the weights of each layer, with different initializers for different layers.",
+ "type": "comment"
+ },
+ "6676": {
+ "file_id": 505,
+ "content": " epsilon=self._eps,\n weight_attr=get_bn_param_attr(bn_weight=0.0),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n # Branch2a.\n x = self.a(x)\n x = self.a_bn(x)\n x = F.relu(x)\n # Branch2b.\n x = self.b(x)\n x = self.b_bn(x)\n x = F.relu(x)\n # Branch2c\n x = self.c(x)\n x = self.c_bn(x)\n return x\nclass ResBlock(paddle.nn.Layer):\n \"\"\"\n Residual block.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups=1,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n ResBlock class constructs redisual blocks. More details can be found in:\n Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\n \"Deep residual learning for image recognition.\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:140-180"
+ },
+ "6677": {
+ "file_id": 505,
+ "content": "ResNetSlowFast forward function performs convolutions and Batch Normalization for each branch (2a, 2b, 2c), then applies ReLU activation. ResBlock is a layer implementing residual blocks with specified dimensions, stride, inner dimension, groups, dilation, and normalization method.",
+ "type": "comment"
+ },
+ "6678": {
+ "file_id": 505,
+ "content": " https://arxiv.org/abs/1512.03385\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck.\n trans_func (string): transform function to be used to construct the\n bottleneck.\n dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): calculate the relu on the original input\n without allocating new memory.\n eps (float): epsilon for batch norm.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:181-198"
+ },
+ "6679": {
+ "file_id": 505,
+ "content": "This code defines the arguments for constructing a ResNet bottleneck. It includes parameters for input and output channel dimensions, temporal kernel size, stride, transform function, inner dimension, number of groups for convolution, whether to apply stride to 1x1 or 3x3 conv, inplace_relu flag, and epsilon for batch normalization.",
+ "type": "comment"
+ },
+ "6680": {
+ "file_id": 505,
+ "content": " dilation (int): size of dilation.\n \"\"\"\n super(ResBlock, self).__init__()\n self._inplace_relu = inplace_relu\n self._eps = eps\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n # Use skip connection with projection if dim or res change.\n if (dim_in != dim_out) or (stride != 1):\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.branch1 = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=1,\n stride=[1, stride, stride],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:199-237"
+ },
+ "6681": {
+ "file_id": 505,
+ "content": "The code defines a ResBlock class, which is a residual block used in deep neural networks. It initializes the block with input and output dimensions, kernel size, stride, inner dimension, number of groups, and skip connection settings. The constructor method _construct creates a 3D convolution layer for the skip connection if there is a change in dimensions or stride.",
+ "type": "comment"
+ },
+ "6682": {
+ "file_id": 505,
+ "content": " padding=0,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False,\n dilation=1)\n self.branch1_bn = self.norm_module(\n num_features=dim_out,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n self.branch2 = BottleneckTransform(dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation,\n norm_module=self.norm_module)\n def forward(self, x):\n if hasattr(self, \"branch1\"):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:238-260"
+ },
+ "6683": {
+ "file_id": 505,
+ "content": "This code defines a ResNet SlowFast backbone model. It includes convolution layers, batch normalization layers, and BottleneckTransform modules. The forward function checks if the \"branch1\" attribute exists to handle different stages of the network.",
+ "type": "comment"
+ },
+ "6684": {
+ "file_id": 505,
+ "content": " x1 = self.branch1(x)\n x1 = self.branch1_bn(x1)\n x2 = self.branch2(x)\n x = paddle.add(x=x1, y=x2)\n else:\n x2 = self.branch2(x)\n x = paddle.add(x=x, y=x2)\n x = F.relu(x)\n return x\nclass ResStage(paddle.nn.Layer):\n \"\"\"\n Stage of 3D ResNet. It expects to have one or more tensors as input for\n multi-pathway (SlowFast) cases. More details can be found here:\n Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n stride,\n temp_kernel_sizes,\n num_blocks,\n dim_inner,\n num_groups,\n num_block_temp_kernel,\n dilation,\n stride_1x1=False,\n inplace_relu=True,\n norm_module=paddle.nn.BatchNorm3D):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:261-294"
+ },
+ "6685": {
+ "file_id": 505,
+ "content": "The code defines a ResNet stage for multi-pathway (SlowFast) cases in video recognition. It takes one or more tensors as input and applies branching to separate paths with different kernel sizes. The output is added together, passed through ReLU activation, and returned. This stage supports 1x1 stride option and uses BatchNorm3D for normalization.",
+ "type": "comment"
+ },
+ "6686": {
+ "file_id": 505,
+ "content": " \"\"\"\n The `__init__` method of any subclass should also contain these arguments.\n ResStage builds p streams, where p can be greater or equal to one.\n Args:\n dim_in (list): list of p the channel dimensions of the input.\n Different channel dimensions control the input dimension of\n different pathways.\n dim_out (list): list of p the channel dimensions of the output.\n Different channel dimensions control the input dimension of\n different pathways.\n temp_kernel_sizes (list): list of the p temporal kernel sizes of the\n convolution in the bottleneck. Different temp_kernel_sizes\n control different pathway.\n stride (list): list of the p strides of the bottleneck. Different\n stride control different pathway.\n num_blocks (list): list of p numbers of blocks for each of the\n pathway.\n dim_inner (list): list of the p inner channel dimensions of the",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:295-312"
+ },
+ "6687": {
+ "file_id": 505,
+ "content": "The ResStage class constructor takes several lists as arguments to build p streams of pathways, controlling input and output dimensions, temporal kernel sizes, strides, and block numbers for each pathway.",
+ "type": "comment"
+ },
+ "6688": {
+ "file_id": 505,
+ "content": " input. Different channel dimensions control the input dimension\n of different pathways.\n num_groups (list): list of number of p groups for the convolution.\n num_groups=1 is for standard ResNet like networks, and\n num_groups>1 is for ResNeXt like networks.\n num_block_temp_kernel (list): extent the temp_kernel_sizes to\n num_block_temp_kernel blocks, then fill temporal kernel size\n of 1 for the rest of the layers.\n dilation (list): size of dilation for each pathway.\n \"\"\"\n super(ResStage, self).__init__()\n assert all((num_block_temp_kernel[i] <= num_blocks[i]\n for i in range(len(temp_kernel_sizes))))\n self.num_blocks = num_blocks\n self.temp_kernel_sizes = [\n (temp_kernel_sizes[i] * num_blocks[i])[:num_block_temp_kernel[i]] +\n [1] * (num_blocks[i] - num_block_temp_kernel[i])\n for i in range(len(temp_kernel_sizes))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:313-330"
+ },
+ "6689": {
+ "file_id": 505,
+ "content": "This code defines a ResStage class for a residual block. It takes input dimensions and channel dimensions as parameters, and initializes the number of blocks and temporal kernel sizes based on these inputs. The code also ensures that the provided number of block temporary kernel sizes does not exceed the specified number of blocks.",
+ "type": "comment"
+ },
+ "6690": {
+ "file_id": 505,
+ "content": " ]\n assert (len({\n len(dim_in),\n len(dim_out),\n len(temp_kernel_sizes),\n len(stride),\n len(num_blocks),\n len(dim_inner),\n len(num_groups),\n len(num_block_temp_kernel),\n }) == 1)\n self.num_pathways = len(self.num_blocks)\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n for pathway in range(self.num_pathways):\n for i in range(self.num_blocks[pathway]):\n res_block = ResBlock(\n dim_in[pathway] if i == 0 else dim_out[pathway],\n dim_out[pathway],\n self.temp_kernel_sizes[pathway][i],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:331-372"
+ },
+ "6691": {
+ "file_id": 505,
+ "content": "The code initializes a ResNet SlowFast model by creating instances of blocks based on given parameters. It ensures that the input and output dimensions are correctly set, creates the desired number of pathways, and applies the specified norm module. The constructor then iterates over each pathway and block, creating ResBlock instances with appropriate sizes and configurations.",
+ "type": "comment"
+ },
+ "6692": {
+ "file_id": 505,
+ "content": " stride[pathway] if i == 0 else 1,\n dim_inner[pathway],\n num_groups[pathway],\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation[pathway],\n norm_module=self.norm_module)\n self.add_sublayer(\"pathway{}_res{}\".format(pathway, i),\n res_block)\n def forward(self, inputs):\n output = []\n for pathway in range(self.num_pathways):\n x = inputs[pathway]\n for i in range(self.num_blocks[pathway]):\n m = getattr(self, \"pathway{}_res{}\".format(pathway, i))\n x = m(x)\n output.append(x)\n return output\nclass ResNetBasicStem(paddle.nn.Layer):\n \"\"\"\n ResNe(X)t 3D stem module.\n Performs spatiotemporal Convolution, BN, and Relu following by a\n spatiotemporal pooling.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:373-404"
+ },
+ "6693": {
+ "file_id": 505,
+ "content": "This code defines a ResNet backbone with slow-fast pathways, which includes residual blocks and basic stem modules. The `forward` method processes inputs from each pathway and returns the outputs as a list. It uses getattr to access the correct residual block module for each iteration in each pathway.",
+ "type": "comment"
+ },
+ "6694": {
+ "file_id": 505,
+ "content": " kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n super(ResNetBasicStem, self).__init__()\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)\n def _construct_stem(self, dim_in, dim_out):\n fan = (dim_out) * (self.kernel[0] * self.kernel[1] * self.kernel[2])\n initializer_tmp = get_conv_init(fan)\n self._conv = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=self.kernel,\n stride=self.stride,\n padding=self.padding,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = self.norm_module(num_features=dim_out,\n epsilon=self.eps,\n weight_attr=get_bn_param_attr(),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:405-432"
+ },
+ "6695": {
+ "file_id": 505,
+ "content": "This code defines a class for ResNet basic stem module with options to specify kernel, stride, padding, and batch normalization. It initializes the Conv3D layer and BatchNorm3D module based on the specified parameters. The constructor also calls the _construct_stem method to further initialize the Conv3D layer and BatchNorm3D module.",
+ "type": "comment"
+ },
+ "6696": {
+ "file_id": 505,
+ "content": " bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x = self._conv(x)\n x = self._bn(x)\n x = F.relu(x)\n x = F.max_pool3d(x=x,\n kernel_size=[1, 3, 3],\n stride=[1, 2, 2],\n padding=[0, 1, 1],\n data_format=\"NCDHW\")\n return x\nclass VideoModelStem(paddle.nn.Layer):\n \"\"\"\n Video 3D stem module. Provides stem operations of Conv, BN, ReLU, MaxPool\n on input data tensor for slow and fast pathways.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (list): the list of channel dimensions of the inputs.\n dim_out (list): the output dimension of the convolution in the stem\n layer.\n kernel (list): the kernels' size of the convolutions in the stem",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:433-466"
+ },
+ "6697": {
+ "file_id": 505,
+ "content": "This code defines a 3D stem module for video input. It consists of convolutional, batch normalization, ReLU, and max pooling operations applied to both slow and fast pathways. The dim_in, dim_out, kernel, stride, padding parameters are used to configure the specifics of these operations. Epsilon (eps) is a small value for numerical stability, and norm_module is the batch normalization module being used.",
+ "type": "comment"
+ },
+ "6698": {
+ "file_id": 505,
+ "content": " layers. Temporal kernel size, height kernel size, width kernel\n size in order.\n stride (list): the stride sizes of the convolutions in the stem\n layer. Temporal kernel stride, height kernel size, width kernel\n size in order.\n padding (list): the paddings' sizes of the convolutions in the stem\n layer. Temporal padding size, height padding size, width padding\n size in order.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(VideoModelStem, self).__init__()\n assert (len({\n len(dim_in),\n len(dim_out),\n len(kernel),\n len(stride),\n len(padding),\n }) == 1), \"Input pathway dimensions are not consistent.\"\n self.num_pathways = len(dim_in)\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:467-492"
+ },
+ "6699": {
+ "file_id": 505,
+ "content": "The code defines a VideoModelStem class with parameters for input and output dimensions, temporal kernel size, stride, padding, epsilon for batch norm, and the normalization module. It checks for consistent dimensions and initializes instance variables before calling a constructor method.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/67.json b/docs/data/67.json
new file mode 100644
index 000000000..d61d9ac7d
--- /dev/null
+++ b/docs/data/67.json
@@ -0,0 +1,550 @@
+{
+ "6700": {
+ "file_id": 505,
+ "content": " def _construct_stem(self, dim_in, dim_out):\n for pathway in range(len(dim_in)):\n stem = ResNetBasicStem(dim_in[pathway], dim_out[pathway],\n self.kernel[pathway], self.stride[pathway],\n self.padding[pathway], self.eps,\n self.norm_module)\n self.add_sublayer(\"pathway{}_stem\".format(pathway), stem)\n def forward(self, x):\n assert (len(x) == self.num_pathways\n ), \"Input tensor does not contain {} pathway\".format(\n self.num_pathways)\n for pathway in range(len(x)):\n m = getattr(self, \"pathway{}_stem\".format(pathway))\n x[pathway] = m(x[pathway])\n return x\nclass FuseFastToSlow(paddle.nn.Layer):\n \"\"\"\n Fuses the information from the Fast pathway to the Slow pathway. Given the\n tensors from Slow pathway and Fast pathway, fuse information from Fast to\n Slow, then return the fused tensors from Slow and Fast pathway in order.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:494-518"
+ },
+ "6701": {
+ "file_id": 505,
+ "content": "This code defines a class that constructs a stem for each pathway in ResNet, then applies it to the input tensors. The FuseFastToSlow class fuses information from the Fast pathway to the Slow pathway and returns the fused tensors in order.",
+ "type": "comment"
+ },
+ "6702": {
+ "file_id": 505,
+ "content": " \"\"\"\n def __init__(self,\n dim_in,\n fusion_conv_channel_ratio,\n fusion_kernel,\n alpha,\n fuse_bn_relu=1,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimension of the input.\n fusion_conv_channel_ratio (int): channel ratio for the convolution\n used to fuse from Fast pathway to Slow pathway.\n fusion_kernel (int): kernel size of the convolution used to fuse\n from Fast pathway to Slow pathway.\n alpha (int): the frame rate ratio between the Fast and Slow pathway.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(FuseFastToSlow, self).__init__()\n self.fuse_bn_relu = fuse_bn_relu\n fan = (dim_in * fusion_conv_channel_ratio) * (fusion_kernel * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self._conv_f2s = paddle.nn.Conv3D(\n in_channels=dim_in,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:519-544"
+ },
+ "6703": {
+ "file_id": 505,
+ "content": "This function initializes the FuseFastToSlow class, which takes in dimensions, fusion parameters, and other options. It sets up a convolutional layer to fuse information from the Fast pathway to the Slow pathway. It uses a specified channel ratio and kernel size for the convolution operation. The epsilon parameter is used for batch normalization.",
+ "type": "comment"
+ },
+ "6704": {
+ "file_id": 505,
+ "content": " out_channels=dim_in * fusion_conv_channel_ratio,\n kernel_size=[fusion_kernel, 1, 1],\n stride=[alpha, 1, 1],\n padding=[fusion_kernel // 2, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = norm_module(num_features=dim_in * fusion_conv_channel_ratio,\n epsilon=eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x_s = x[0]\n x_f = x[1]\n fuse = self._conv_f2s(x_f)\n # TODO: For AVA, set fuse_bn_relu=1, check mAP's improve.\n if self.fuse_bn_relu:\n fuse = self._bn(fuse)\n fuse = F.relu(fuse)\n x_s_fuse = paddle.concat(x=[x_s, fuse], axis=1, name=None)\n return [x_s_fuse, x_f]\n@BACKBONES.register()\nclass ResNetSlowFast(paddle.nn.Layer):\n \"\"\"\n SlowFast model builder for SlowFast network.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:545-572"
+ },
+ "6705": {
+ "file_id": 505,
+ "content": "This code defines a ResNetSlowFast model in PaddlePaddle, which is a variation of the SlowFast network. It includes a fusion convolution layer followed by batch normalization and ReLU activation if the fuse_bn_relu flag is set to True. The forward function performs concatenation of the input features and returns the result. This model is registered under BACKBONES for future use.",
+ "type": "comment"
+ },
+ "6706": {
+ "file_id": 505,
+ "content": " Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(\n self,\n alpha,\n beta,\n bn_norm_type=\"batchnorm\",\n bn_num_splits=1,\n num_pathways=2,\n depth=50,\n num_groups=1,\n input_channel_num=[3, 3],\n width_per_group=64,\n fusion_conv_channel_ratio=2,\n fusion_kernel_sz=7, #5?\n pool_size_ratio=[[1, 1, 1], [1, 1, 1]],\n fuse_bn_relu = 1,\n spatial_strides = [[1, 1], [2, 2], [2, 2], [2, 2]],\n use_pool_af_s2 = 1,\n ):\n \"\"\"\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n super(ResNetSlowFast, self).__init__()\n self.alpha = alpha #8\n self.beta = beta #8\n self.norm_module = get_norm(bn_norm_type, bn_num_splits)\n self.num_pathways = num_pathways\n self.depth = depth",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:574-607"
+ },
+ "6707": {
+ "file_id": 505,
+ "content": "This code defines a class ResNetSlowFast, which is a variant of the ResNet architecture with slow and fast paths for video recognition. It takes various parameters such as alpha, beta, bn_norm_type, etc., to build the network. The class also includes methods for initializing the model with the given parameters.",
+ "type": "comment"
+ },
+ "6708": {
+ "file_id": 505,
+ "content": " self.num_groups = num_groups\n self.input_channel_num = input_channel_num\n self.width_per_group = width_per_group\n self.fusion_conv_channel_ratio = fusion_conv_channel_ratio\n self.fusion_kernel_sz = fusion_kernel_sz # NOTE: modify to 7 in 8*8, 5 in old implement\n self.pool_size_ratio = pool_size_ratio\n self.fuse_bn_relu = fuse_bn_relu\n self.spatial_strides = spatial_strides\n self.use_pool_af_s2 = use_pool_af_s2\n self._construct_network()\n def _construct_network(self):\n \"\"\"\n Builds a SlowFast model.\n The first pathway is the Slow pathway\n and the second pathway is the Fast pathway.\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n temp_kernel = [\n [[1], [5]], # conv1 temporal kernel for slow and fast pathway.\n [[1], [3]], # res2 temporal kernel for slow and fast pathway.\n [[1], [3]], # res3 temporal kernel for slow and fast pathway.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:608-632"
+ },
+ "6709": {
+ "file_id": 505,
+ "content": "This code defines a SlowFast model for computer vision tasks. It takes in several parameters including the number of groups, input channel number, and others. The construct_network function builds the SlowFast model with separate pathways for Slow and Fast pathways using different temporal kernels.",
+ "type": "comment"
+ },
+ "6710": {
+ "file_id": 505,
+ "content": " [[3], [3]], # res4 temporal kernel for slow and fast pathway.\n [[3], [3]],\n ] # res5 temporal kernel for slow and fast pathway.\n self.s1 = VideoModelStem(\n dim_in=self.input_channel_num,\n dim_out=[self.width_per_group, self.width_per_group // self.beta],\n kernel=[temp_kernel[0][0] + [7, 7], temp_kernel[0][1] + [7, 7]],\n stride=[[1, 2, 2]] * 2,\n padding=[\n [temp_kernel[0][0][0] // 2, 3, 3],\n [temp_kernel[0][1][0] // 2, 3, 3],\n ],\n norm_module=self.norm_module)\n self.s1_fuse = FuseFastToSlow(\n dim_in=self.width_per_group // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu)\n # ResNet backbone\n MODEL_STAGE_DEPTH = {50: (3, 4, 6, 3)}\n (d2, d3, d4, d5) = MODEL_STAGE_DEPTH[self.depth]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:633-657"
+ },
+ "6711": {
+ "file_id": 505,
+ "content": "The code initializes a SlowFast ResNet backbone model. It defines temporal kernels for res4 and res5 pathways, creates a VideoModelStem layer with specific dimensions and parameters, and a FuseFastToSlow layer for fusion. The code also sets the model stage depth according to the chosen depth (50, 101, or 152).",
+ "type": "comment"
+ },
+ "6712": {
+ "file_id": 505,
+ "content": " num_block_temp_kernel = [[3, 3], [4, 4], [6, 6], [3, 3]]\n spatial_dilations = [[1, 1], [1, 1], [1, 1], [1, 1]]\n spatial_strides = self.spatial_strides\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [2, 2]]\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [1, 1]] #TODO:check which value is FAIR's impliment\n out_dim_ratio = self.beta // self.fusion_conv_channel_ratio #4\n dim_inner = self.width_per_group * self.num_groups #64\n self.s2 = ResStage(dim_in=[\n self.width_per_group + self.width_per_group // out_dim_ratio,\n self.width_per_group // self.beta,\n ],\n dim_out=[\n self.width_per_group * 4,\n self.width_per_group * 4 // self.beta,\n ],\n dim_inner=[dim_inner, dim_inner // self.beta],\n temp_kernel_sizes=temp_kernel[1],\n stride=spatial_strides[0],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:659-678"
+ },
+ "6713": {
+ "file_id": 505,
+ "content": "This code defines a ResStage, which is a stage in the ResNet SlowFast model. It sets the dimensions and parameters for this stage including input and output widths, kernel sizes, and strides. The code also specifies temporary kernel sizes and spatial strides for this particular stage of the model.",
+ "type": "comment"
+ },
+ "6714": {
+ "file_id": 505,
+ "content": " num_blocks=[d2] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[0],\n dilation=spatial_dilations[0],\n norm_module=self.norm_module)\n self.s2_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 4 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s3 = ResStage(\n dim_in=[\n self.width_per_group * 4 +\n self.width_per_group * 4 // out_dim_ratio,\n self.width_per_group * 4 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 8,\n self.width_per_group * 8 // self.beta,\n ],\n dim_inner=[dim_inner * 2, dim_inner * 2 // self.beta],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:679-704"
+ },
+ "6715": {
+ "file_id": 505,
+ "content": "The code defines a resnet_slowfast model with two main components: s1 and s3. The s1 component consists of three branches, with the first two having 2x repeat_num_body blocks each, while the third has (2*repeat_num_body + 1) blocks. It also includes norm_module and spatial_dilations for the first branch. The s3 component contains a ResStage layer. The model uses parameters such as width_per_group, out_dim_ratio, dim_inner, repeat_num_body, alpha, beta, fusion_conv_channel_ratio, fusion_kernel_sz, norm_module, and fuse_bn_relu.",
+ "type": "comment"
+ },
+ "6716": {
+ "file_id": 505,
+ "content": " temp_kernel_sizes=temp_kernel[2],\n stride=spatial_strides[1],\n num_blocks=[d3] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[1],\n dilation=spatial_dilations[1],\n norm_module=self.norm_module,\n )\n self.s3_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 8 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s4 = ResStage(\n dim_in=[\n self.width_per_group * 8 +\n self.width_per_group * 8 // out_dim_ratio,\n self.width_per_group * 8 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 16,\n self.width_per_group * 16 // self.beta,\n ],\n dim_inner=[dim_inner * 4, dim_inner * 4 // self.beta],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:705-733"
+ },
+ "6717": {
+ "file_id": 505,
+ "content": "The code initializes and defines different layers for the ResNet SlowFast model. It includes operations such as creating convolutional layers, fusing fast to slow features, and defining a stage layer with specified input and output dimensions. The alpha, fusion_kernel_sz, out_dim_ratio, beta, dim_inner values are used to control the specifics of these operations.",
+ "type": "comment"
+ },
+ "6718": {
+ "file_id": 505,
+ "content": " temp_kernel_sizes=temp_kernel[3],\n stride=spatial_strides[2],\n num_blocks=[d4] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[2],\n dilation=spatial_dilations[2],\n norm_module=self.norm_module,\n )\n self.s4_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 16 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s5 = ResStage(\n dim_in=[\n self.width_per_group * 16 +\n self.width_per_group * 16 // out_dim_ratio,\n self.width_per_group * 16 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 32,\n self.width_per_group * 32 // self.beta,\n ],\n dim_inner=[dim_inner * 8, dim_inner * 8 // self.beta],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:734-762"
+ },
+ "6719": {
+ "file_id": 505,
+ "content": "The code defines the ResStage and FuseFastToSlow modules for a ResNet SlowFast model. It initializes these modules with specific dimensions and parameters, such as number of channels, fusion kernel size, alpha value, and dilation rates. These modules are used to extract features from the input and fuse them together for further processing in the network.",
+ "type": "comment"
+ },
+ "6720": {
+ "file_id": 505,
+ "content": " temp_kernel_sizes=temp_kernel[4],\n stride=spatial_strides[3],\n num_blocks=[d5] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[3],\n dilation=spatial_dilations[3],\n norm_module=self.norm_module,\n )\n def init_weights(self):\n pass\n def forward(self, x):\n x = self.s1(x) #VideoModelStem\n x = self.s1_fuse(x) #FuseFastToSlow\n x = self.s2(x) #ResStage\n x = self.s2_fuse(x)\n # TODO: For AVA, set use_pool_af_s2=1, check mAP's improve.\n if self.use_pool_af_s2:\n for pathway in range(self.num_pathways):\n x[pathway] = F.max_pool3d(x=x[pathway],\n kernel_size=self.pool_size_ratio[pathway],\n stride=self.pool_size_ratio[pathway],\n padding=[0, 0, 0],\n data_format=\"NCDHW\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:763-788"
+ },
+ "6721": {
+ "file_id": 505,
+ "content": "This code defines a ResNet slowfast model with specified parameters, initializes the weights, and forwards the input data. It also includes an optional max-pooling operation for one of its stages (s2) depending on a flag value.",
+ "type": "comment"
+ },
+ "6722": {
+ "file_id": 505,
+ "content": " x = self.s3(x)\n x = self.s3_fuse(x)\n x = self.s4(x)\n x = self.s4_fuse(x)\n x = self.s5(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast.py:790-795"
+ },
+ "6723": {
+ "file_id": 505,
+ "content": "This code snippet represents the final segment of a neural network model. It processes input data (x) through four sequential layers (s3, s4, and s5), then fuses their outputs before returning the result as output. Each layer is likely responsible for feature extraction or transformation at different levels of the network's architecture.",
+ "type": "comment"
+ },
+ "6724": {
+ "file_id": 506,
+ "content": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py",
+ "type": "filepath"
+ },
+ "6725": {
+ "file_id": 506,
+ "content": "The PaddleVideo library includes functions for convolutional layers, ResBlock classes, Slow and Fast branches, and a VideoModelStem class to initialize a ResNet SlowFast model with MRI for video analysis.",
+ "type": "summary"
+ },
+ "6726": {
+ "file_id": 506,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import KaimingNormal\nfrom ..registry import BACKBONES\nfrom paddlevideo.utils.multigrid import get_norm\nimport sys\nimport numpy as np\nimport paddle.distributed as dist\n# seed random seed\npaddle.framework.seed(0)\n# get init parameters for conv layer\ndef get_conv_init(fan_out):\n return KaimingNormal(fan_in=fan_out)\ndef get_bn_param_attr(bn_weight=1.0, coeff=0.0):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:1-33"
+ },
+ "6727": {
+ "file_id": 506,
+ "content": "This code is part of the PaddleVideo library, licensed under Apache 2.0. It imports necessary modules and defines functions for initializing convolutional layers with KaimingNormal distribution. It also includes a function for batch normalization parameters and registers backbones in PaddlePaddle's registry. The code uses paddle, numpy, and other libraries for various functionalities such as seeding random numbers for reproducibility.",
+ "type": "comment"
+ },
+ "6728": {
+ "file_id": 506,
+ "content": " param_attr = paddle.ParamAttr(\n initializer=paddle.nn.initializer.Constant(bn_weight),\n regularizer=paddle.regularizer.L2Decay(coeff))\n return param_attr\n\"\"\"Video models.\"\"\"\nclass BottleneckTransform(paddle.nn.Layer):\n \"\"\"\n Bottleneck transformation: Tx1x1, 1x3x3, 1x1x1, where T is the size of\n temporal kernel.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:34-66"
+ },
+ "6729": {
+ "file_id": 506,
+ "content": "This code defines a BottleneckTransform class, which is a layer for video models. It performs temporal convolutions with 1x1, 1x3x3, and 1x1x1 layers, where T is the size of the temporal kernel. The class takes various parameters such as dimension, stride, and kernel sizes.",
+ "type": "comment"
+ },
+ "6730": {
+ "file_id": 506,
+ "content": " dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): if True, calculate the relu on the original\n input without allocating new memory.\n eps (float): epsilon for batch norm.\n dilation (int): size of dilation.\n \"\"\"\n super(BottleneckTransform, self).__init__()\n self.temp_kernel_size = temp_kernel_size\n self._inplace_relu = inplace_relu\n self._eps = eps\n self._stride_1x1 = stride_1x1\n self.norm_module = norm_module\n self._construct(dim_in, dim_out, stride, dim_inner, num_groups,\n dilation)\n def _construct(self, dim_in, dim_out, stride, dim_inner, num_groups,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:67-87"
+ },
+ "6731": {
+ "file_id": 506,
+ "content": "The code defines a class 'BottleneckTransform' with parameters for dimensions, stride, inner dimension, number of groups, and other attributes. It inherits from another class and initializes its own instance variables before constructing the model structure. The constructor method takes in various arguments to configure the bottleneck transformation block.",
+ "type": "comment"
+ },
+ "6732": {
+ "file_id": 506,
+ "content": " dilation):\n str1x1, str3x3 = (stride, 1) if self._stride_1x1 else (1, stride)\n fan = (dim_inner) * (self.temp_kernel_size * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.a = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_inner,\n kernel_size=[self.temp_kernel_size, 1, 1],\n stride=[1, str1x1, str1x1],\n padding=[int(self.temp_kernel_size // 2), 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.a_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x3x3, BN, ReLU.\n fan = (dim_inner) * (1 * 3 * 3)\n initializer_tmp = get_conv_init(fan)\n self.b = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_inner,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:88-113"
+ },
+ "6733": {
+ "file_id": 506,
+ "content": "This code initializes two 3D convolutional layers with Batch Normalization and ReLU activation. The first layer has stride 1 for all dimensions, while the second layer has stride 1 for the first dimension and a different value (determined by _stride_1x1) for the remaining two dimensions. Both layers have specified kernel sizes, padding, and use custom initializers.",
+ "type": "comment"
+ },
+ "6734": {
+ "file_id": 506,
+ "content": " kernel_size=[1, 3, 3],\n stride=[1, str3x3, str3x3],\n padding=[0, dilation, dilation],\n groups=num_groups,\n dilation=[1, dilation, dilation],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.b_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x1x1, BN.\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.c = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_out,\n kernel_size=[1, 1, 1],\n stride=[1, 1, 1],\n padding=[0, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.c_bn = self.norm_module(\n num_features=dim_out,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:114-139"
+ },
+ "6735": {
+ "file_id": 506,
+ "content": "The code defines a Conv3D layer with 1x3x3 kernel, stride of str3x3, and dilation of dilation. It also includes a batch normalization (BN) module for the output. The BN module is applied to the output of the previous layer and has no bias. Finally, another Conv3D layer with 1x1x1 kernel and no BN is defined followed by another BN without a bias.",
+ "type": "comment"
+ },
+ "6736": {
+ "file_id": 506,
+ "content": " epsilon=self._eps,\n weight_attr=get_bn_param_attr(bn_weight=0.0),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n # Branch2a.\n x = self.a(x)\n x = self.a_bn(x)\n x = F.relu(x)\n # Branch2b.\n x = self.b(x)\n x = self.b_bn(x)\n x = F.relu(x)\n # Branch2c\n x = self.c(x)\n x = self.c_bn(x)\n return x\nclass ResBlock(paddle.nn.Layer):\n \"\"\"\n Residual block.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups=1,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n ResBlock class constructs redisual blocks. More details can be found in:\n Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\n \"Deep residual learning for image recognition.\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:140-180"
+ },
+ "6737": {
+ "file_id": 506,
+ "content": "ResNetSlowFastMRI: A residual network model that adds Slow and Fast branches to extract temporal features. ResBlock is a residual block class used in the architecture, which utilizes BatchNorm3D for normalization and applies ReLU activations after each branch.",
+ "type": "comment"
+ },
+ "6738": {
+ "file_id": 506,
+ "content": " https://arxiv.org/abs/1512.03385\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck.\n trans_func (string): transform function to be used to construct the\n bottleneck.\n dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): calculate the relu on the original input\n without allocating new memory.\n eps (float): epsilon for batch norm.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:181-198"
+ },
+ "6739": {
+ "file_id": 506,
+ "content": "This function defines a bottleneck for ResNet and ResNeXt-like networks with specified parameters. It takes channel dimensions, temporal kernel sizes, stride, transform function, inner dimension, number of groups, if applying stride to 1x1 conv, inplace relu calculation, and epsilon for batch norm as arguments.",
+ "type": "comment"
+ },
+ "6740": {
+ "file_id": 506,
+ "content": " dilation (int): size of dilation.\n \"\"\"\n super(ResBlock, self).__init__()\n self._inplace_relu = inplace_relu\n self._eps = eps\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n # Use skip connection with projection if dim or res change.\n if (dim_in != dim_out) or (stride != 1):\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.branch1 = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=1,\n stride=[1, stride, stride],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:199-237"
+ },
+ "6741": {
+ "file_id": 506,
+ "content": "This code defines a ResBlock class with skip connection, which performs convolution operations for image processing tasks. The constructor takes various parameters like dimensions, kernel size, stride, etc., and initializes the necessary components based on whether a skip connection is needed or not.",
+ "type": "comment"
+ },
+ "6742": {
+ "file_id": 506,
+ "content": " padding=0,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False,\n dilation=1)\n self.branch1_bn = self.norm_module(\n num_features=dim_out,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n self.branch2 = BottleneckTransform(dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation,\n norm_module=self.norm_module)\n def forward(self, x):\n if hasattr(self, \"branch1\"):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:238-260"
+ },
+ "6743": {
+ "file_id": 506,
+ "content": "This code defines a ResNet SlowFast MRI model with batch normalization (BN) for the branch1 and a BottleneckTransform layer for branch2. The forward function checks if \"branch1\" exists, suggesting it may be conditionally defined elsewhere in the codebase.",
+ "type": "comment"
+ },
+ "6744": {
+ "file_id": 506,
+ "content": " x1 = self.branch1(x)\n x1 = self.branch1_bn(x1)\n x2 = self.branch2(x)\n x = paddle.add(x=x1, y=x2)\n else:\n x2 = self.branch2(x)\n x = paddle.add(x=x, y=x2)\n x = F.relu(x)\n return x\nclass ResStage(paddle.nn.Layer):\n \"\"\"\n Stage of 3D ResNet. It expects to have one or more tensors as input for\n multi-pathway (SlowFast) cases. More details can be found here:\n Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n stride,\n temp_kernel_sizes,\n num_blocks,\n dim_inner,\n num_groups,\n num_block_temp_kernel,\n dilation,\n stride_1x1=False,\n inplace_relu=True,\n norm_module=paddle.nn.BatchNorm3D):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:261-294"
+ },
+ "6745": {
+ "file_id": 506,
+ "content": "This code defines a ResStage class for 3D ResNet, which can handle multi-pathway cases in SlowFast networks. It consists of one or more tensors as input. The stage includes operations such as adding tensors and applying ReLU activation. It also has a branch1 and branch2 for different paths, with batch normalization applied to the first path.",
+ "type": "comment"
+ },
+ "6746": {
+ "file_id": 506,
+ "content": " \"\"\"\n The `__init__` method of any subclass should also contain these arguments.\n ResStage builds p streams, where p can be greater or equal to one.\n Args:\n dim_in (list): list of p the channel dimensions of the input.\n Different channel dimensions control the input dimension of\n different pathways.\n dim_out (list): list of p the channel dimensions of the output.\n Different channel dimensions control the input dimension of\n different pathways.\n temp_kernel_sizes (list): list of the p temporal kernel sizes of the\n convolution in the bottleneck. Different temp_kernel_sizes\n control different pathway.\n stride (list): list of the p strides of the bottleneck. Different\n stride control different pathway.\n num_blocks (list): list of p numbers of blocks for each of the\n pathway.\n dim_inner (list): list of the p inner channel dimensions of the",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:295-312"
+ },
+ "6747": {
+ "file_id": 506,
+ "content": "The given code is the initialization method of a ResStage class in PaddleVideo. It accepts arguments such as dim_in, dim_out, temp_kernel_sizes, stride, and num_blocks to build p pathways with different channel dimensions, temporal kernel sizes, strides, and numbers of blocks for each pathway.",
+ "type": "comment"
+ },
+ "6748": {
+ "file_id": 506,
+ "content": " input. Different channel dimensions control the input dimension\n of different pathways.\n num_groups (list): list of number of p groups for the convolution.\n num_groups=1 is for standard ResNet like networks, and\n num_groups>1 is for ResNeXt like networks.\n num_block_temp_kernel (list): extent the temp_kernel_sizes to\n num_block_temp_kernel blocks, then fill temporal kernel size\n of 1 for the rest of the layers.\n dilation (list): size of dilation for each pathway.\n \"\"\"\n super(ResStage, self).__init__()\n assert all((num_block_temp_kernel[i] <= num_blocks[i]\n for i in range(len(temp_kernel_sizes))))\n self.num_blocks = num_blocks\n self.temp_kernel_sizes = [\n (temp_kernel_sizes[i] * num_blocks[i])[:num_block_temp_kernel[i]] +\n [1] * (num_blocks[i] - num_block_temp_kernel[i])\n for i in range(len(temp_kernel_sizes))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:313-330"
+ },
+ "6749": {
+ "file_id": 506,
+ "content": "This function initializes a ResStage object, which is a layer of a network. It takes in parameters such as the number of blocks and temporal kernel sizes for each pathway, and asserts that the number of block_temp_kernel does not exceed the number of blocks. The temp_kernel_sizes are extended to num_block_temp_kernel blocks with temporal kernel size 1 for the rest of the layers.",
+ "type": "comment"
+ },
+ "6750": {
+ "file_id": 506,
+ "content": " ]\n assert (len({\n len(dim_in),\n len(dim_out),\n len(temp_kernel_sizes),\n len(stride),\n len(num_blocks),\n len(dim_inner),\n len(num_groups),\n len(num_block_temp_kernel),\n }) == 1)\n self.num_pathways = len(self.num_blocks)\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n for pathway in range(self.num_pathways):\n for i in range(self.num_blocks[pathway]):\n res_block = ResBlock(\n dim_in[pathway] if i == 0 else dim_out[pathway],\n dim_out[pathway],\n self.temp_kernel_sizes[pathway][i],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:331-372"
+ },
+ "6751": {
+ "file_id": 506,
+ "content": "The code creates an instance of a backbone network with adjustable pathways and blocks. It checks the input parameters' length, assigns the number of pathways, initializes a norm module, and calls a private method to construct the network structure. The _construct method loops through each pathway and block, creating ResBlock instances.",
+ "type": "comment"
+ },
+ "6752": {
+ "file_id": 506,
+ "content": " stride[pathway] if i == 0 else 1,\n dim_inner[pathway],\n num_groups[pathway],\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation[pathway],\n norm_module=self.norm_module)\n self.add_sublayer(\"pathway{}_res{}\".format(pathway, i),\n res_block)\n def forward(self, inputs):\n output = []\n for pathway in range(self.num_pathways):\n x = inputs[pathway]\n for i in range(self.num_blocks[pathway]):\n m = getattr(self, \"pathway{}_res{}\".format(pathway, i))\n x = m(x)\n output.append(x)\n return output\nclass ResNetBasicStem(paddle.nn.Layer):\n \"\"\"\n ResNe(X)t 3D stem module.\n Performs spatiotemporal Convolution, BN, and Relu following by a\n spatiotemporal pooling.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:373-404"
+ },
+ "6753": {
+ "file_id": 506,
+ "content": "This code defines a ResNet backbone with SlowFast pathways, including stem module and residual blocks. It initializes the layers for each pathway and then defines a forward function to process inputs through the specified number of pathways and blocks. The ResNetBasicStem performs spatiotemporal convolution, batch normalization, and ReLU before pooling in the 3D stem module.",
+ "type": "comment"
+ },
+ "6754": {
+ "file_id": 506,
+ "content": " kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n super(ResNetBasicStem, self).__init__()\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)\n def _construct_stem(self, dim_in, dim_out):\n fan = (dim_out) * (self.kernel[0] * self.kernel[1] * self.kernel[2])\n initializer_tmp = get_conv_init(fan)\n self._conv = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=self.kernel,\n stride=self.stride,\n padding=self.padding,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = self.norm_module(num_features=dim_out,\n epsilon=self.eps,\n weight_attr=get_bn_param_attr(),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:405-432"
+ },
+ "6755": {
+ "file_id": 506,
+ "content": "This code defines a ResNetBasicStem class that initializes the stem of a ResNet network. The constructor takes parameters for kernel size, stride, padding, epsilon value, and norm module. The _construct_stem method creates a 3D convolutional layer with specified dimensions and uses an appropriate initializer for its weights. A batch normalization layer is also created using the provided norm module and epsilon value.",
+ "type": "comment"
+ },
+ "6756": {
+ "file_id": 506,
+ "content": " bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x = self._conv(x)\n x = self._bn(x)\n x = F.relu(x)\n x = F.max_pool3d(x=x,\n kernel_size=[1, 3, 3],\n stride=[1, 2, 2],\n padding=[0, 1, 1],\n data_format=\"NCDHW\")\n return x\nclass VideoModelStem(paddle.nn.Layer):\n \"\"\"\n Video 3D stem module. Provides stem operations of Conv, BN, ReLU, MaxPool\n on input data tensor for slow and fast pathways.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (list): the list of channel dimensions of the inputs.\n dim_out (list): the output dimension of the convolution in the stem\n layer.\n kernel (list): the kernels' size of the convolutions in the stem",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:433-466"
+ },
+ "6757": {
+ "file_id": 506,
+ "content": "This code defines a video stem module for slow and fast pathways, performing Conv, BN, ReLU, MaxPool operations on input data tensors. The function takes dim_in, dim_out, kernel, stride, padding, eps, and norm_module as arguments.",
+ "type": "comment"
+ },
+ "6758": {
+ "file_id": 506,
+ "content": " layers. Temporal kernel size, height kernel size, width kernel\n size in order.\n stride (list): the stride sizes of the convolutions in the stem\n layer. Temporal kernel stride, height kernel size, width kernel\n size in order.\n padding (list): the paddings' sizes of the convolutions in the stem\n layer. Temporal padding size, height padding size, width padding\n size in order.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(VideoModelStem, self).__init__()\n assert (len({\n len(dim_in),\n len(dim_out),\n len(kernel),\n len(stride),\n len(padding),\n }) == 1), \"Input pathway dimensions are not consistent.\"\n self.num_pathways = len(dim_in)\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:467-492"
+ },
+ "6759": {
+ "file_id": 506,
+ "content": "The code defines a class VideoModelStem with parameters for dimensions, kernel size, stride, padding, and epsilon for batch normalization. It checks for consistent input pathway dimensions and initializes instance variables before constructing the stem layer.",
+ "type": "comment"
+ },
+ "6760": {
+ "file_id": 506,
+ "content": " def _construct_stem(self, dim_in, dim_out):\n for pathway in range(len(dim_in)):\n stem = ResNetBasicStem(dim_in[pathway], dim_out[pathway],\n self.kernel[pathway], self.stride[pathway],\n self.padding[pathway], self.eps,\n self.norm_module)\n self.add_sublayer(\"pathway{}_stem\".format(pathway), stem)\n def forward(self, x):\n assert (len(x) == self.num_pathways\n ), \"Input tensor does not contain {} pathway\".format(\n self.num_pathways)\n for pathway in range(len(x)):\n m = getattr(self, \"pathway{}_stem\".format(pathway))\n x[pathway] = m(x[pathway])\n return x\nclass FuseFastToSlow(paddle.nn.Layer):\n \"\"\"\n Fuses the information from the Fast pathway to the Slow pathway. Given the\n tensors from Slow pathway and Fast pathway, fuse information from Fast to\n Slow, then return the fused tensors from Slow and Fast pathway in order.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:494-518"
+ },
+ "6761": {
+ "file_id": 506,
+ "content": "This code defines a class that constructs and fuses two pathways (slow and fast) in a video processing model. It initializes the stem layers for each pathway and then fuses the information from the fast pathway to the slow pathway. The input tensor should contain the specified number of pathways, and the output is returned as tensors from both pathways in order.",
+ "type": "comment"
+ },
+ "6762": {
+ "file_id": 506,
+ "content": " \"\"\"\n def __init__(self,\n dim_in,\n fusion_conv_channel_ratio,\n fusion_kernel,\n alpha,\n fuse_bn_relu=1,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimension of the input.\n fusion_conv_channel_ratio (int): channel ratio for the convolution\n used to fuse from Fast pathway to Slow pathway.\n fusion_kernel (int): kernel size of the convolution used to fuse\n from Fast pathway to Slow pathway.\n alpha (int): the frame rate ratio between the Fast and Slow pathway.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(FuseFastToSlow, self).__init__()\n self.fuse_bn_relu = fuse_bn_relu\n fan = (dim_in * fusion_conv_channel_ratio) * (fusion_kernel * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self._conv_f2s = paddle.nn.Conv3D(\n in_channels=dim_in,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:519-544"
+ },
+ "6763": {
+ "file_id": 506,
+ "content": "The code is initializing a class called FuseFastToSlow with parameters for input channel dimension, fusion convolution channel ratio, fusion kernel size, and frame rate ratio. It sets the number of channels in the 3D Convolution layer for fusing frames from Fast to Slow pathways based on given ratios. The fusion operation is performed using a Conv3D layer initialized with the given fan value.",
+ "type": "comment"
+ },
+ "6764": {
+ "file_id": 506,
+ "content": " out_channels=dim_in * fusion_conv_channel_ratio,\n kernel_size=[fusion_kernel, 1, 1],\n stride=[alpha, 1, 1],\n padding=[fusion_kernel // 2, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = norm_module(num_features=dim_in * fusion_conv_channel_ratio,\n epsilon=eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x_s = x[0]\n x_f = x[1]\n fuse = self._conv_f2s(x_f)\n # TODO: For AVA, set fuse_bn_relu=1, check mAP's improve.\n if self.fuse_bn_relu:\n fuse = self._bn(fuse)\n fuse = F.relu(fuse)\n x_s_fuse = paddle.concat(x=[x_s, fuse], axis=1, name=None)\n return [x_s_fuse, x_f]\n@BACKBONES.register()\nclass ResNetSlowFast_MRI(paddle.nn.Layer):\n \"\"\"\n SlowFast model builder for SlowFast network.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:545-572"
+ },
+ "6765": {
+ "file_id": 506,
+ "content": "This code defines a ResNetSlowFast_MRI model, which is a type of SlowFast network. It includes a fusion convolution layer and a batch normalization (BN) layer with optional ReLU activation after the fusion convolution. The forward method combines input x_s and x_f using concat before returning both inputs.",
+ "type": "comment"
+ },
+ "6766": {
+ "file_id": 506,
+ "content": " Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(\n self,\n alpha,\n beta,\n bn_norm_type=\"batchnorm\",\n bn_num_splits=1,\n num_pathways=2,\n depth=50,\n num_groups=1,\n input_channel_num=[1, 1],\n width_per_group=64,\n fusion_conv_channel_ratio=2,\n fusion_kernel_sz=7, #5?\n pool_size_ratio=[[1, 1, 1], [1, 1, 1]],\n fuse_bn_relu=1,\n spatial_strides=[[1, 1], [2, 2], [2, 2], [2, 2]],\n use_pool_af_s2=1,\n ):\n \"\"\"\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n super(ResNetSlowFast_MRI, self).__init__()\n self.alpha = alpha #8\n self.beta = beta #8\n self.norm_module = get_norm(bn_norm_type, bn_num_splits)\n self.num_pathways = num_pathways\n self.depth = depth",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:574-607"
+ },
+ "6767": {
+ "file_id": 506,
+ "content": "This code initializes a ResNetSlowFast_MRI model with specified parameters, including alpha and beta values for the network architecture. The class extends from an existing superclass and includes properties like the norm module type, number of pathways, depth, group numbers, input channel numbers, width per group, fusion convolution channel ratio, pool size ratios, whether to use a pooling average operation at spatial stride 2, and spatial strides. The class also initializes these specified attributes for the model configuration.",
+ "type": "comment"
+ },
+ "6768": {
+ "file_id": 506,
+ "content": " self.num_groups = num_groups\n self.input_channel_num = input_channel_num\n self.width_per_group = width_per_group\n self.fusion_conv_channel_ratio = fusion_conv_channel_ratio\n self.fusion_kernel_sz = fusion_kernel_sz # NOTE: modify to 7 in 8*8, 5 in old implement\n self.pool_size_ratio = pool_size_ratio\n self.fuse_bn_relu = fuse_bn_relu\n self.spatial_strides = spatial_strides\n self.use_pool_af_s2 = use_pool_af_s2\n self._construct_network()\n def _construct_network(self):\n \"\"\"\n Builds a SlowFast model.\n The first pathway is the Slow pathway\n and the second pathway is the Fast pathway.\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n temp_kernel = [\n [[1], [5]], # conv1 temporal kernel for slow and fast pathway.\n [[1], [3]], # res2 temporal kernel for slow and fast pathway.\n [[1], [3]], # res3 temporal kernel for slow and fast pathway.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:608-632"
+ },
+ "6769": {
+ "file_id": 506,
+ "content": "The code defines a SlowFast model with separate slow and fast pathways. The constructor sets parameters like input channel number, group number, fusion convolution channel ratio, and more. It also includes functions to build the network structure. The temporal kernels for each layer of both pathways are defined within the code.",
+ "type": "comment"
+ },
+ "6770": {
+ "file_id": 506,
+ "content": " [[3], [3]], # res4 temporal kernel for slow and fast pathway.\n [[3], [3]],\n ] # res5 temporal kernel for slow and fast pathway.\n self.s1 = VideoModelStem(\n dim_in=self.input_channel_num,\n dim_out=[self.width_per_group, self.width_per_group // self.beta],\n kernel=[temp_kernel[0][0] + [7, 7], temp_kernel[0][1] + [7, 7]],\n stride=[[1, 2, 2]] * 2,\n padding=[\n [temp_kernel[0][0][0] // 2, 3, 3],\n [temp_kernel[0][1][0] // 2, 3, 3],\n ],\n norm_module=self.norm_module)\n self.s1_fuse = FuseFastToSlow(\n dim_in=self.width_per_group // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu)\n # ResNet backbone\n MODEL_STAGE_DEPTH = {50: (3, 4, 6, 3)}\n (d2, d3, d4, d5) = MODEL_STAGE_DEPTH[self.depth]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:633-657"
+ },
+ "6771": {
+ "file_id": 506,
+ "content": "This code defines a ResNet backbone for the MRI dataset. It sets the temporal kernels for res4 and res5 pathways, initializes the video model stem (s1) with specified dimensions and stride, adds a fuseFastToSlow module for fusion, and defines model stage depth based on the chosen depth of ResNet.",
+ "type": "comment"
+ },
+ "6772": {
+ "file_id": 506,
+ "content": " num_block_temp_kernel = [[3, 3], [4, 4], [6, 6], [3, 3]]\n spatial_dilations = [[1, 1], [1, 1], [1, 1], [1, 1]]\n spatial_strides = self.spatial_strides\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [2, 2]]\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [1, 1]] #TODO:check which value is FAIR's impliment\n out_dim_ratio = self.beta // self.fusion_conv_channel_ratio #4\n dim_inner = self.width_per_group * self.num_groups #64\n self.s2 = ResStage(dim_in=[\n self.width_per_group + self.width_per_group // out_dim_ratio,\n self.width_per_group // self.beta,\n ],\n dim_out=[\n self.width_per_group * 4,\n self.width_per_group * 4 // self.beta,\n ],\n dim_inner=[dim_inner, dim_inner // self.beta],\n temp_kernel_sizes=temp_kernel[1],\n stride=spatial_strides[0],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:659-678"
+ },
+ "6773": {
+ "file_id": 506,
+ "content": "The code is defining the parameters for a ResStage layer in a residual network model. It sets the input and output dimensions, inner dimensions, temporal kernel sizes, and stride values based on previously defined values from the function.",
+ "type": "comment"
+ },
+ "6774": {
+ "file_id": 506,
+ "content": " num_blocks=[d2] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[0],\n dilation=spatial_dilations[0],\n norm_module=self.norm_module)\n self.s2_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 4 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s3 = ResStage(\n dim_in=[\n self.width_per_group * 4 +\n self.width_per_group * 4 // out_dim_ratio,\n self.width_per_group * 4 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 8,\n self.width_per_group * 8 // self.beta,\n ],\n dim_inner=[dim_inner * 2, dim_inner * 2 // self.beta],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:679-704"
+ },
+ "6775": {
+ "file_id": 506,
+ "content": "The code initializes a ResNet SlowFast model with multiple layers and parameters. It creates two branches (slow and fast) for the network, each with its own set of layers and parameters. The slow branch has 2x more blocks than the fast branch, and both branches have identical group numbers and dilation rates. The code also initializes a fusing layer that combines features from the fast and slow branches and a stage for the third level of the network.",
+ "type": "comment"
+ },
+ "6776": {
+ "file_id": 506,
+ "content": " temp_kernel_sizes=temp_kernel[2],\n stride=spatial_strides[1],\n num_blocks=[d3] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[1],\n dilation=spatial_dilations[1],\n norm_module=self.norm_module,\n )\n self.s3_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 8 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s4 = ResStage(\n dim_in=[\n self.width_per_group * 8 +\n self.width_per_group * 8 // out_dim_ratio,\n self.width_per_group * 8 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 16,\n self.width_per_group * 16 // self.beta,\n ],\n dim_inner=[dim_inner * 4, dim_inner * 4 // self.beta],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:705-733"
+ },
+ "6777": {
+ "file_id": 506,
+ "content": "This code is initializing a ResNet SlowFast model for MRI. It creates an instance of the class, sets parameters like kernel sizes, strides, block numbers, group numbers, dilation rates, and normalization module. The model consists of multiple stages, including s1, s2, s3_fuse, and s4, each with different dimensions, inner dimensions, and configurations.",
+ "type": "comment"
+ },
+ "6778": {
+ "file_id": 506,
+ "content": " temp_kernel_sizes=temp_kernel[3],\n stride=spatial_strides[2],\n num_blocks=[d4] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[2],\n dilation=spatial_dilations[2],\n norm_module=self.norm_module,\n )\n self.s4_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 16 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s5 = ResStage(\n dim_in=[\n self.width_per_group * 16 +\n self.width_per_group * 16 // out_dim_ratio,\n self.width_per_group * 16 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 32,\n self.width_per_group * 32 // self.beta,\n ],\n dim_inner=[dim_inner * 8, dim_inner * 8 // self.beta],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:734-762"
+ },
+ "6779": {
+ "file_id": 506,
+ "content": "This code defines a ResNet SlowFast model with MRI for video analysis. It includes creating layers for stage 4, fusing fast and slow features, and defining the stage 5 layer with specific dimensions for input, output, and inner dimensions. The model utilizes group width, out_dim_ratio, beta, and dim_inner parameters to control its behavior.",
+ "type": "comment"
+ },
+ "6780": {
+ "file_id": 506,
+ "content": " temp_kernel_sizes=temp_kernel[4],\n stride=spatial_strides[3],\n num_blocks=[d5] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[3],\n dilation=spatial_dilations[3],\n norm_module=self.norm_module,\n )\n def init_weights(self):\n pass\n def forward(self, x):\n x = self.s1(x) #VideoModelStem\n x = self.s1_fuse(x) #FuseFastToSlow\n x = self.s2(x) #ResStage\n x = self.s2_fuse(x)\n # TODO: For AVA, set use_pool_af_s2=1, check mAP's improve.\n if self.use_pool_af_s2:\n for pathway in range(self.num_pathways):\n x[pathway] = F.max_pool3d(\n x=x[pathway],\n kernel_size=self.pool_size_ratio[pathway],\n stride=self.pool_size_ratio[pathway],\n padding=[0, 0, 0],\n data_format=\"NCDHW\")\n x = self.s3(x)\n x = self.s3_fuse(x)\n x = self.s4(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:763-793"
+ },
+ "6781": {
+ "file_id": 506,
+ "content": "This code initializes a 3D ResNet SlowFast model, sets the weights, and applies several stages of convolutions and fusions to process video input. The forward function sequentially passes the input through multiple stages, potentially applying max pooling for AVA if use_pool_af_s2 is True.",
+ "type": "comment"
+ },
+ "6782": {
+ "file_id": 506,
+ "content": " x = self.s4_fuse(x)\n x = self.s5(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py:794-796"
+ },
+ "6783": {
+ "file_id": 506,
+ "content": "This code snippet is part of a ResNet SlowFast model. It fuses the outputs from previous stages (s4), passes them through another stage (s5) and returns the result as output.",
+ "type": "comment"
+ },
+ "6784": {
+ "file_id": 507,
+ "content": "/paddlevideo/modeling/backbones/resnet_tsm.py",
+ "type": "filepath"
+ },
+ "6785": {
+ "file_id": 507,
+ "content": "The code initializes a ResNet-TSM backbone with convolutional layers and bottleneck blocks, but may be deprecated and needs data preparation changes for better compatibility.",
+ "type": "summary"
+ },
+ "6786": {
+ "file_id": 507,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout, MaxPool2D,\n AvgPool2D)\nfrom paddle import ParamAttr\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt\nclass ConvBNLayer(nn.Layer):\n \"\"\"Conv2D and BatchNorm2D layer.\n Args:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:1-30"
+ },
+ "6787": {
+ "file_id": 507,
+ "content": "This code snippet defines a ConvBNLayer class which combines a Conv2D and BatchNorm2D layer. It is imported from paddle.nn and will be used for creating convolutional neural network layers in the PaddlePaddle framework. The layer can be utilized to process image data or other types of spatial input data by applying convolution operations followed by batch normalization.",
+ "type": "comment"
+ },
+ "6788": {
+ "file_id": 507,
+ "content": " in_channels (int): Number of channels for the input.\n out_channels (int): Number of channels for the output.\n kernel_size (int): Kernel size.\n stride (int): Stride in the Conv2D layer. Default: 1.\n groups (int): Groups in the Conv2D, Default: 1.\n act (str): Indicate activation after BatchNorm2D layer.\n name (str): the name of an instance of ConvBNLayer.\n Note: weight and bias initialization include initialize values and name the restored parameters, values initialization are explicit declared in the ```init_weights``` method.\n \"\"\"\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n act=None,\n name=None,\n data_format=\"NCHW\"):\n super(ConvBNLayer, self).__init__()\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:31-53"
+ },
+ "6789": {
+ "file_id": 507,
+ "content": "This code defines a class called ConvBNLayer which inherits from an unspecified parent class. It takes in parameters such as number of input and output channels, kernel size, stride, groups, activation function, name, and data format for the Conv2D layer. The class initializes a Conv2D layer using these parameters and adds BatchNorm2D and ReLU layers after it. Weight and bias initialization values are named in the restore parameters, and they are explicitly declared in the init_weights method.",
+ "type": "comment"
+ },
+ "6790": {
+ "file_id": 507,
+ "content": " stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False,\n data_format=data_format)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._act = act\n self._batch_norm = BatchNorm2D(\n out_channels,\n weight_attr=ParamAttr(name=bn_name + \"_scale\",\n regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(name=bn_name + \"_offset\",\n regularizer=L2Decay(0.0)),\n data_format=data_format)\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if self._act:\n y = getattr(paddle.nn.functional, self._act)(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:54-84"
+ },
+ "6791": {
+ "file_id": 507,
+ "content": "This code defines a BottleneckBlock class with a convolution layer, batch normalization, and optional activation function. The forward pass applies the convolution, batch normalization, and activation if present.",
+ "type": "comment"
+ },
+ "6792": {
+ "file_id": 507,
+ "content": " in_channels,\n out_channels,\n stride,\n shortcut=True,\n num_seg=8,\n name=None,\n data_format=\"NCHW\"):\n super(BottleneckBlock, self).__init__()\n self.data_format = data_format\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act=\"relu\",\n name=name + \"_branch2a\",\n data_format=data_format)\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act=\"relu\",\n name=name + \"_branch2b\",\n data_format=data_format)\n self.conv2 = ConvBNLayer(in_channels=out_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:85-108"
+ },
+ "6793": {
+ "file_id": 507,
+ "content": "This code defines a class called BottleneckBlock, which is a part of a neural network model. It contains several ConvBNLayer objects for processing input data, with different parameters such as in_channels, out_channels, kernel_size, stride, and act (activation function). The class also has an attribute for data_format and initializes the ConvBNLayer objects with specific names.",
+ "type": "comment"
+ },
+ "6794": {
+ "file_id": 507,
+ "content": " out_channels=out_channels * 4,\n kernel_size=1,\n act=None,\n name=name + \"_branch2c\",\n data_format=data_format)\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=stride,\n name=name + \"_branch1\",\n data_format=data_format)\n self.shortcut = shortcut\n self.num_seg = num_seg\n def forward(self, inputs):\n if paddle.is_compiled_with_custom_device('npu'):\n x = inputs\n seg_num = self.num_seg\n shift_ratio = 1.0 / self.num_seg\n shape = x.shape #[N*T, C, H, W]\n reshape_x = x.reshape(\n (-1, seg_num, shape[1], shape[2], shape[3])) #[N, T, C, H, W]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:109-134"
+ },
+ "6795": {
+ "file_id": 507,
+ "content": "This code is initializing a ConvBNLayer and a shortcut connection for the TSM backbone. The layers have specific out_channels, kernel_size, stride, name, and data_format configurations. If a shortcut is not provided, it initializes another ConvBNLayer. The forward function reshapes input to have a shape of [N, T, C, H, W] for further processing.",
+ "type": "comment"
+ },
+ "6796": {
+ "file_id": 507,
+ "content": " pad_x = F.pad(reshape_x, [\n 0,\n 0,\n 1,\n 1,\n 0,\n 0,\n 0,\n 0,\n 0,\n 0,\n ]) #[N, T+2, C, H, W]\n c1 = int(shape[1] * shift_ratio)\n c2 = int(shape[1] * 2 * shift_ratio)\n slice1 = pad_x[:, :seg_num, :c1, :, :]\n slice2 = pad_x[:, 2:seg_num + 2, c1:c2, :, :]\n slice3 = pad_x[:, 1:seg_num + 1, c2:, :, :]\n concat_x = paddle.concat([slice1, slice2, slice3],\n axis=2) #[N, T, C, H, W]\n shifts = concat_x.reshape(shape)\n else:\n shifts = F.temporal_shift(inputs,\n self.num_seg,\n 1.0 / self.num_seg,\n data_format=self.data_format)\n y = self.conv0(shifts)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:135-164"
+ },
+ "6797": {
+ "file_id": 507,
+ "content": "This code performs temporal shift operation on the input tensor. If a certain condition is met, it pads and concatenates slices of the input tensor before performing the reshape and temporal shift operations. The resulting output is then passed through several convolutional layers.",
+ "type": "comment"
+ },
+ "6798": {
+ "file_id": 507,
+ "content": " short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n return F.relu(y)\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n name=None,\n data_format=\"NCHW\"):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(\n in_channels=in_channels,\n out_channels=out_channels,\n filter_size=3,\n stride=stride,\n act=\"relu\",\n name=name + \"_branch2a\",\n data_format=data_format,\n )\n self.conv1 = ConvBNLayer(\n in_channels=out_channels,\n out_channels=out_channels,\n filter_size=3,\n act=None,\n name=name + \"_branch2b\",\n data_format=data_format,\n )\n if not shortcut:\n self.short = ConvBNLayer(\n in_channels=in_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:165-202"
+ },
+ "6799": {
+ "file_id": 507,
+ "content": "The code defines a BasicBlock class which is a residual block. It contains two 3x3 convolutional layers followed by BN and ReLU activations. If the shortcut connection is not used, it also includes an additional convolution layer for the shortcut path. The purpose of this residual block is to alleviate the problem of vanishing gradients in deeper networks.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/68.json b/docs/data/68.json
new file mode 100644
index 000000000..3365cc890
--- /dev/null
+++ b/docs/data/68.json
@@ -0,0 +1,548 @@
+{
+ "6800": {
+ "file_id": 507,
+ "content": " out_channels=out_channels,\n filter_size=1,\n stride=stride,\n name=name + \"_branch1\",\n data_format=data_format,\n )\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(short, conv1)\n y = F.relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTSM(nn.Layer):\n \"\"\"ResNet TSM backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self, depth, num_seg=8, data_format=\"NCHW\", pretrained=None):\n super(ResNetTSM, self).__init__()\n self.pretrained = pretrained\n self.layers = depth\n self.num_seg = num_seg\n self.data_format = data_format\n supported_layers = [18, 34, 50, 101, 152]\n assert self.layers in supported_layers, \\",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:203-241"
+ },
+ "6801": {
+ "file_id": 507,
+ "content": "The code defines a ResNet TSM backbone model with specified depth and data format. It consists of an initialization, a forward function for processing inputs, and the ability to be registered at BACKBONES. It also supports different layers like 18, 34, 50, 101, and 152.",
+ "type": "comment"
+ },
+ "6802": {
+ "file_id": 507,
+ "content": " \"supported layers are {} but input layer is {}\".format(\n supported_layers, self.layers)\n if self.layers == 18:\n depth = [2, 2, 2, 2]\n elif self.layers == 34 or self.layers == 50:\n depth = [3, 4, 6, 3]\n elif self.layers == 101:\n depth = [3, 4, 23, 3]\n elif self.layers == 152:\n depth = [3, 8, 36, 3]\n in_channels = 64\n out_channels = [64, 128, 256, 512]\n self.conv = ConvBNLayer(in_channels=3,\n out_channels=64,\n kernel_size=7,\n stride=2,\n act=\"relu\",\n name=\"conv1\",\n data_format=self.data_format)\n self.pool2D_max = MaxPool2D(\n kernel_size=3,\n stride=2,\n padding=1,\n data_format=self.data_format,\n )\n self.block_list = []\n if self.layers >= 50:\n for block in range(len(depth)):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:242-273"
+ },
+ "6803": {
+ "file_id": 507,
+ "content": "This code initializes a ResNet-TSM backbone with different depth configurations based on the input layers. It includes a convolution layer, max pooling 2D layer, and a block list for deeper networks. The code checks if the layers are supported (18, 34, 50, 101, or 152) and assigns corresponding depth and number of channels accordingly.",
+ "type": "comment"
+ },
+ "6804": {
+ "file_id": 507,
+ "content": " shortcut = False\n for i in range(depth[block]):\n if self.layers in [101, 152] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n conv_name,\n BottleneckBlock(\n in_channels=in_channels\n if i == 0 else out_channels[block] * 4,\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n num_seg=self.num_seg,\n shortcut=shortcut,\n name=conv_name,\n data_format=self.data_format))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:274-293"
+ },
+ "6805": {
+ "file_id": 507,
+ "content": "Code creates bottleneck blocks for ResNet TSM architecture, varying the number of input channels based on layer index and configuration. It adds sublayers with specified parameters including number of segments and stride for each block.",
+ "type": "comment"
+ },
+ "6806": {
+ "file_id": 507,
+ "content": " in_channels = out_channels[block] * 4\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n conv_name,\n BasicBlock(\n in_channels=in_channels[block]\n if i == 0 else out_channels[block],\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n name=conv_name,\n data_format=self.data_format,\n ))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:294-316"
+ },
+ "6807": {
+ "file_id": 507,
+ "content": "Code initializes a ResNet TSM model backbone, with block-specific in_channels and adds either bottleneck or basic blocks depending on the depth configuration. Init_weights function is also defined to initialize weights for the model.",
+ "type": "comment"
+ },
+ "6808": {
+ "file_id": 507,
+ "content": " \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n #XXX: check bias!!! check pretrained!!!\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n #XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:317-332"
+ },
+ "6809": {
+ "file_id": 507,
+ "content": "This code initializes parameters for a ResNet TSM backbone. If a pretrained loading path is provided, it loads the weights from that path; otherwise, it uses specific initialization functions for Conv2D and BatchNorm2d layers. No bias is used in Conv2D layers.",
+ "type": "comment"
+ },
+ "6810": {
+ "file_id": 507,
+ "content": " weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n \"\"\"Define how the backbone is going to run.\n \"\"\"\n #NOTE: (deprecated design) Already merge axis 0(batches) and axis 1(clips) before extracting feature phase,\n # please refer to paddlevideo/modeling/framework/recognizers/recognizer2d.py#L27\n #y = paddle.reshape(\n # inputs, [-1, inputs.shape[2], inputs.shape[3], inputs.shape[4]])\n #NOTE: As paddlepaddle to_static method need a \"pure\" model to trim. It means from\n # 1. the phase of generating data[images, label] from dataloader\n # to\n # 2. last layer of a model, always is FC layer\n y = self.conv(inputs)\n y = self.pool2D_max(y)\n for block in self.block_list:\n y = block(y)\n return y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm.py:333-353"
+ },
+ "6811": {
+ "file_id": 507,
+ "content": "This code defines a forward function for a backbone model. It uses convolution and pooling layers to extract features from input data. The comments indicate that this implementation may be deprecated, and the data preparation should be modified according to recognizer2d.py for better compatibility with paddlepaddle's to_static method.",
+ "type": "comment"
+ },
+ "6812": {
+ "file_id": 508,
+ "content": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py",
+ "type": "filepath"
+ },
+ "6813": {
+ "file_id": 508,
+ "content": "The code defines a ResNet-TSM model in PaddleVideo with Batch Normalization, Leaky ReLU activation, and optional shortcut connections for MRI applications, using ConvBNLayer and ResNetTSM_MRI classes.",
+ "type": "summary"
+ },
+ "6814": {
+ "file_id": 508,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport math\nimport sys\nimport paddle\nimport paddle.nn as nn\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout, MaxPool2D,\n AvgPool2D)\nfrom paddle import ParamAttr\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils.save_load import load_ckpt\nfrom paddle.regularizer import L2Decay\nclass ConvBNLayer(nn.Layer):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:1-32"
+ },
+ "6815": {
+ "file_id": 508,
+ "content": "This code is importing necessary libraries and defining a class for a Convolutional Batch Normalization Layer. It also provides information about copyright, license, and contact details of the PaddlePaddle Authors.",
+ "type": "comment"
+ },
+ "6816": {
+ "file_id": 508,
+ "content": " \"\"\"Conv2D and BatchNorm2D layer.\n Args:\n in_channels (int): Number of channels for the input.\n out_channels (int): Number of channels for the output.\n kernel_size (int): Kernel size.\n stride (int): Stride in the Conv2D layer. Default: 1.\n groups (int): Groups in the Conv2D, Default: 1.\n is_tweaks_mode (bool): switch for tweaks. Default: False.\n act (str): Indicate activation after BatchNorm2D layer.\n name (str): the name of an instance of ConvBNLayer.\n Note: weight and bias initialization include initialize values and name the restored parameters, values initialization are explicit declared in the ```init_weights``` method.\n \"\"\"\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n is_tweaks_mode=False,\n act=None,\n name=None):\n super(ConvBNLayer, self).__init__()\n self.is_tweaks_mode = is_tweaks_mode",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:33-58"
+ },
+ "6817": {
+ "file_id": 508,
+ "content": "This code defines a ConvBNLayer class with various parameters such as in_channels, out_channels, kernel_size, stride, groups, is_tweaks_mode, act, and name. It inherits from the base class and initializes the layer's weights and biases using explicit declarations in the init_weights method.",
+ "type": "comment"
+ },
+ "6818": {
+ "file_id": 508,
+ "content": " #ResNet-D 1/2:add a 2×2 average pooling layer with a stride of 2 before the convolution,\n # whose stride is changed to 1, works well in practice.\n self._pool2d_avg = AvgPool2D(kernel_size=2,\n stride=2,\n padding=0,\n ceil_mode=True)\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._act = act\n self._batch_norm = BatchNorm2D(\n out_channels,\n weight_attr=ParamAttr(name=bn_name + \"_scale\",",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:59-83"
+ },
+ "6819": {
+ "file_id": 508,
+ "content": "This code snippet initializes ResNet-D with a 2x2 average pooling layer followed by a convolution operation. The pooling layer has a stride of 2 and is changed to 1 later in practice. The convolution uses the specified parameters such as in_channels, out_channels, kernel size, stride, padding, groups, and names for weights and batch normalization.",
+ "type": "comment"
+ },
+ "6820": {
+ "file_id": 508,
+ "content": " regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(bn_name + \"_offset\", regularizer=L2Decay(0.0)))\n def forward(self, inputs):\n if self.is_tweaks_mode:\n inputs = self._pool2d_avg(inputs)\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if self._act:\n y = getattr(paddle.nn.functional, self._act)(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n if_first=False,\n num_seg=8,\n name=None):\n super(BottleneckBlock, self).__init__()\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act=\"leaky_relu\",\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:84-112"
+ },
+ "6821": {
+ "file_id": 508,
+ "content": "The code defines a class `ResNetTSM_MRI` which appears to be a backbone model for ResNet-TSM. It contains a forward function that applies pooling, convolution, batch normalization, and activation (if specified) to the inputs. The BottleneckBlock class is defined with options for stride, shortcut connection, number of segments, and name. It initializes a ConvBNLayer instance for the first branch, and another ConvBNLayer instance for the second branch.",
+ "type": "comment"
+ },
+ "6822": {
+ "file_id": 508,
+ "content": " out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act=\"leaky_relu\",\n name=name + \"_branch2b\")\n self.conv2 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n act=None,\n name=name + \"_branch2c\")\n if not shortcut:\n self.short = ConvBNLayer(\n in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=\n 1, #ResNet-D 2/2:add a 2×2 average pooling layer with a stride of 2 before the convolution,\n # whose stride is changed to 1, works well in practice.\n is_tweaks_mode=False if if_first else True,\n name=name + \"_branch1\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:113-134"
+ },
+ "6823": {
+ "file_id": 508,
+ "content": "In this code, the function creates three ConvBNLayer instances: a \"branch2a\", \"branch2b\", and \"branch2c\". The \"branch2a\" instance is created with specified parameters. If shortcut is not set, an additional \"branch1\" instance (ConvBNLayer) is created with a 1x1 convolution layer and a stride of 1. This is explained to be useful in ResNet-D 2/2 configuration where a 2x2 average pooling layer with a stride of 2 is added before the convolution, which is later changed to 1 in practice.",
+ "type": "comment"
+ },
+ "6824": {
+ "file_id": 508,
+ "content": " self.shortcut = shortcut\n self.num_seg = num_seg\n def forward(self, inputs):\n shifts = F.temporal_shift(inputs, self.num_seg, 1.0 / self.num_seg)\n y = self.conv0(shifts)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n return F.leaky_relu(y)\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n name=None):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n filter_size=3,\n stride=stride,\n act=\"leaky_relu\",\n name=name + \"_branch2a\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:136-166"
+ },
+ "6825": {
+ "file_id": 508,
+ "content": "The code defines a class for a ResNet-TSM backbone model, with the forward function applying temporal shifts and convolutions. The BasicBlock class is used for the basic building block of the network, with optional shortcut connections.",
+ "type": "comment"
+ },
+ "6826": {
+ "file_id": 508,
+ "content": " self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n filter_size=3,\n act=None,\n name=name + \"_branch2b\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n filter_size=1,\n stride=stride,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(short, conv1)\n y = F.leaky_relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTSM_MRI(nn.Layer):\n \"\"\"ResNet TSM backbone.\n Args:\n depth (int): Depth of resnet model.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:167-200"
+ },
+ "6827": {
+ "file_id": 508,
+ "content": "This code defines a ResNet TSM backbone model with Batch Normalization and Leaky ReLU activation. It includes a ConvBNLayer for the branch2b, and an optional shortcut connection depending on the input. The forward function performs convolution, adds the shortcut, applies leaky ReLU activation, and returns the result. The ResNetTSM_MRI class is registered with BACKBONES.",
+ "type": "comment"
+ },
+ "6828": {
+ "file_id": 508,
+ "content": " pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self, depth, num_seg=8, pretrained=None, in_channels=1):\n super(ResNetTSM_MRI, self).__init__()\n self.pretrained = pretrained\n self.layers = depth\n self.num_seg = num_seg\n self.in_channels = in_channels\n supported_layers = [18, 34, 50, 101, 152]\n assert self.layers in supported_layers, \\\n \"supported layers are {} but input layer is {}\".format(\n supported_layers, self.layers)\n if self.layers == 18:\n depth = [2, 2, 2, 2]\n elif self.layers == 34 or self.layers == 50:\n depth = [3, 4, 6, 3]\n elif self.layers == 101:\n depth = [3, 4, 23, 3]\n elif self.layers == 152:\n depth = [3, 8, 36, 3]\n in_channels = 64\n out_channels = [64, 128, 256, 512]\n #ResNet-C: use three 3x3 conv, replace, one 7x7 conv\n self.conv1_1 = ConvBNLayer(in_channels=self.in_channels,\n out_channels=32,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:201-229"
+ },
+ "6829": {
+ "file_id": 508,
+ "content": "This code defines a ResNetTSM_MRI class with parameters for depth, num_seg, pretrained (default None), and in_channels. It checks if the input layer is supported, sets the depth based on the input layer, sets out channels, and initializes ConvBNLayer instances accordingly.",
+ "type": "comment"
+ },
+ "6830": {
+ "file_id": 508,
+ "content": " kernel_size=3,\n stride=2,\n act='leaky_relu',\n name=\"conv1_1\")\n self.conv1_2 = ConvBNLayer(in_channels=32,\n out_channels=32,\n kernel_size=3,\n stride=1,\n act='leaky_relu',\n name=\"conv1_2\")\n self.conv1_3 = ConvBNLayer(in_channels=32,\n out_channels=64,\n kernel_size=3,\n stride=1,\n act='leaky_relu',\n name=\"conv1_3\")\n self.pool2D_max = MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.block_list = []\n if self.layers >= 50:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:230-252"
+ },
+ "6831": {
+ "file_id": 508,
+ "content": "This code defines the ResNet-TSM backbone architecture in PaddleVideo. It includes multiple ConvBNLayer instances for different stages of feature extraction and a MaxPool2D layer for downsampling. The depth of each block is specified by the provided depth list, with shortcut connections determined based on the number of layers specified.",
+ "type": "comment"
+ },
+ "6832": {
+ "file_id": 508,
+ "content": " if self.layers in [101, 152] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n 'bb_%d_%d' %\n (block, i), #same with PaddleClas, for loading pretrain\n BottleneckBlock(\n in_channels=in_channels\n if i == 0 else out_channels[block] * 4,\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n num_seg=self.num_seg,\n shortcut=shortcut,\n if_first=block == i == 0,\n name=conv_name))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:253-271"
+ },
+ "6833": {
+ "file_id": 508,
+ "content": "The code dynamically assigns a name to the BottleneckBlock based on its block and layer indices. If the layers are 101 or 152 at block 2, it further distinguishes between convolutions 'a' and 'b'. The 'bb_%d_%d' naming is used for loading pre-trained models. The BottleneckBlock parameters include in_channels based on if i == 0 or not, out_channels of the block, stride depending on if it's the first layer or not, num_seg for segmentation, shortcut type, and a flag for if it's the first layer.",
+ "type": "comment"
+ },
+ "6834": {
+ "file_id": 508,
+ "content": " in_channels = out_channels[block] * 4\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n conv_name,\n BasicBlock(in_channels=in_channels[block]\n if i == 0 else out_channels[block],\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n name=conv_name))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):\n \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:272-294"
+ },
+ "6835": {
+ "file_id": 508,
+ "content": "This code defines a ResNet TSM backbone with multiple blocks and basic blocks. It dynamically creates convolutional layers using the add_sublayer function. The number of blocks and their configuration is defined by the input parameters \"depth\". Shortcuts are used between layers, and the function init_weights initializes the parameters of the backbone.",
+ "type": "comment"
+ },
+ "6836": {
+ "file_id": 508,
+ "content": " 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n #XXX: check bias!!! check pretrained!!!\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n #XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n \"\"\"Define how the backbone is going to run.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:295-311"
+ },
+ "6837": {
+ "file_id": 508,
+ "content": "This code initializes the backbone of a neural network. If a pretrained path is not specified, it uses specific initialization methods for Conv2D and BatchNorm2d layers. The KaimingNormal function initializes the Conv2D layer, while the Constant function with value 1 initializes the BatchNorm2d layer.",
+ "type": "comment"
+ },
+ "6838": {
+ "file_id": 508,
+ "content": " \"\"\"\n #NOTE: Already merge axis 0(batches) and axis 1(channels) before extracting feature phase,\n # please refer to paddlevideo/modeling/framework/recognizers/recognizer2d.py#L27\n #y = paddle.reshape(\n # inputs, [-1, inputs.shape[2], inputs.shape[3], inputs.shape[4]])\n ####ResNet-C: use three 3x3 conv, replace, one 7x7 conv\n y = self.conv1_1(inputs)\n y = self.conv1_2(y)\n y = self.conv1_3(y)\n y = self.pool2D_max(y)\n for block in self.block_list:\n y = block(y)\n return y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsm_MRI.py:313-327"
+ },
+ "6839": {
+ "file_id": 508,
+ "content": "Reshaping and applying convolutional layers, max pooling, and iterating through a list of blocks to perform operations on the input feature map.",
+ "type": "comment"
+ },
+ "6840": {
+ "file_id": 509,
+ "content": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py",
+ "type": "filepath"
+ },
+ "6841": {
+ "file_id": 509,
+ "content": "This code imports libraries, defines a ResNet-TSN model with basic and bottleneck blocks in PaddlePaddle, initializes weights for training, and outputs results.",
+ "type": "summary"
+ },
+ "6842": {
+ "file_id": 509,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nimport paddle\nfrom paddle import ParamAttr\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom paddle.nn import Conv2D, BatchNorm\nfrom paddle.nn import MaxPool2D, AvgPool2D\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:1-29"
+ },
+ "6843": {
+ "file_id": 509,
+ "content": "This code is for importing necessary libraries, defining a ResNet-TSN backbone model in PaddlePaddle, and registering it to the BACKBONES registry. It also includes license information and mentions function-level future imports for compatibility and division settings. The code initializes parameters, defines Conv2D, BatchNorm, MaxPool2D, AvgPool2D layers, and sets up weight initialization functions and loading checkpoints utilities.",
+ "type": "comment"
+ },
+ "6844": {
+ "file_id": 509,
+ "content": "__all__ = [\"ResNetTSN_MRI\"]\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n is_tweaks_mode=False,\n act=None,\n lr_mult=1.0,\n name=None):\n super(ConvBNLayer, self).__init__()\n self.is_tweaks_mode = is_tweaks_mode\n self._pool2d_avg = AvgPool2D(kernel_size=2,\n stride=2,\n padding=0,\n ceil_mode=True)\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\",\n learning_rate=lr_mult),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:31-58"
+ },
+ "6845": {
+ "file_id": 509,
+ "content": "Defines a ConvBNLayer class with an average pooling layer and convolutional layer. The class takes input parameters for channels, kernel size, stride, groups, and more. It initializes the layers and sets is_tweaks_mode flag.",
+ "type": "comment"
+ },
+ "6846": {
+ "file_id": 509,
+ "content": " bias_attr=False)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._batch_norm = BatchNorm(\n out_channels,\n act=act,\n param_attr=ParamAttr(name=bn_name + '_scale',\n learning_rate=lr_mult,\n regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(bn_name + '_offset',\n learning_rate=lr_mult,\n regularizer=L2Decay(0.0)),\n moving_mean_name=bn_name + '_mean',\n moving_variance_name=bn_name + '_variance')\n def forward(self, inputs):\n if self.is_tweaks_mode:\n inputs = self._pool2d_avg(inputs)\n y = self._conv(inputs)\n y = self._batch_norm(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:59-89"
+ },
+ "6847": {
+ "file_id": 509,
+ "content": "This code defines a class for Resnet_TSN, which is a type of backbone model. It includes an initialization function that initializes the BatchNorm layer and a forward function that applies pooling (if in tweaks mode), convolution, and batch normalization to inputs. Additionally, there is a BottleneckBlock class defined for creating bottleneck blocks within the network.",
+ "type": "comment"
+ },
+ "6848": {
+ "file_id": 509,
+ "content": " if_first=False,\n lr_mult=1.0,\n name=None):\n super(BottleneckBlock, self).__init__()\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2b\")\n self.conv2 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n act=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:90-111"
+ },
+ "6849": {
+ "file_id": 509,
+ "content": "The code defines a BottleneckBlock class, which is a layer in the ResNet model. It consists of three ConvBNLayer layers with different properties such as kernel size, stride, and activation functions. The class initializes these layers and takes input and output channel counts, learning rate multiplier, and name as parameters.",
+ "type": "comment"
+ },
+ "6850": {
+ "file_id": 509,
+ "content": " lr_mult=lr_mult,\n name=name + \"_branch2c\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n lr_mult=lr_mult,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n y = F.relu(y)\n return y\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:112-144"
+ },
+ "6851": {
+ "file_id": 509,
+ "content": "The code defines a ResNet TSN backbone with two branches, where the first branch contains convolutional layers and the second branch has a shortcut connection. The forward function performs addition between the shortcut connection and the output of the convolutional layers. The BasicBlock class is a subclass for implementing basic building blocks.",
+ "type": "comment"
+ },
+ "6852": {
+ "file_id": 509,
+ "content": " shortcut=True,\n if_first=False,\n lr_mult=1.0,\n name=None):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n act=None,\n lr_mult=lr_mult,\n name=name + \"_branch2b\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:145-167"
+ },
+ "6853": {
+ "file_id": 509,
+ "content": "This code defines a BasicBlock class in PaddleVideo for ResNet TSN MRI model. It has an input, output channels, and stride. The class initializes convolution layers (conv0 and conv1) with specified parameters. If shortcut is not set, it also includes a ConvBNLayer as the 'short' attribute.",
+ "type": "comment"
+ },
+ "6854": {
+ "file_id": 509,
+ "content": " kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n lr_mult=lr_mult,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv1)\n y = F.relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTSN_MRI(nn.Layer):\n \"\"\"ResNetTweaksTSN backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self,\n layers=50,\n pretrained=None,\n lr_mult_list=[1.0, 1.0, 1.0, 1.0, 1.0],\n in_channels=1):\n super(ResNetTSN_MRI, self).__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:168-202"
+ },
+ "6855": {
+ "file_id": 509,
+ "content": "This code defines a ResNetTSN_MRI backbone with specified depth, pretrained model option, and learning rate multipliers for each layer. The forward function performs convolutions and shortcut connections, applying ReLU activation at the end. This backbone is registered in BACKBONES for use in the PaddleVideo library.",
+ "type": "comment"
+ },
+ "6856": {
+ "file_id": 509,
+ "content": " self.pretrained = pretrained\n self.layers = layers\n supported_layers = [18, 34, 50, 101, 152, 200]\n assert layers in supported_layers, \\\n \"supported layers are {} but input layer is {}\".format(\n supported_layers, layers)\n self.lr_mult_list = lr_mult_list\n self.in_channels = in_channels\n assert isinstance(\n self.lr_mult_list,\n (list, tuple\n )), \"lr_mult_list should be in (list, tuple) but got {}\".format(\n type(self.lr_mult_list))\n assert len(\n self.lr_mult_list\n ) == 5, \"lr_mult_list length should should be 5 but got {}\".format(\n len(self.lr_mult_list))\n if layers == 18:\n depth = [2, 2, 2, 2]\n elif layers == 34 or layers == 50:\n depth = [3, 4, 6, 3]\n elif layers == 101:\n depth = [3, 4, 23, 3]\n elif layers == 152:\n depth = [3, 8, 36, 3]\n elif layers == 200:\n depth = [3, 12, 48, 3]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:204-232"
+ },
+ "6857": {
+ "file_id": 509,
+ "content": "This code initializes a ResNet TSN backbone model with specified layers, in_channels, and pretrained weight option. It supports specific layer options (18, 34, 50, 101, 152, 200) and checks if the input layer is within supported range. The code also ensures lr_mult_list is a list or tuple and has a length of 5. Depending on the layers, it assigns depth values for each block in the model.",
+ "type": "comment"
+ },
+ "6858": {
+ "file_id": 509,
+ "content": " num_channels = [64, 256, 512, 1024\n ] if layers >= 50 else [64, 64, 128, 256]\n num_filters = [64, 128, 256, 512]\n self.conv1_1 = ConvBNLayer(in_channels=self.in_channels,\n out_channels=32,\n kernel_size=3,\n stride=2,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_1\")\n self.conv1_2 = ConvBNLayer(in_channels=32,\n out_channels=32,\n kernel_size=3,\n stride=1,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_2\")\n self.conv1_3 = ConvBNLayer(in_channels=32,\n out_channels=64,\n kernel_size=3,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:233-253"
+ },
+ "6859": {
+ "file_id": 509,
+ "content": "This code defines a ResNet model for Temporal Segment Networks (TSN) with multiple branch inputs. It initializes the layers of the network, including convolutional and batch normalization operations. The number of channels and filters used in each layer depend on the total number of layers specified. Different learning rate multipliers are assigned to each layer for efficient training.",
+ "type": "comment"
+ },
+ "6860": {
+ "file_id": 509,
+ "content": " stride=1,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_3\")\n self.pool2d_max = MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.block_list = []\n if layers >= 50:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n if layers in [101, 152, 200] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n 'bb_%d_%d' % (block, i),\n BottleneckBlock(\n in_channels=num_channels[block]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:254-275"
+ },
+ "6861": {
+ "file_id": 509,
+ "content": "Initializing layers of ResNet-TSN with specified depth, creating bottleneck blocks for each layer. If layers are 101, 152 or 200 and block is 2, specific naming convention applied. BottleneckBlock is added as sublayer in a sequential manner.",
+ "type": "comment"
+ },
+ "6862": {
+ "file_id": 509,
+ "content": " if i == 0 else num_filters[block] * 4,\n out_channels=num_filters[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n if_first=block == i == 0,\n lr_mult=self.lr_mult_list[block + 1],\n name=conv_name))\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n 'bb_%d_%d' % (block, i),\n BasicBlock(in_channels=num_channels[block]\n if i == 0 else num_filters[block],\n out_channels=num_filters[block],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:276-294"
+ },
+ "6863": {
+ "file_id": 509,
+ "content": "The code creates a ResNet TSN model with bottleneck and basic blocks. It initializes the block_list by adding each block, sets shortcut to True for the first block of each stage, and appends each block to block_list. The number of filters, out_channels, stride, and other parameters are determined based on the stage and block indexes. The name of each block is also specified according to its position in the network.",
+ "type": "comment"
+ },
+ "6864": {
+ "file_id": 509,
+ "content": " stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n if_first=block == i == 0,\n name=conv_name,\n lr_mult=self.lr_mult_list[block + 1]))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):\n \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be\n initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n # XXX: check bias!!! check pretrained!!!",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:295-311"
+ },
+ "6865": {
+ "file_id": 509,
+ "content": "This code initializes the weights of a ResNet TSN backbone model. It creates blocks with specified parameters and appends them to the block list. The `init_weights` function initializes the parameters based on whether pretrained loading path is indicated or not, following specific initialization functions for Conv2D and BatchNorm2d layers.",
+ "type": "comment"
+ },
+ "6866": {
+ "file_id": 509,
+ "content": " if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n # XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n y = self.conv1_1(inputs)\n y = self.conv1_2(y)\n y = self.conv1_3(y)\n y = self.pool2d_max(y)\n for block in self.block_list:\n y = block(y)\n return y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tsn_MRI.py:313-331"
+ },
+ "6867": {
+ "file_id": 509,
+ "content": "This code is checking if the pretrained model path is provided and initializing weights for Conv2D and BatchNorm2D layers if not. The forward function performs convolutions, max pooling, and processes through blocks to output a result.",
+ "type": "comment"
+ },
+ "6868": {
+ "file_id": 510,
+ "content": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py",
+ "type": "filepath"
+ },
+ "6869": {
+ "file_id": 510,
+ "content": "The code defines a TSM ResNet backbone class for feature extraction in Temporal Segment Networks, with customizable depth, segments, and pretrained options. It applies temporal shift modules and convolutions across various ResNet models (18-152 layers).",
+ "type": "summary"
+ },
+ "6870": {
+ "file_id": 510,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport math\nimport sys\nimport paddle\nimport paddle.nn as nn\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout, MaxPool2D,\n AvgPool2D)\nfrom paddle import ParamAttr\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils.save_load import load_ckpt\n# Download URL of pretrained model",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:1-31"
+ },
+ "6871": {
+ "file_id": 510,
+ "content": "This code imports necessary libraries, defines a class for a TSM ResNet backbone, and includes functions for loading pre-trained models. The code also contains a license notice and copyright information.",
+ "type": "comment"
+ },
+ "6872": {
+ "file_id": 510,
+ "content": "# {\n# \"ResNet50_vd\":\n# \"wget https://videotag.bj.bcebos.com/PaddleVideo/PretrainModel/ResNet50_vd_ssld_v2_pretrained.pdparams\",\n# \"ResNet101_vd\":\n# \"https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ResNet101_vd_ssld_pretrained.pdparams\",\n# \"ResNet18_vd\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/ResNet18_vd_pretrained.pdparams\",\n# \"ResNet34_vd\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ResNet34_vd_ssld_pretrained.pdparams\",\n# \"ResNet152_vd\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/ResNet152_vd_pretrained.pdparams\",\n# \"ResNet200_vd\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/ResNet200_vd_pretrained.pdparams\",\n# }\nclass ConvBNLayer(nn.Layer):\n \"\"\"Conv2D and BatchNorm2D layer.\n Args:\n in_channels (int): Number of channels for the input.\n out_channels (int): Number of channels for the output.\n kernel_size (int): Kernel size.\n stride (int): Stride in the Conv2D layer. Default: 1.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:32-54"
+ },
+ "6873": {
+ "file_id": 510,
+ "content": "This code defines a class called \"ConvBNLayer\" that combines Conv2D and BatchNorm2D layers, taking input and output channel counts, kernel size, and stride as arguments. It also includes a dictionary of pre-trained model URLs for ResNet variations.",
+ "type": "comment"
+ },
+ "6874": {
+ "file_id": 510,
+ "content": " groups (int): Groups in the Conv2D, Default: 1.\n is_tweaks_mode (bool): switch for tweaks. Default: False.\n act (str): Indicate activation after BatchNorm2D layer.\n name (str): the name of an instance of ConvBNLayer.\n \"\"\"\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n is_tweaks_mode=False,\n act=None,\n name=None):\n super(ConvBNLayer, self).__init__()\n self.is_tweaks_mode = is_tweaks_mode\n #ResNet-D 1/2:add a 2×2 average pooling layer with a stride of 2 before the convolution,\n # whose stride is changed to 1, works well in practice.\n self._pool2d_avg = AvgPool2D(kernel_size=2,\n stride=2,\n padding=0,\n ceil_mode=True)\n self._conv = Conv2D(in_channels=in_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:55-78"
+ },
+ "6875": {
+ "file_id": 510,
+ "content": "The code defines a class \"ConvBNLayer\" with parameters such as in_channels, out_channels, kernel_size, stride, groups, is_tweaks_mode, act, and name. It also adds an average pooling layer before the convolution for ResNet-D 1/2 tweak, which works well in practice.",
+ "type": "comment"
+ },
+ "6876": {
+ "file_id": 510,
+ "content": " out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._act = act\n self._batch_norm = BatchNorm2D(\n out_channels,\n weight_attr=ParamAttr(name=bn_name + \"_scale\",\n regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(bn_name + \"_offset\", regularizer=L2Decay(0.0)))\n def forward(self, inputs):\n if self.is_tweaks_mode:\n inputs = self._pool2d_avg(inputs)\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if self._act:\n y = getattr(paddle.nn.functional, self._act)(y)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:79-105"
+ },
+ "6877": {
+ "file_id": 510,
+ "content": "This code defines a Convolutional Neural Network (CNN) layer with Batch Normalization and optionally activation function. The layer can have tweaks mode for pooling and average pooling operations. It also includes a forward method for passing inputs through the defined layers.",
+ "type": "comment"
+ },
+ "6878": {
+ "file_id": 510,
+ "content": " return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n if_first=False,\n num_seg=8,\n name=None):\n super(BottleneckBlock, self).__init__()\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act=\"leaky_relu\",\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act=\"leaky_relu\",\n name=name + \"_branch2b\")\n self.conv2 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels * 4,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:106-132"
+ },
+ "6879": {
+ "file_id": 510,
+ "content": "The code defines a BottleneckBlock class with in_channels, out_channels, stride, shortcut (optional), if_first (boolean), and num_seg as parameters. It initializes ConvBNLayer objects for conv0, conv1, and conv2 layers. The BottleneckBlock is a part of the ResNet architecture with tweaks and TSM.",
+ "type": "comment"
+ },
+ "6880": {
+ "file_id": 510,
+ "content": " kernel_size=1,\n act=None,\n name=name + \"_branch2c\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n self.num_seg = num_seg\n def forward(self, inputs):\n if paddle.is_compiled_with_custom_device('npu'):\n x = inputs\n seg_num = self.num_seg\n shift_ratio = 1.0 / self.num_seg\n shape = x.shape #[N*T, C, H, W]\n reshape_x = x.reshape(\n (-1, seg_num, shape[1], shape[2], shape[3])) #[N, T, C, H, W]\n pad_x = F.pad(reshape_x, [\n 0,\n 0,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:133-159"
+ },
+ "6881": {
+ "file_id": 510,
+ "content": "This code defines a class with an initializer and a forward method. The initializer sets the number of segments (num_seg) and whether to use shortcut connection. The forward method reshapes input, pads it based on segment numbers, and likely performs some computations for Temporal Segment Networks.",
+ "type": "comment"
+ },
+ "6882": {
+ "file_id": 510,
+ "content": " 1,\n 1,\n 0,\n 0,\n 0,\n 0,\n 0,\n 0,\n ]) #[N, T+2, C, H, W]\n c1 = int(shape[1] * shift_ratio)\n c2 = int(shape[1] * 2 * shift_ratio)\n slice1 = pad_x[:, :seg_num, :c1, :, :]\n slice2 = pad_x[:, 2:seg_num + 2, c1:c2, :, :]\n slice3 = pad_x[:, 1:seg_num + 1, c2:, :, :]\n concat_x = paddle.concat([slice1, slice2, slice3],\n axis=2) #[N, T, C, H, W]\n shifts = concat_x.reshape(shape)\n else:\n shifts = F.temporal_shift(inputs, self.num_seg, 1.0 / self.num_seg)\n y = self.conv0(shifts)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n return F.leaky_relu(y)\nclass BasicBlock(nn.Layer):\n def __init__(self,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:160-192"
+ },
+ "6883": {
+ "file_id": 510,
+ "content": "This code defines a function and a class, both part of a ResNet backbone model. The function takes in an input tensor and applies temporal shifts, convolutions, and shortcut connections to form the output. The BasicBlock class initializes a basic block layer with convolutional layers and a shortcut connection.",
+ "type": "comment"
+ },
+ "6884": {
+ "file_id": 510,
+ "content": " in_channels,\n out_channels,\n stride,\n shortcut=True,\n num_seg=8,\n name=None):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.num_seg = num_seg\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act=\"leaky_relu\",\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n act=None,\n name=name + \"_branch2b\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:193-217"
+ },
+ "6885": {
+ "file_id": 510,
+ "content": "This code defines a BasicBlock class for ResNet TSM model, with parameters including input and output channels, stride, shortcut flag, and number of segments. It initializes instance variables and creates convolution layers (conv0, conv1) for feature extraction. If not using shortcut connections, it also initializes a short layer for residual connections.",
+ "type": "comment"
+ },
+ "6886": {
+ "file_id": 510,
+ "content": " stride=stride,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n # add temporal shift module\n shifts = F.temporal_shift(inputs, self.num_seg, 1.0 / self.num_seg)\n y = self.conv0(shifts)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(short, conv1)\n y = F.leaky_relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTweaksTSM(nn.Layer):\n \"\"\"ResNet TSM backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self, depth, num_seg=8, pretrained=None):\n super(ResNetTweaksTSM, self).__init__()\n self.pretrained = pretrained\n self.layers = depth\n self.num_seg = num_seg\n supported_layers = [18, 34, 50, 101, 152]\n assert self.layers in supported_layers, \\",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:218-253"
+ },
+ "6887": {
+ "file_id": 510,
+ "content": "The code defines a ResNet TSM backbone model. It has an init function that initializes the model with specified depth, number of segments, and pretrained options. The forward function applies temporal shift module, convolution, shortcut connection if applicable, and Leaky ReLU activation for feature extraction.",
+ "type": "comment"
+ },
+ "6888": {
+ "file_id": 510,
+ "content": " \"supported layers are {} but input layer is {}\".format(\n supported_layers, self.layers)\n if self.layers == 18:\n depth = [2, 2, 2, 2]\n elif self.layers == 34 or self.layers == 50:\n depth = [3, 4, 6, 3]\n elif self.layers == 101:\n depth = [3, 4, 23, 3]\n elif self.layers == 152:\n depth = [3, 8, 36, 3]\n in_channels = 64\n out_channels = [64, 128, 256, 512]\n #ResNet-C: use three 3x3 conv, replace, one 7x7 conv\n self.conv1_1 = ConvBNLayer(in_channels=3,\n out_channels=32,\n kernel_size=3,\n stride=2,\n act='leaky_relu',\n name=\"conv1_1\")\n self.conv1_2 = ConvBNLayer(in_channels=32,\n out_channels=32,\n kernel_size=3,\n stride=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:254-279"
+ },
+ "6889": {
+ "file_id": 510,
+ "content": "The code defines a ResNet model with different depths based on the specified number of layers. It initializes the layers, including a 7x7 convolution and multiple 3x3 convolutions, as well as Batch Normalization and Leaky ReLU activation functions. The model structure is determined by the input layer size, with supported layers ranging from 18 to 152.",
+ "type": "comment"
+ },
+ "6890": {
+ "file_id": 510,
+ "content": " act='leaky_relu',\n name=\"conv1_2\")\n self.conv1_3 = ConvBNLayer(in_channels=32,\n out_channels=64,\n kernel_size=3,\n stride=1,\n act='leaky_relu',\n name=\"conv1_3\")\n self.pool2D_max = MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.block_list = []\n if self.layers >= 50:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n if self.layers in [101, 152] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:280-301"
+ },
+ "6891": {
+ "file_id": 510,
+ "content": "This code defines a ResNet backbone with TSM tweaks. It includes convolutional layers, Batch Normalization, Leaky ReLU activation functions, and max pooling. The block_list is initialized and the structure of the network adapts depending on the specified layers.",
+ "type": "comment"
+ },
+ "6892": {
+ "file_id": 510,
+ "content": " bottleneck_block = self.add_sublayer(\n 'bb_%d_%d' %\n (block, i), #same with PaddleClas, for loading pretrain\n BottleneckBlock(\n in_channels=in_channels\n if i == 0 else out_channels[block] * 4,\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n num_seg=self.num_seg,\n shortcut=shortcut,\n if_first=block == i == 0,\n name=conv_name))\n in_channels = out_channels[block] * 4\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n in_channels = [64, 64, 128, 256]\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:302-322"
+ },
+ "6893": {
+ "file_id": 510,
+ "content": "Iterates through ResNet blocks and Bottleneck blocks, assigning in_channels based on previous block's out_channels. Inserts each block into the block_list. Adjusts shortcut value accordingly. Initializes in_channels as 64 for specified blocks if depth is not defined.",
+ "type": "comment"
+ },
+ "6894": {
+ "file_id": 510,
+ "content": " basic_block = self.add_sublayer(\n conv_name,\n BasicBlock(in_channels=in_channels[block]\n if i == 0 else out_channels[block],\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n num_seg=self.num_seg,\n name=conv_name))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):\n \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:323-339"
+ },
+ "6895": {
+ "file_id": 510,
+ "content": "This code defines a function for initializing weights in the ResNet TSMBackbone. It loads pre-trained weights if a loading path is specified or uses specific initialization methods otherwise, with Conv2D layers using KaimingNormal and BatchNorm2d layers using Constant initialization.",
+ "type": "comment"
+ },
+ "6896": {
+ "file_id": 510,
+ "content": " Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n # no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n \"\"\"Define how the backbone is going to run.\n \"\"\"\n y = self.conv1_1(inputs)\n y = self.conv1_2(y)\n y = self.conv1_3(y)\n y = self.pool2D_max(y)\n for block in self.block_list:\n y = block(y)\n return y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsm.py:340-362"
+ },
+ "6897": {
+ "file_id": 510,
+ "content": "This function initializes the backbone's weights, with Kaiming Normal distribution for Conv2D layers and constant value 1 for BatchNorm2D. The forward function defines how the backbone processes inputs through a series of convolutions and pooling, then passes the result to each block in the block_list.",
+ "type": "comment"
+ },
+ "6898": {
+ "file_id": 511,
+ "content": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py",
+ "type": "filepath"
+ },
+ "6899": {
+ "file_id": 511,
+ "content": "PaddleVideo library contains ResNet TSN model backbones, licensed under Apache 2.0, includes ConvBNLayer, offers modified ResNet with BatchNorm and pooling layers, uses BottleneckBlock for deeper networks, and initializes configurable parameters with BasicBlock. The code constructs a ResNet backbone, performs forward pass through network, applies convolutions and pooling, and returns output after passing through each block in the block list using input lists to determine layers and filters while initializing model weights.",
+ "type": "summary"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/69.json b/docs/data/69.json
new file mode 100644
index 000000000..977ee33a1
--- /dev/null
+++ b/docs/data/69.json
@@ -0,0 +1,549 @@
+{
+ "6900": {
+ "file_id": 511,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nimport paddle\nfrom paddle import ParamAttr\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom paddle.nn import Conv2D, BatchNorm\nfrom paddle.nn import MaxPool2D, AvgPool2D\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:1-29"
+ },
+ "6901": {
+ "file_id": 511,
+ "content": "This code is a part of the PaddleVideo library, which provides model backbones including ResNet Tweaks TSN. It imports necessary modules and defines functions for creating convolutional layers, batch normalization, pooling layers, initializing weights, and loading checkpoints. The code follows the Apache License 2.0 and is distributed under an \"AS IS\" basis.",
+ "type": "comment"
+ },
+ "6902": {
+ "file_id": 511,
+ "content": "__all__ = [\"ResNetTweaksTSN\"]\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n is_tweaks_mode=False,\n act=None,\n lr_mult=1.0,\n name=None):\n super(ConvBNLayer, self).__init__()\n self.is_tweaks_mode = is_tweaks_mode\n self._pool2d_avg = AvgPool2D(kernel_size=2,\n stride=2,\n padding=0,\n ceil_mode=True)\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\",\n learning_rate=lr_mult),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:31-58"
+ },
+ "6903": {
+ "file_id": 511,
+ "content": "This code defines a ConvBNLayer class with an average pooling operation, a convolution layer, and optional tweaks mode. It also initializes a Conv2D layer and sets parameters for weight attributes and learning rates.",
+ "type": "comment"
+ },
+ "6904": {
+ "file_id": 511,
+ "content": " bias_attr=False)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._batch_norm = BatchNorm(\n out_channels,\n act=act,\n param_attr=ParamAttr(name=bn_name + '_scale',\n learning_rate=lr_mult,\n regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(bn_name + '_offset',\n learning_rate=lr_mult,\n regularizer=L2Decay(0.0)),\n moving_mean_name=bn_name + '_mean',\n moving_variance_name=bn_name + '_variance')\n def forward(self, inputs):\n if self.is_tweaks_mode:\n inputs = self._pool2d_avg(inputs)\n y = self._conv(inputs)\n y = self._batch_norm(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:59-89"
+ },
+ "6905": {
+ "file_id": 511,
+ "content": "The code defines a ResNet backbone with Temporal Segment Network (TSN) modifications. It includes a BatchNorm layer for normalization and has a forward function that applies pooling if in tweaks mode, followed by the batch norm and convolution layers. The BottleneckBlock class is also defined as a sublayer.",
+ "type": "comment"
+ },
+ "6906": {
+ "file_id": 511,
+ "content": " if_first=False,\n lr_mult=1.0,\n name=None):\n super(BottleneckBlock, self).__init__()\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2b\")\n self.conv2 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n act=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:90-111"
+ },
+ "6907": {
+ "file_id": 511,
+ "content": "This code defines the BottleneckBlock class, which is a layer in ResNet backbone. It consists of three ConvBNLayer instances: conv0, conv1, and conv2. The first one performs a 1x1 convolution, while the second one does a 3x3 convolution with stride. Lastly, the third one executes a 1x1 convolution without activation function. This block is designed to reduce parameters for deeper networks.",
+ "type": "comment"
+ },
+ "6908": {
+ "file_id": 511,
+ "content": " lr_mult=lr_mult,\n name=name + \"_branch2c\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n lr_mult=lr_mult,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n y = F.relu(y)\n return y\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:112-144"
+ },
+ "6909": {
+ "file_id": 511,
+ "content": "This code defines a ResNet block with two convolution layers, one optional shortcut connection, and applies ReLU activation after the addition of the branch outputs. The BasicBlock class is used for the basic building block of the network.",
+ "type": "comment"
+ },
+ "6910": {
+ "file_id": 511,
+ "content": " shortcut=True,\n if_first=False,\n lr_mult=1.0,\n name=None):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n act=None,\n lr_mult=lr_mult,\n name=name + \"_branch2b\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:145-167"
+ },
+ "6911": {
+ "file_id": 511,
+ "content": "This code defines a BasicBlock class with convolutional layers and Batch Normalization. It initializes the block's parameters like stride, convolution layers, and batch normalization. The shortcut connection is optional and depends on the 'shortcut' parameter. The 'if_first', 'lr_mult', and 'name' parameters are also provided for customization.",
+ "type": "comment"
+ },
+ "6912": {
+ "file_id": 511,
+ "content": " kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n lr_mult=lr_mult,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv1)\n y = F.relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTweaksTSN(nn.Layer):\n \"\"\"ResNetTweaksTSN backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self,\n layers=50,\n pretrained=None,\n lr_mult_list=[1.0, 1.0, 1.0, 1.0, 1.0]):\n super(ResNetTweaksTSN, self).__init__()\n self.pretrained = pretrained",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:168-203"
+ },
+ "6913": {
+ "file_id": 511,
+ "content": "This code defines a ResNetTweaksTSN backbone for deep learning models. It includes layers such as convolution, shortcut connections, and ReLU activation function. The constructor takes parameters like depth (layers), pretrained model, and learning rate multipliers for different layers.",
+ "type": "comment"
+ },
+ "6914": {
+ "file_id": 511,
+ "content": " self.layers = layers\n supported_layers = [18, 34, 50, 101, 152, 200]\n assert layers in supported_layers, \\\n \"supported layers are {} but input layer is {}\".format(\n supported_layers, layers)\n self.lr_mult_list = lr_mult_list\n assert isinstance(\n self.lr_mult_list,\n (list, tuple\n )), \"lr_mult_list should be in (list, tuple) but got {}\".format(\n type(self.lr_mult_list))\n assert len(\n self.lr_mult_list\n ) == 5, \"lr_mult_list length should should be 5 but got {}\".format(\n len(self.lr_mult_list))\n if layers == 18:\n depth = [2, 2, 2, 2]\n elif layers == 34 or layers == 50:\n depth = [3, 4, 6, 3]\n elif layers == 101:\n depth = [3, 4, 23, 3]\n elif layers == 152:\n depth = [3, 8, 36, 3]\n elif layers == 200:\n depth = [3, 12, 48, 3]\n num_channels = [64, 256, 512, 1024\n ] if layers >= 50 else [64, 64, 128, 256]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:204-232"
+ },
+ "6915": {
+ "file_id": 511,
+ "content": "This code initializes a ResNet backbone with different configurations based on the input layer. It checks if the provided layer is supported, asserts the type and length of the learning rate multiplier list, and assigns depth and number of channels for each layer configuration.",
+ "type": "comment"
+ },
+ "6916": {
+ "file_id": 511,
+ "content": " num_filters = [64, 128, 256, 512]\n self.conv1_1 = ConvBNLayer(in_channels=3,\n out_channels=32,\n kernel_size=3,\n stride=2,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_1\")\n self.conv1_2 = ConvBNLayer(in_channels=32,\n out_channels=32,\n kernel_size=3,\n stride=1,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_2\")\n self.conv1_3 = ConvBNLayer(in_channels=32,\n out_channels=64,\n kernel_size=3,\n stride=1,\n act='relu',\n lr_mult=self.lr_mult_list[0],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:233-254"
+ },
+ "6917": {
+ "file_id": 511,
+ "content": "This code defines the first layer of the ResNet backbone, including three ConvBNLayer instances for different operations. The first layer consists of a 2x downsampling convolution, followed by two 1x1 convolutions to reduce dimensionality and apply relu activation. Lr_mult ensures that these layers are trained with different learning rates based on their importance.",
+ "type": "comment"
+ },
+ "6918": {
+ "file_id": 511,
+ "content": " name=\"conv1_3\")\n self.pool2d_max = MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.block_list = []\n if layers >= 50:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n if layers in [101, 152, 200] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n 'bb_%d_%d' % (block, i),\n BottleneckBlock(\n in_channels=num_channels[block]\n if i == 0 else num_filters[block] * 4,\n out_channels=num_filters[block],\n stride=2 if i == 0 and block != 0 else 1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:255-276"
+ },
+ "6919": {
+ "file_id": 511,
+ "content": "This code defines a ResNet backbone with optional Temporal Segment Network (TSN) modifications. It adds BottleneckBlock layers, specifies pooling operations, and handles shortcut connections for blocks 0-56. The number of layers and filters are determined by the provided depth and num_filters lists.",
+ "type": "comment"
+ },
+ "6920": {
+ "file_id": 511,
+ "content": " shortcut=shortcut,\n if_first=block == i == 0,\n lr_mult=self.lr_mult_list[block + 1],\n name=conv_name))\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n 'bb_%d_%d' % (block, i),\n BasicBlock(in_channels=num_channels[block]\n if i == 0 else num_filters[block],\n out_channels=num_filters[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n if_first=block == i == 0,\n name=conv_name,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:277-296"
+ },
+ "6921": {
+ "file_id": 511,
+ "content": "This code adds layers to the ResNet backbone model. It uses conditional statements and loops to determine the number of layers added at each block based on a given depth configuration, and applies different configurations for the first block. Layers are added with specific parameters such as in_channels, out_channels, stride, shortcut, if_first flag, and name.",
+ "type": "comment"
+ },
+ "6922": {
+ "file_id": 511,
+ "content": " lr_mult=self.lr_mult_list[block + 1]))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):\n \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be\n initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n # XXX: check bias!!! check pretrained!!!\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:297-314"
+ },
+ "6923": {
+ "file_id": 511,
+ "content": "This code initializes a backbone model and handles the loading of pre-trained weights. If pre-trained path is specified, it loads the weights; otherwise, it follows specific initialization for Conv2D layers and BatchNorm2d using KaimingNormal and Constant functions respectively.",
+ "type": "comment"
+ },
+ "6924": {
+ "file_id": 511,
+ "content": " if isinstance(layer, nn.Conv2D):\n # XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n y = self.conv1_1(inputs)\n y = self.conv1_2(y)\n y = self.conv1_3(y)\n y = self.pool2d_max(y)\n for block in self.block_list:\n y = block(y)\n return y",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnet_tweaks_tsn.py:315-328"
+ },
+ "6925": {
+ "file_id": 511,
+ "content": "This code initializes the weights of convolutional layers without bias and batch normalization layers with constant value 1. It then performs forward pass through the network, applying convolutions and pooling operations. The output is returned after passing through each block in the block list.",
+ "type": "comment"
+ },
+ "6926": {
+ "file_id": 512,
+ "content": "/paddlevideo/modeling/backbones/resnext101.py",
+ "type": "filepath"
+ },
+ "6927": {
+ "file_id": 512,
+ "content": "The code defines a ResNeXt-101 model in PaddlePaddle, including downsample and residual blocks, BottleneckBlock, performs convolutions, activation, max pooling on input image.",
+ "type": "summary"
+ },
+ "6928": {
+ "file_id": 512,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom functools import partial\nimport paddle\nclass ConvBNLayer(paddle.nn.Layer):\n def __init__(self,\n num_channels,\n num_filters,\n filter_size,\n stride=1,\n padding=0,\n dilation=1,\n groups=1,\n padding_mode='zeros',\n weight_attr=None,\n bias_attr=None,\n name=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:1-31"
+ },
+ "6929": {
+ "file_id": 512,
+ "content": "This code defines a ConvBNLayer class in PaddlePaddle, which is a convolution-batch normalization layer. It takes inputs like num_channels, num_filters, filter_size, stride, padding, dilation, groups, padding_mode, weight_attr, bias_attr, and name.",
+ "type": "comment"
+ },
+ "6930": {
+ "file_id": 512,
+ "content": " data_format=\"NCDHW\"):\n super(ConvBNLayer, self).__init__()\n self._conv = paddle.nn.Conv3D(\n in_channels=num_channels,\n out_channels=num_filters,\n kernel_size=filter_size,\n stride=stride,\n padding=padding,\n dilation=dilation,\n groups=groups,\n padding_mode=padding_mode,\n weight_attr=paddle.ParamAttr(initializer=paddle.nn.initializer.KaimingNormal(\n fan_in=num_filters * filter_size * filter_size), name=name+'_weights'),\n bias_attr=bias_attr,\n data_format=data_format)\n bn_name = \"bn_\" + name\n self._batch_norm = paddle.nn.BatchNorm3D(\n num_filters,\n momentum=0.9,\n epsilon=1e-05,\n weight_attr=paddle.ParamAttr(initializer=paddle.nn.initializer.Constant(\n 1.), name=bn_name + '_scale'),\n bias_attr=paddle.ParamAttr(initializer=paddle.nn.initializer.Constant(\n 0.), name=bn_name + '_offset'),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:32-55"
+ },
+ "6931": {
+ "file_id": 512,
+ "content": "This code defines a ConvBNLayer class with specified parameters for convolutional and batch normalization layers. The convolutional layer uses Kaiming Normal initialization, while the batch normalization layer has fixed scales and offsets initialized to 1 and 0 respectively.",
+ "type": "comment"
+ },
+ "6932": {
+ "file_id": 512,
+ "content": " data_format=data_format)\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n return y\ndef _downsample_basic_block(self, x, planes, stride):\n out = paddle.nn.functional.avg_pool3d(x, kernel_size=1, stride=stride)\n shape = out.shape\n zero_pads = paddle.zeros(shape=[shape[0], planes - shape[1], shape[2], shape[3], shape[4]],\n dtype='float32')\n out = paddle.concat(x=[out, zero_pads], axis=1)\nclass BottleneckBlock(paddle.nn.Layer):\n expansion = 2\n def __init__(self, inplanes, planes, cardinality, stride=1, downsample=None, name=None):\n super(BottleneckBlock, self).__init__()\n mid_planes = cardinality * int(planes / 32)\n self.conv0 = ConvBNLayer(\n inplanes, mid_planes, filter_size=1, bias_attr=False, name=name+'_branch2a')\n self.conv1 = ConvBNLayer(mid_planes, mid_planes, filter_size=3, stride=stride,\n padding=1, groups=cardinality, bias_attr=False, name=name+'_branch2b')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:56-82"
+ },
+ "6933": {
+ "file_id": 512,
+ "content": "This code defines a BottleneckBlock class and a downsample function for the ResNeXt101 model in PaddlePaddle. The BottleneckBlock has an expansion factor of 2 and uses ConvBNLayer for convolution and batch normalization. The downsample function performs average pooling and concatenation to perform downsampling.",
+ "type": "comment"
+ },
+ "6934": {
+ "file_id": 512,
+ "content": " self.conv2 = ConvBNLayer(mid_planes, planes * self.expansion,\n filter_size=1, bias_attr=False, name=name+'_branch2c')\n self.downsample = downsample\n self.stride = stride\n self.relu = paddle.nn.ReLU()\n def forward(self, x):\n residual = x\n out = self.conv0(x)\n out = self.relu(out)\n out = self.conv1(out)\n out = self.relu(out)\n out = self.conv2(out)\n if self.downsample is not None:\n residual = self.downsample(x)\n out += residual\n out = self.relu(out)\n return out\nclass ResNeXt(paddle.nn.Layer):\n def __init__(self,\n block,\n layers,\n shortcut_type='B',\n cardinality=32):\n self.inplanes = 64\n super(ResNeXt, self).__init__()\n self.conv = ConvBNLayer(\n 3,\n 64,\n filter_size=7,\n stride=(1, 2, 2),\n padding=(3, 3, 3),\n bias_attr=False,\n name=\"res_conv1\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:83-122"
+ },
+ "6935": {
+ "file_id": 512,
+ "content": "This code defines a ResNeXt model. The class ResNeXt has an initialization that sets inplanes to 64 and inherits from paddle.nn.Layer. It contains a convolution layer (conv) with 3 input channels, 64 output channels, filter size of 7, and stride of (1,2,2). The class also includes a ResNet-style residual block as a member variable named 'block'. It has layers, shortcut_type (defaults to B), and cardinality parameters.",
+ "type": "comment"
+ },
+ "6936": {
+ "file_id": 512,
+ "content": " )\n self.relu = paddle.nn.ReLU()\n self.maxpool = paddle.nn.MaxPool3D(kernel_size=(3, 3, 3), stride=2, padding=1)\n self.layer1 = self._make_layer(block, 128, layers[0], shortcut_type,\n cardinality, stride=1, name='layer1')\n self.layer2 = self._make_layer(\n block, 256, layers[1], shortcut_type, cardinality, stride=2, name='layer2')\n self.layer3 = self._make_layer(\n block, 512, layers[2], shortcut_type, cardinality, stride=2, name='layer3')\n self.layer4 = self._make_layer(\n block, 1024, layers[3], shortcut_type, cardinality, stride=2, name='layer4')\n self.avgpool = paddle.nn.AvgPool3D((2, 1, 1), stride=1, exclusive=False)\n def _make_layer(self,\n block,\n planes,\n blocks,\n shortcut_type,\n cardinality,\n stride=1,\n name=None):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:123-148"
+ },
+ "6937": {
+ "file_id": 512,
+ "content": "The code defines a ResNext101 backbone for a deep learning model. It includes a ReLU activation function and max pooling operation, followed by four residual layers (layer1 to layer4) with varying numbers of planes (256, 512, 1024 respectively). The _make_layer method is used to create the layers, with options for downsampling and varying expansion rates.",
+ "type": "comment"
+ },
+ "6938": {
+ "file_id": 512,
+ "content": " if shortcut_type == 'A':\n downsample = partial(self._downsample_basic_block,\n planes=planes * block.expansion,\n stride=stride)\n else:\n downsample = ConvBNLayer(\n self.inplanes,\n planes * block.expansion,\n 1,\n stride=stride,\n bias_attr=False,\n name=name+'downsample'\n )\n layers = []\n layers.append(\n block(self.inplanes, planes, cardinality, stride, downsample, name=name+'_downsample'))\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(block(self.inplanes, planes,\n cardinality, name=name+'_res_block'+str(i)))\n return paddle.nn.Sequential(*layers)\n def forward(self, x):\n x = self.conv(x)\n x = self.relu(x)\n x = self.maxpool(x)\n x = self.layer1(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:149-176"
+ },
+ "6939": {
+ "file_id": 512,
+ "content": "This code defines a ResNeXt-101 model, implementing its downsample and residual blocks. It takes an input image, performs convolutions, applies ReLU activation, and max pooling before passing through the specified number of residual blocks.",
+ "type": "comment"
+ },
+ "6940": {
+ "file_id": 512,
+ "content": " x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n return x\ndef ResNext101():\n \"\"\"Constructs a ResNext-101 model.\n \"\"\"\n model = ResNeXt(BottleneckBlock, [3, 4, 23, 3])\n return model",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/resnext101.py:177-187"
+ },
+ "6941": {
+ "file_id": 512,
+ "content": "The ResNext101 function constructs a ResNeXt-101 model using BottleneckBlock and the specified block configurations. It applies the layer2, layer3, and layer4 operations to x before returning the result.",
+ "type": "comment"
+ },
+ "6942": {
+ "file_id": 513,
+ "content": "/paddlevideo/modeling/backbones/stgcn.py",
+ "type": "filepath"
+ },
+ "6943": {
+ "file_id": 513,
+ "content": "The code incorporates image processing and Graph class, supports layouts like 'stgcn' and 'coco_keypoint'. It defines a STGCN model for spatio-temporal data processing using ConvTemporalGraphical layer. The code creates a STGCN class for skeleton-based action recognition with edge importance, applies networks, pools results, and averages before returning output.",
+ "type": "summary"
+ },
+ "6944": {
+ "file_id": 513,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\ndef zero(x):\n return 0\ndef iden(x):\n return x\ndef einsum(x, A):\n \"\"\"paddle.einsum will be implemented in release/2.2.\n \"\"\"\n x = x.transpose((0, 2, 3, 1, 4))\n n, c, t, k, v = x.shape\n k2, v2, w = A.shape\n assert (k == k2 and v == v2), \"Args of einsum not match!\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:1-37"
+ },
+ "6945": {
+ "file_id": 513,
+ "content": "This code snippet imports necessary libraries, defines several functions (zero, iden, einsum), and registers the BACKBONES. The purpose of this module seems to be defining backbone architectures or functions used in image processing tasks. However, more context is needed to understand the specific functionality of these functions or their use within the BACKBONES registry.",
+ "type": "comment"
+ },
+ "6946": {
+ "file_id": 513,
+ "content": " x = x.reshape((n, c, t, k * v))\n A = A.reshape((k * v, w))\n y = paddle.matmul(x, A)\n return y\ndef get_hop_distance(num_node, edge, max_hop=1):\n A = np.zeros((num_node, num_node))\n for i, j in edge:\n A[j, i] = 1\n A[i, j] = 1\n # compute hop steps\n hop_dis = np.zeros((num_node, num_node)) + np.inf\n transfer_mat = [np.linalg.matrix_power(A, d) for d in range(max_hop + 1)]\n arrive_mat = (np.stack(transfer_mat) > 0)\n for d in range(max_hop, -1, -1):\n hop_dis[arrive_mat[d]] = d\n return hop_dis\ndef normalize_digraph(A):\n Dl = np.sum(A, 0)\n num_node = A.shape[0]\n Dn = np.zeros((num_node, num_node))\n for i in range(num_node):\n if Dl[i] > 0:\n Dn[i, i] = Dl[i]**(-1)\n AD = np.dot(A, Dn)\n return AD\nclass Graph():\n def __init__(self,\n layout='openpose',\n strategy='uniform',\n max_hop=1,\n dilation=1):\n self.max_hop = max_hop\n self.dilation = dilation\n self.get_edge(layout)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:38-80"
+ },
+ "6947": {
+ "file_id": 513,
+ "content": "This code defines a Graph class and three functions: get_hop_distance, normalize_digraph, and a constructor for the Graph class. The Graph class initializes with layout, strategy, max_hop, and dilation parameters. The get_hop_distance function computes hop distances between nodes in a graph up to max_hop level. The normalize_digraph function calculates and applies row-wise node degrees to normalize the adjacency matrix. The constructor initializes an instance of the Graph class with given parameters.",
+ "type": "comment"
+ },
+ "6948": {
+ "file_id": 513,
+ "content": " self.hop_dis = get_hop_distance(self.num_node,\n self.edge,\n max_hop=max_hop)\n self.get_adjacency(strategy)\n def __str__(self):\n return self.A\n def get_edge(self, layout):\n # edge is a list of [child, parent] paris\n if layout == 'fsd10':\n self.num_node = 25\n self_link = [(i, i) for i in range(self.num_node)]\n neighbor_link = [(1, 8), (0, 1), (15, 0), (17, 15), (16, 0),\n (18, 16), (5, 1), (6, 5), (7, 6), (2, 1), (3, 2),\n (4, 3), (9, 8), (10, 9), (11, 10), (24, 11),\n (22, 11), (23, 22), (12, 8), (13, 12), (14, 13),\n (21, 14), (19, 14), (20, 19)]\n self.edge = self_link + neighbor_link\n self.center = 8\n elif layout == 'ntu-rgb+d':\n self.num_node = 25\n self_link = [(i, i) for i in range(self.num_node)]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:81-104"
+ },
+ "6949": {
+ "file_id": 513,
+ "content": "The code initializes the hop distance and edge based on the number of nodes, maximum hops, and layout type. It defines self_link as intra-node connections and neighbor_link as inter-node connections. The center node is determined based on the layout.",
+ "type": "comment"
+ },
+ "6950": {
+ "file_id": 513,
+ "content": " neighbor_1base = [(1, 2), (2, 21), (3, 21), (4, 3), (5, 21), (6, 5),\n (7, 6), (8, 7), (9, 21), (10, 9), (11, 10),\n (12, 11), (13, 1), (14, 13), (15, 14), (16, 15),\n (17, 1), (18, 17), (19, 18), (20, 19), (22, 23),\n (23, 8), (24, 25), (25, 12)]\n neighbor_link = [(i - 1, j - 1) for (i, j) in neighbor_1base]\n self.edge = self_link + neighbor_link\n self.center = 21 - 1\n elif layout == 'coco_keypoint':\n self.num_node = 17\n self_link = [(i, i) for i in range(self.num_node)]\n neighbor_1base = [(0, 1), (0, 2), (1, 3), (2, 4), (3, 5), (4, 6),\n (5, 7), (6, 8), (7, 9), (8, 10), (5, 11), (6, 12),\n (11, 13), (12, 14), (13, 15), (14, 16), (11, 12)]\n neighbor_link = [(i, j) for (i, j) in neighbor_1base]\n self.edge = self_link + neighbor_link",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:105-120"
+ },
+ "6951": {
+ "file_id": 513,
+ "content": "The code initializes 'self.edge' and 'self.center', defining nodes, self-links, and neighboring node links based on the specified layout ('stgcn' or 'coco_keypoint'). For 'stgcn', there are 25 nodes with various connections, while for 'coco_keypoint', there are 17 nodes with specific connections.",
+ "type": "comment"
+ },
+ "6952": {
+ "file_id": 513,
+ "content": " self.center = 11\n else:\n raise ValueError(\"Do Not Exist This Layout.\")\n def get_adjacency(self, strategy):\n valid_hop = range(0, self.max_hop + 1, self.dilation)\n adjacency = np.zeros((self.num_node, self.num_node))\n for hop in valid_hop:\n adjacency[self.hop_dis == hop] = 1\n normalize_adjacency = normalize_digraph(adjacency)\n if strategy == 'spatial':\n A = []\n for hop in valid_hop:\n a_root = np.zeros((self.num_node, self.num_node))\n a_close = np.zeros((self.num_node, self.num_node))\n a_further = np.zeros((self.num_node, self.num_node))\n for i in range(self.num_node):\n for j in range(self.num_node):\n if self.hop_dis[j, i] == hop:\n if self.hop_dis[j, self.center] == self.hop_dis[\n i, self.center]:\n a_root[j, i] = normalize_adjacency[j, i]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:121-143"
+ },
+ "6953": {
+ "file_id": 513,
+ "content": "This function sets the adjacency matrix for STGCN based on the strategy. It initializes the adjacency matrix as a zero matrix, then fills it with 1s for valid hops. The adjacency matrix is normalized using `normalize_digraph`. If the strategy is 'spatial', it iterates over each pair of nodes and populates the adjacency matrix accordingly based on their hop distance from the center node and their hop distance to each other.",
+ "type": "comment"
+ },
+ "6954": {
+ "file_id": 513,
+ "content": " elif self.hop_dis[j, self.center] > self.hop_dis[\n i, self.center]:\n a_close[j, i] = normalize_adjacency[j, i]\n else:\n a_further[j, i] = normalize_adjacency[j, i]\n if hop == 0:\n A.append(a_root)\n else:\n A.append(a_root + a_close)\n A.append(a_further)\n A = np.stack(A)\n self.A = A\n else:\n raise ValueError(\"Do Not Exist This Strategy\")\nclass ConvTemporalGraphical(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n t_kernel_size=1,\n t_stride=1,\n t_padding=0,\n t_dilation=1):\n super().__init__()\n self.kernel_size = kernel_size\n self.conv = nn.Conv2D(in_channels,\n out_channels * kernel_size,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:144-174"
+ },
+ "6955": {
+ "file_id": 513,
+ "content": "This code implements a ConvTemporalGraphical layer, which is a backbone architecture for STGCN. It initializes the ConvTemporalGraphical layer with input and output channels, kernel size, and temporal parameters. The code handles different strategies to build the adjacency matrix (A) for the graph convolution by considering close and further nodes based on hop distance from the central node. If hop == 0, it appends A to a list, otherwise appends both close and further A matrices. Finally, it stacks the A matrices into a numpy array and assigns it to self.A.",
+ "type": "comment"
+ },
+ "6956": {
+ "file_id": 513,
+ "content": " kernel_size=(t_kernel_size, 1),\n padding=(t_padding, 0),\n stride=(t_stride, 1),\n dilation=(t_dilation, 1))\n def forward(self, x, A):\n assert A.shape[0] == self.kernel_size\n x = self.conv(x)\n n, kc, t, v = x.shape\n x = x.reshape((n, self.kernel_size, kc // self.kernel_size, t, v))\n x = einsum(x, A)\n return x, A\nclass st_gcn_block(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n dropout=0,\n residual=True):\n super(st_gcn_block, self).__init__()\n assert len(kernel_size) == 2\n assert kernel_size[0] % 2 == 1\n padding = ((kernel_size[0] - 1) // 2, 0)\n self.gcn = ConvTemporalGraphical(in_channels, out_channels,\n kernel_size[1])\n self.tcn = nn.Sequential(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:175-209"
+ },
+ "6957": {
+ "file_id": 513,
+ "content": "The code defines a ConvTemporalGraphical layer and an STGCNBlock. The ConvTemporalGraphical layer is a 2D convolutional layer with temporal kernel size, padding, stride, and dilation. The STGCNBlock is a residual block that takes input and output channels, temporal kernel size, stride, and dropout as inputs. It initializes the GCN layer and TCN layers sequentially, with the GCN layer performing temporal graph convolution and the TCN layers performing temporal convolutions.",
+ "type": "comment"
+ },
+ "6958": {
+ "file_id": 513,
+ "content": " nn.BatchNorm2D(out_channels),\n nn.ReLU(),\n nn.Conv2D(\n out_channels,\n out_channels,\n (kernel_size[0], 1),\n (stride, 1),\n padding,\n ),\n nn.BatchNorm2D(out_channels),\n nn.Dropout(dropout),\n )\n if not residual:\n self.residual = zero\n elif (in_channels == out_channels) and (stride == 1):\n self.residual = iden\n else:\n self.residual = nn.Sequential(\n nn.Conv2D(in_channels,\n out_channels,\n kernel_size=1,\n stride=(stride, 1)),\n nn.BatchNorm2D(out_channels),\n )\n self.relu = nn.ReLU()\n def forward(self, x, A):\n res = self.residual(x)\n x, A = self.gcn(x, A)\n x = self.tcn(x) + res\n return self.relu(x), A\n@BACKBONES.register()\nclass STGCN(nn.Layer):\n \"\"\"\n ST-GCN model from:\n ",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:210-251"
+ },
+ "6959": {
+ "file_id": 513,
+ "content": "This code defines a STGCN (Spatio-Temporal Graph Convolutional Network) model. It includes layers such as BatchNormalization, ReLU activation, and convolution operations for processing spatial and temporal data. The forward method applies these operations to input features x and adjacency matrix A.",
+ "type": "comment"
+ },
+ "6960": {
+ "file_id": 513,
+ "content": " `\"Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition\" `_\n Args:\n in_channels: int, channels of vertex coordinate. 2 for (x,y), 3 for (x,y,z). Default 2.\n edge_importance_weighting: bool, whether to use edge attention. Default True.\n data_bn: bool, whether to use data BatchNorm. Default True.\n \"\"\"\n def __init__(self,\n in_channels=2,\n edge_importance_weighting=True,\n data_bn=True,\n layout='fsd10',\n strategy='spatial',\n **kwargs):\n super(STGCN, self).__init__()\n self.data_bn = data_bn\n # load graph\n self.graph = Graph(\n layout=layout,\n strategy=strategy,\n )\n A = paddle.to_tensor(self.graph.A, dtype='float32')\n self.register_buffer('A', A)\n # build networks\n spatial_kernel_size = A.shape[0]\n temporal_kernel_size = 9\n kernel_size = (temporal_kernel_size, spatial_kernel_size)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:251-278"
+ },
+ "6961": {
+ "file_id": 513,
+ "content": "This code defines the STGCN (Spatial Temporal Graph Convolutional Networks) class, which is a model for skeleton-based action recognition. It takes arguments like in_channels, edge_importance_weighting, and data_bn to determine the network configuration. It loads graph data and builds networks with specific kernel sizes for spatial and temporal dimensions.",
+ "type": "comment"
+ },
+ "6962": {
+ "file_id": 513,
+ "content": " self.data_bn = nn.BatchNorm1D(in_channels *\n A.shape[1]) if self.data_bn else iden\n kwargs0 = {k: v for k, v in kwargs.items() if k != 'dropout'}\n self.st_gcn_networks = nn.LayerList((\n st_gcn_block(in_channels,\n 64,\n kernel_size,\n 1,\n residual=False,\n **kwargs0),\n st_gcn_block(64, 64, kernel_size, 1, **kwargs),\n st_gcn_block(64, 64, kernel_size, 1, **kwargs),\n st_gcn_block(64, 64, kernel_size, 1, **kwargs),\n st_gcn_block(64, 128, kernel_size, 2, **kwargs),\n st_gcn_block(128, 128, kernel_size, 1, **kwargs),\n st_gcn_block(128, 128, kernel_size, 1, **kwargs),\n st_gcn_block(128, 256, kernel_size, 2, **kwargs),\n st_gcn_block(256, 256, kernel_size, 1, **kwargs),\n st_gcn_block(256, 256, kernel_size, 1, **kwargs),\n ))\n # initialize parameters for edge importance weighting",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:279-300"
+ },
+ "6963": {
+ "file_id": 513,
+ "content": "This code initializes a series of ST-GCN blocks with different configurations for the ST-GCN backbone, including batch normalization and specific layer dimensions. These blocks are stored in a LayerList for flexibility and efficient computation.",
+ "type": "comment"
+ },
+ "6964": {
+ "file_id": 513,
+ "content": " if edge_importance_weighting:\n self.edge_importance = nn.ParameterList([\n self.create_parameter(\n shape=self.A.shape,\n default_initializer=nn.initializer.Constant(1))\n for i in self.st_gcn_networks\n ])\n else:\n self.edge_importance = [1] * len(self.st_gcn_networks)\n self.pool = nn.AdaptiveAvgPool2D(output_size=(1, 1))\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer, 'Normal', mean=0.0, std=0.02)\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Normal', mean=1.0, std=0.02)\n elif isinstance(layer, nn.BatchNorm1D):\n weight_init_(layer, 'Normal', mean=1.0, std=0.02)\n def forward(self, x):\n # data normalization\n N, C, T, V, M = x.shape\n x = x.transpose((0, 4, 3, 1, 2)) # N, M, V, C, T",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:301-327"
+ },
+ "6965": {
+ "file_id": 513,
+ "content": "Code creates edge importance parameters if edge_importance_weighting is True, otherwise sets all edge importances to 1. Initializes weights for convolutional layers and batch normalization layers with specified means and standard deviations. The forward function transposes the input tensor shape before processing.",
+ "type": "comment"
+ },
+ "6966": {
+ "file_id": 513,
+ "content": " x = x.reshape((N * M, V * C, T))\n if self.data_bn:\n x.stop_gradient = False\n x = self.data_bn(x)\n x = x.reshape((N, M, V, C, T))\n x = x.transpose((0, 1, 3, 4, 2)) # N, M, C, T, V\n x = x.reshape((N * M, C, T, V))\n # forward\n for gcn, importance in zip(self.st_gcn_networks, self.edge_importance):\n x, _ = gcn(x, paddle.multiply(self.A, importance))\n x = self.pool(x) # NM,C,T,V --> NM,C,1,1\n C = x.shape[1]\n x = paddle.reshape(x, (N, M, C, 1, 1)).mean(axis=1) # N,C,1,1\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/stgcn.py:328-343"
+ },
+ "6967": {
+ "file_id": 513,
+ "content": "This code reshapes the input tensor and applies batch normalization before reshaping again. It then transposes the dimensions and reshapes once more. The main operation involves iterating through each ST-GCN network and applying it to the input with multiplied edge importance, followed by pooling. Finally, it reshapes the output, performs averaging over the third dimension, and returns the result.",
+ "type": "comment"
+ },
+ "6968": {
+ "file_id": 514,
+ "content": "/paddlevideo/modeling/backbones/swin_transformer.py",
+ "type": "filepath"
+ },
+ "6969": {
+ "file_id": 514,
+ "content": "The code introduces a DropPath layer, Swin Transformer backbone with window-based multi-head attention for image processing, and implements the Swin Transformer Block 3D in PaddleVideo, which also features a 3D PatchEmbed3D and 3D backbone.",
+ "type": "summary"
+ },
+ "6970": {
+ "file_id": 514,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom functools import lru_cache, reduce\nfrom operator import mul\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import Constant\nfrom ...utils import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import trunc_normal_\nzeros_ = Constant(value=0.)\nones_ = Constant(value=1.)\ndef drop_path(x, drop_prob=0., training=False):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:1-33"
+ },
+ "6971": {
+ "file_id": 514,
+ "content": "Copyright notice, import statements, and drop_path function definition for stochastic depth in residual blocks.",
+ "type": "comment"
+ },
+ "6972": {
+ "file_id": 514,
+ "content": " the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...\n # issuecomment-532968956 ...\n See discussion: https://github.com/tensorflow/tpu/issues/494\n \"\"\"\n if drop_prob == 0. or not training:\n return x\n keep_prob = paddle.to_tensor(1 - drop_prob)\n shape = (paddle.shape(x)[0], ) + (1, ) * (x.ndim - 1)\n random_tensor = keep_prob + paddle.rand(shape, dtype=x.dtype)\n random_tensor = paddle.floor(random_tensor) # binarize\n output = x.divide(keep_prob) * random_tensor\n return output\nclass DropPath(nn.Layer):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n \"\"\"\n def __init__(self, drop_prob=None):\n super(DropPath, self).__init__()\n self.drop_prob = drop_prob\n def forward(self, x):\n return drop_path(x, self.drop_prob, self.training)\nclass Mlp(nn.Layer):\n \"\"\" Multilayer perceptron.\"\"\"\n def __init__(self,\n in_features,\n hidden_features=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:34-64"
+ },
+ "6973": {
+ "file_id": 514,
+ "content": "This code snippet defines a \"DropPath\" layer that applies drop paths (Stochastic Depth) to the input, based on the provided drop probability. The drop paths are applied in the main path of residual blocks for each sample. This class also includes a forward method that drops out elements from the input with the specified probability during training but returns the original input unchanged when not training or if the drop probability is 0.",
+ "type": "comment"
+ },
+ "6974": {
+ "file_id": 514,
+ "content": " out_features=None,\n act_layer=nn.GELU,\n drop=0.):\n super().__init__()\n out_features = out_features or in_features\n hidden_features = hidden_features or in_features\n self.fc1 = nn.Linear(in_features, hidden_features)\n self.act = act_layer()\n self.fc2 = nn.Linear(hidden_features, out_features)\n self.drop = nn.Dropout(drop)\n def forward(self, x):\n x = self.fc1(x)\n x = self.act(x)\n x = self.drop(x)\n x = self.fc2(x)\n x = self.drop(x)\n return x\ndef window_partition(x, window_size):\n \"\"\"window_partition\n Args:\n x (Tensor): x.shape = [B, D, H, W, C]\n window_size (tuple[int]): window_size\n Returns:\n Tensor: (B*num_windows, window_size*window_size, C)\n \"\"\"\n B, D, H, W, C = x.shape\n x = x.reshape([\n B, D // window_size[0], window_size[0], H // window_size[1],\n window_size[1], W // window_size[2], window_size[2], C\n ])\n windows = x.transpose([0, 1, 3, 5, 2, 4, 6,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:65-99"
+ },
+ "6975": {
+ "file_id": 514,
+ "content": "The code above defines a layer for the Swin Transformer backbone. It contains two linear layers, an activation function (GELU), and a dropout layer. The `window_partition` function partitions input tensor based on specified window size.",
+ "type": "comment"
+ },
+ "6976": {
+ "file_id": 514,
+ "content": " 7]).reshape([-1, reduce(mul, window_size), C])\n return windows\nclass Identity(nn.Layer):\n def __init__(self):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\ndef window_reverse(windows, window_size, B, D, H, W):\n \"\"\"\n Args:\n windows: (B*num_windows, window_size, window_size, C)\n window_size (tuple[int]): Window size\n H (int): Height of image\n W (int): Width of image\n Returns:\n x: (B, D, H, W, C)\n \"\"\"\n x = windows.reshape([\n B, D // window_size[0], H // window_size[1], W // window_size[2],\n window_size[0], window_size[1], window_size[2], -1\n ])\n x = x.transpose([0, 1, 4, 2, 5, 3, 6, 7]).reshape([B, D, H, W, -1])\n return x\ndef get_window_size(x_size, window_size, shift_size=None):\n use_window_size = list(window_size)\n if shift_size is not None:\n use_shift_size = list(shift_size)\n for i in range(len(x_size)):\n if x_size[i] <= window_size[i]:\n use_window_size[i] = x_size[i]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:100-137"
+ },
+ "6977": {
+ "file_id": 514,
+ "content": "The code defines a function `window_reverse` that takes a set of windows and rearranges them back into the original image shape. The `get_window_size` function determines the appropriate window size based on input dimensions. Both functions are used in the Swin Transformer backbone model.",
+ "type": "comment"
+ },
+ "6978": {
+ "file_id": 514,
+ "content": " if shift_size is not None:\n use_shift_size[i] = 0\n if shift_size is None:\n return tuple(use_window_size)\n else:\n return tuple(use_window_size), tuple(use_shift_size)\nclass WindowAttention3D(nn.Layer):\n \"\"\" Window based multi-head self attention (W-MSA) module with relative position bias.\n It supports both of shifted and non-shifted window.\n Args:\n dim (int): Number of input channels.\n window_size (tuple[int]): The temporal length, height and width of the window.\n num_heads (int): Number of attention heads.\n qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True\n qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set\n attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0\n proj_drop (float, optional): Dropout ratio of output. Default: 0.0\n \"\"\"\n def __init__(self,\n dim,\n window_size,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:138-161"
+ },
+ "6979": {
+ "file_id": 514,
+ "content": "This code defines a class called \"WindowAttention3D\" which implements a window-based multi-head self attention module with relative position bias. It supports both shifted and non-shifted windows, and takes in parameters such as the number of input channels (dim), temporal length, height and width of the window (window_size), number of attention heads (num_heads), whether to add a learnable bias to query, key, value (qkv_bias), override default qk scale of head_dim ** -0.5 if set (qk_scale), dropout ratio of attention weight (attn_drop), and dropout ratio of output (proj_drop). The function at the top part of the code determines whether to use window or shift size based on a given value.",
+ "type": "comment"
+ },
+ "6980": {
+ "file_id": 514,
+ "content": " num_heads,\n qkv_bias=False,\n qk_scale=None,\n attn_drop=0.,\n proj_drop=0.):\n super().__init__()\n self.dim = dim\n self.window_size = window_size # Wd, Wh, Ww\n self.num_heads = num_heads\n head_dim = dim // num_heads\n self.scale = qk_scale or head_dim**-0.5\n # define a parameter table of relative position bias\n self.relative_position_bias_table = self.create_parameter(\n shape=((2 * window_size[0] - 1) * (2 * window_size[1] - 1) *\n (2 * window_size[2] - 1), num_heads),\n default_initializer=zeros_,\n ) # 2*Wd-1 * 2*Wh-1 * 2*Ww-1, nH\n self.add_parameter(\"relative_position_bias_table\",\n self.relative_position_bias_table)\n # get pair-wise relative position index for each token inside the window\n coords_d = paddle.arange(self.window_size[0])\n coords_h = paddle.arange(self.window_size[1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:162-185"
+ },
+ "6981": {
+ "file_id": 514,
+ "content": "This code initializes the Swin Transformer's self-attention module. It defines a window size and number of attention heads, calculates head dimensions, sets up position bias table, and adds parameters for position bias table and head dimensions. The code also creates coordinate arrays for dimension and height inside the window.",
+ "type": "comment"
+ },
+ "6982": {
+ "file_id": 514,
+ "content": " coords_w = paddle.arange(self.window_size[2])\n coords = paddle.stack(paddle.meshgrid(coords_d, coords_h,\n coords_w)) # 3, Wd, Wh, Ww\n coords_flatten = paddle.flatten(coords, 1) # 3, Wd*Wh*Ww\n relative_coords = coords_flatten.unsqueeze(\n axis=2) - coords_flatten.unsqueeze(axis=1) # 3, Wd*Wh*Ww, Wd*Wh*Ww\n # relative_coords = coords_flatten.unsqueeze(2) - coords_flatten.unsqueeze(1) # 3, Wd*Wh*Ww, Wd*Wh*Ww\n relative_coords = relative_coords.transpose([1, 2, 0\n ]) # Wd*Wh*Ww, Wd*Wh*Ww, 3\n relative_coords[:, :,\n 0] += self.window_size[0] - 1 # shift to start from 0\n relative_coords[:, :, 1] += self.window_size[1] - 1\n relative_coords[:, :, 2] += self.window_size[2] - 1\n relative_coords[:, :, 0] *= (2 * self.window_size[1] -\n 1) * (2 * self.window_size[2] - 1)\n relative_coords[:, :, 1] *= (2 * self.window_size[2] - 1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:186-204"
+ },
+ "6983": {
+ "file_id": 514,
+ "content": "This code performs relative position encoding for the Swin Transformer by calculating relative coordinates of patches within a sliding window. It first creates 2D and 3D coordinate grids, then subtracts them to obtain relative positions. Finally, it shifts and scales the relative coordinates to fit the range of the window size.",
+ "type": "comment"
+ },
+ "6984": {
+ "file_id": 514,
+ "content": " relative_position_index = relative_coords.sum(\n axis=-1) # Wd*Wh*Ww, Wd*Wh*Ww\n self.register_buffer(\"relative_position_index\", relative_position_index)\n self.qkv = nn.Linear(dim, dim * 3, bias_attr=qkv_bias)\n self.attn_drop = nn.Dropout(attn_drop)\n self.proj = nn.Linear(dim, dim)\n self.proj_drop = nn.Dropout(proj_drop)\n trunc_normal_(self.relative_position_bias_table, std=0.02)\n self.softmax = nn.Softmax(axis=-1)\n def forward(self, x, mask=None):\n \"\"\" Forward function.\n Args:\n x: input features with shape of (num_windows*B, N, C)\n mask: (0/-inf) mask with shape of (num_windows, N, N) or None\n \"\"\"\n B_, N, C = x.shape\n qkv = self.qkv(x).reshape(\n [B_, N, 3, self.num_heads,\n C // self.num_heads]).transpose([2, 0, 3, 1, 4])\n q, k, v = qkv[0], qkv[1], qkv[2] # B_, nH, N, C\n q = q * self.scale\n attn = q @ k.transpose([0, 1, 3, 2])\n relative_position_bias = self.relative_position_bias_table[",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:205-232"
+ },
+ "6985": {
+ "file_id": 514,
+ "content": "This code initializes a Swin Transformer backbone by registering a buffer for relative position indices and defining the linear projections, dropouts, softmax function, and forward pass. The forward function takes input features of shape (num_windows*B, N, C) and performs multi-head self-attention with learned query, key, and value matrices, scaled by the square root of the dimension. Attention is calculated using dot product between queries and keys, and then passed through a softmax function for normalization before being multiplied by values and projected back to the original feature space.",
+ "type": "comment"
+ },
+ "6986": {
+ "file_id": 514,
+ "content": " self.relative_position_index[:N, :N].reshape([-1])].reshape(\n [N, N, -1]) # Wd*Wh*Ww,Wd*Wh*Ww,nH\n relative_position_bias = relative_position_bias.transpose(\n [2, 0, 1]) # nH, Wd*Wh*Ww, Wd*Wh*Ww\n attn = attn + relative_position_bias.unsqueeze(0) # B_, nH, N, N\n if mask is not None:\n nW = mask.shape[0]\n attn = attn.reshape([B_ // nW, nW, self.num_heads, N, N\n ]) + mask.unsqueeze(1).unsqueeze(0).astype(attn.dtype)\n attn = attn.reshape([-1, self.num_heads, N, N])\n attn = self.softmax(attn)\n else:\n attn = self.softmax(attn)\n attn = self.attn_drop(attn)\n x = (attn @ v).transpose([0, 2, 1, 3]).reshape([B_, N, C])\n x = self.proj(x)\n x = self.proj_drop(x)\n return x\nclass SwinTransformerBlock3D(nn.Layer):\n \"\"\" Swin Transformer Block.\n Args:\n dim (int): Number of input channels.\n num_heads (int): Number of attention heads.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:233-261"
+ },
+ "6987": {
+ "file_id": 514,
+ "content": "This code defines the Swin Transformer Block 3D, which implements a self-attention mechanism for multi-dimensional data. It adds relative position biases to the attention scores, applies a mask if provided, and applies softmax normalization. Finally, it passes the result through two dropout layers before outputting the transformed feature map.",
+ "type": "comment"
+ },
+ "6988": {
+ "file_id": 514,
+ "content": " window_size (tuple[int]): Window size.\n shift_size (tuple[int]): Shift size for SW-MSA.\n mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.\n qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True\n qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.\n drop (float, optional): Dropout rate. Default: 0.0\n attn_drop (float, optional): Attention dropout rate. Default: 0.0\n drop_path (float, optional): Stochastic depth rate. Default: 0.0\n act_layer (nn.Layer, optional): Activation layer. Default: nn.GELU\n norm_layer (nn.Layer, optional): Normalization layer. Default: nn.LayerNorm\n \"\"\"\n def __init__(self,\n dim,\n num_heads,\n window_size=(2, 7, 7),\n shift_size=(0, 0, 0),\n mlp_ratio=4.,\n qkv_bias=True,\n qk_scale=None,\n drop=0.,\n attn_drop=0.,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:262-282"
+ },
+ "6989": {
+ "file_id": 514,
+ "content": "This code initializes a class for the Swin Transformer backbone, specifying the dimensions, number of heads, window size, shift size, mlp ratio, and various optional parameters like dropout rates and activation layers.",
+ "type": "comment"
+ },
+ "6990": {
+ "file_id": 514,
+ "content": " drop_path=0.,\n act_layer=nn.GELU,\n norm_layer=nn.LayerNorm,\n use_checkpoint=False):\n super().__init__()\n self.dim = dim\n self.num_heads = num_heads\n self.window_size = window_size\n self.shift_size = shift_size\n self.mlp_ratio = mlp_ratio\n # self.use_checkpoint=use_checkpoint\n assert 0 <= self.shift_size[0] < self.window_size[\n 0], \"shift_size must in 0-window_size\"\n assert 0 <= self.shift_size[1] < self.window_size[\n 1], \"shift_size must in 0-window_size\"\n assert 0 <= self.shift_size[2] < self.window_size[\n 2], \"shift_size must in 0-window_size\"\n self.norm1 = norm_layer(dim)\n self.attn = WindowAttention3D(dim,\n window_size=self.window_size,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:283-307"
+ },
+ "6991": {
+ "file_id": 514,
+ "content": "The code defines a class for the Swin Transformer backbone in PaddleVideo. It takes input parameters such as dimension, number of attention heads, window size, and shift size, and initializes layers including norm_layer and attn layer. It performs assertions on shift sizes to ensure they are within the window size limits and then initializes the normalization layer.",
+ "type": "comment"
+ },
+ "6992": {
+ "file_id": 514,
+ "content": " attn_drop=attn_drop,\n proj_drop=drop)\n self.drop_path = DropPath(drop_path) if drop_path > 0. else Identity()\n self.norm2 = norm_layer(dim)\n mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = Mlp(in_features=dim,\n hidden_features=mlp_hidden_dim,\n act_layer=act_layer,\n drop=drop)\n def forward_part1(self, x, mask_matrix):\n B = paddle.shape(x)[0]\n _, D, H, W, C = x.shape\n window_size, shift_size = get_window_size((D, H, W), self.window_size,\n self.shift_size)\n x = self.norm1(x)\n # pad feature maps to multiples of window size\n pad_l = pad_t = pad_d0 = 0\n pad_d1 = (window_size[0] - D % window_size[0]) % window_size[0]\n pad_b = (window_size[1] - H % window_size[1]) % window_size[1]\n pad_r = (window_size[2] - W % window_size[2]) % window_size[2]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:308-330"
+ },
+ "6993": {
+ "file_id": 514,
+ "content": "This code defines a Swin Transformer backbone class with parameters like window size, shift size, and drop path. It initializes the layers including attention and mlp blocks. The forward_part1 function pads input features to multiples of window size for processing.",
+ "type": "comment"
+ },
+ "6994": {
+ "file_id": 514,
+ "content": " x = F.pad(x, (pad_l, pad_r, pad_t, pad_b, pad_d0, pad_d1),\n data_format='NDHWC')\n _, Dp, Hp, Wp, _ = x.shape\n # cyclic shift\n if any(i > 0 for i in shift_size):\n shifted_x = paddle.roll(x,\n shifts=(-shift_size[0], -shift_size[1],\n -shift_size[2]),\n axis=(1, 2, 3))\n attn_mask = mask_matrix\n else:\n shifted_x = x\n attn_mask = None\n # partition windows\n x_windows = window_partition(shifted_x,\n window_size) # B*nW, Wd*Wh*Ww, C\n # W-MSA/SW-MSA\n attn_windows = self.attn(x_windows, mask=attn_mask) # B*nW, Wd*Wh*Ww, C\n # merge windows\n attn_windows = attn_windows.reshape([-1, *(window_size + (C, ))])\n shifted_x = window_reverse(attn_windows, window_size, B, Dp, Hp,\n Wp) # B D' H' W' C\n # reverse cyclic shift",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:331-353"
+ },
+ "6995": {
+ "file_id": 514,
+ "content": "This code performs a cyclic shift on the input feature map, depending on the shift size. If any of the shift sizes are greater than 0, it applies the roll operation to the feature map along specific axes (1, 2, and 3). The shifted feature map is then partitioned into windows based on the window size specified. These windows go through a self-attention layer (self.attn) and are reshaped accordingly. Finally, a reverse cyclic shift is applied to the result before returning the output feature map. This process helps in performing window-based self-attention or spatial-wise self-attention in the Swin Transformer architecture.",
+ "type": "comment"
+ },
+ "6996": {
+ "file_id": 514,
+ "content": " if any(i > 0 for i in shift_size):\n x = paddle.roll(shifted_x,\n shifts=(shift_size[0], shift_size[1],\n shift_size[2]),\n axis=(1, 2, 3))\n else:\n x = shifted_x\n if pad_d1 > 0 or pad_r > 0 or pad_b > 0:\n x = x[:, :D, :H, :W, :]\n return x\n def forward_part2(self, x):\n return self.drop_path(self.mlp(self.norm2(x)))\n def forward(self, x, mask_matrix):\n \"\"\" Forward function.\n Args:\n x: Input feature, tensor size (B, D, H, W, C).\n mask_matrix: Attention mask for cyclic shift.\n \"\"\"\n shortcut = x\n x = self.forward_part1(x, mask_matrix)\n x = shortcut + self.drop_path(x).astype(shortcut.dtype)\n x = x + self.forward_part2(x).astype(x.dtype)\n return x\nclass PatchMerging(nn.Layer):\n \"\"\" Patch Merging Layer\n Args:\n dim (int): Number of input channels.\n norm_layer (nn.Layer, optional): Normalization layer. Default: nn.LayerNorm",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:354-390"
+ },
+ "6997": {
+ "file_id": 514,
+ "content": "The code defines a function for the forward pass of a neural network. It consists of two parts: `forward_part1` and `forward_part2`. The function takes an input tensor, performs some operations, and returns the result. The `forward_part1` function applies a shift operation to the input based on a specified shift size, followed by a padding operation if necessary. The `forward_part2` function passes the input through a multi-layer perceptron (MLP) and applies dropout. Finally, the `forward` function combines the outputs of these two parts and returns the result after adding it to an initial shortcut connection.",
+ "type": "comment"
+ },
+ "6998": {
+ "file_id": 514,
+ "content": " \"\"\"\n def __init__(self, dim, norm_layer=nn.LayerNorm):\n super().__init__()\n self.dim = dim\n self.reduction = nn.Linear(4 * dim, 2 * dim, bias_attr=False)\n self.norm = norm_layer(4 * dim)\n def forward(self, x):\n \"\"\" Forward function.\n Args:\n x: Input feature, tensor size (B, D, H, W, C).\n \"\"\"\n B, D, H, W, C = x.shape\n # padding\n pad_input = (H % 2 == 1) or (W % 2 == 1)\n if pad_input:\n x = F.pad(x, (0, W % 2, 0, H % 2, 0, 0), data_format='NDHWC')\n x0 = x[:, :, 0::2, 0::2, :] # B D H/2 W/2 C\n x1 = x[:, :, 1::2, 0::2, :] # B D H/2 W/2 C\n x2 = x[:, :, 0::2, 1::2, :] # B D H/2 W/2 C\n x3 = x[:, :, 1::2, 1::2, :] # B D H/2 W/2 C\n x = paddle.concat([x0, x1, x2, x3], -1) # B D H/2 W/2 4*C\n x = self.norm(x)\n x = self.reduction(x)\n return x\n# cache each stage results\n@lru_cache()\ndef compute_mask(D, H, W, window_size, shift_size):\n img_mask = paddle.zeros((1, D, H, W, 1)) # 1 Dp Hp Wp 1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:391-426"
+ },
+ "6999": {
+ "file_id": 514,
+ "content": "The code defines a Swin Transformer backbone for an image model. The `__init__` method initializes the Swin Transformer with specified dimension and normalization layer. The forward function processes input feature by splitting, concatenating, normalizing, and reducing dimensions. The `compute_mask` function generates an image mask using LRU caching.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/7.json b/docs/data/7.json
new file mode 100644
index 000000000..f819521cb
--- /dev/null
+++ b/docs/data/7.json
@@ -0,0 +1,549 @@
+{
+ "700": {
+ "file_id": 66,
+ "content": "import os.path as osp\ndef load(file_name, model, **cfg):\n if not osp.isfile(file_name):\n raise IOError(f'{file_name} not exist')\n try:\n state_dicts_ = paddle.load(file_name)['state_dict']\n except:\n state_dicts_ = paddle.load(file_name)\n state_dicts = {}\n for k in model.keys():\n if 'num_batches_tracked' not in k:\n if ('head.' + k) not in state_dicts_.keys():\n if k not in state_dicts_.keys():\n print(f'model -----{k} -------is not in pretrained')\n else:\n state_dicts[k] = state_dicts_[k]\n else:\n state_dicts[k] = state_dicts_['head.' + k]\n write_dict(state_dicts, 'state_dicts.txt', **cfg)\n write_dict(model, 'model.txt', **cfg)\n return state_dicts\n#####\ndef write_dict(state_dict, file_name, **cfg):\n lines = []\n tot = 0\n for k, v in state_dict.items():\n # 目前只发现了torch和paddle模型参数命名的这三种不一致\n # 不一致1\n if 'num_batches_tracked' in k:\n tot += 1",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:239-272"
+ },
+ "701": {
+ "file_id": 66,
+ "content": "This code defines a function `load` that loads a pretrained model from a file. It checks if the file exists, then loads either the 'state_dict' or the entire dictionary depending on compatibility with the model keys. The function also filters out 'num_batches_tracked' and 'head.' before assigning the correct values to state_dicts. Finally, it writes both the modified state_dicts and model to separate text files.",
+ "type": "comment"
+ },
+ "702": {
+ "file_id": 66,
+ "content": " continue\n try:\n line = str(k) + '\\t' + str(v.cpu().detach().numpy().shape) + '\\n'\n except:\n line = str(k) + '\\t' + str(v.shape) + '\\n'\n lines.append(line)\n # with open(cfg.get(\"output_dir\", f\"./output/{file_name}\"), 'w') as f:\n # f.writelines(lines)\n # print('%d num_batches_tracked skipped' % tot)\ndef damage_masks(labels, shift=True, scale=True, rotate=True):\n \"\"\"\n Args:\n labels: numpy array (batch_size * 1 * h * w)\n \"\"\"\n bs, _, h, w = labels.shape\n labels = labels.transpose([0, 2, 3, 1])\n labels = labels.numpy()\n final_label = []\n for i in range(bs):\n label = labels[i]\n damaged_label = damage_masks_np(label, shift, scale, rotate)\n final_label.append(damaged_label)\n final_label = np.array(final_label)\n final_label = paddle.to_tensor(final_label)\n final_label = final_label.transpose([0, 3, 1, 2])\n return final_label\ndef damage_masks_np(labels, shift=True, scale=True, rotate=True):\n \"\"\"Performs the actual mask damaging in numpy.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:273-304"
+ },
+ "703": {
+ "file_id": 66,
+ "content": "The code defines two functions: `damage_masks` and `damage_masks_np`. Both functions take in a labels array, and apply damage to the masks by applying shift, scale, and rotate transformations. The output is returned as a tensor after being transposed. The functions are designed for PaddlePaddle and NumPy respectively, and the input must be of shape (batch_size * 1 * h * w).",
+ "type": "comment"
+ },
+ "704": {
+ "file_id": 66,
+ "content": " Args:\n labels: Int32 numpy array of shape (height, width, 1).\n shift: Boolean, whether to damage the masks by shifting.\n scale: Boolean, whether to damage the masks by scaling.\n rotate: Boolean, whether to damage the masks by rotation.\n dilate: Boolean, whether to damage the masks by dilation.\n Returns:\n The damaged version of labels.\n \"\"\"\n unique_labels = np.unique(labels)\n unique_labels = np.setdiff1d(unique_labels, [0])\n # Shuffle to get random depth ordering when combining together.\n np.random.shuffle(unique_labels)\n damaged_labels = np.zeros_like(labels)\n for l in unique_labels:\n obj_mask = (labels == l)\n damaged_obj_mask = _damage_single_object_mask(obj_mask, shift, scale,\n rotate)\n damaged_labels[damaged_obj_mask] = l\n return damaged_labels\ndef _damage_single_object_mask(mask, shift, scale, rotate):\n \"\"\"Performs mask damaging in numpy for a single object.\n Args:\n mask: Boolean numpy array of shape(height, width, 1).",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:305-330"
+ },
+ "705": {
+ "file_id": 66,
+ "content": "This function damages the input labels by randomly shifting, scaling, rotating, and dilating the object masks. It first extracts unique labels, then shuffles them before iterating through each unique label to generate a damaged version of the labels. The `_damage_single_object_mask` function is used internally for performing mask damage on a single object.",
+ "type": "comment"
+ },
+ "706": {
+ "file_id": 66,
+ "content": " shift: Boolean, whether to damage the masks by shifting.\n scale: Boolean, whether to damage the masks by scaling.\n rotate: Boolean, whether to damage the masks by rotation.\n dilate: Boolean, whether to damage the masks by dilation.\n Returns:\n The damaged version of mask.\n \"\"\"\n if shift:\n mask = _shift_mask(mask)\n if scale:\n mask = _scale_mask(mask)\n if rotate:\n mask = _rotate_mask(mask)\n return mask\ndef _shift_mask(mask, max_shift_factor=0.05):\n \"\"\"Damages a mask for a single object by randomly shifting it in numpy.\n Args:\n mask: Boolean numpy array of shape(height, width, 1).\n max_shift_factor: Float scalar, the maximum factor for random shifting.\n Returns:\n The shifted version of mask.\n \"\"\"\n nzy, nzx, _ = mask.nonzero()\n h = nzy.max() - nzy.min()\n w = nzx.max() - nzx.min()\n size = np.sqrt(h * w)\n offset = np.random.uniform(-size * max_shift_factor,\n size * max_shift_factor, 2)\n shifted_mask = interpolation.shift(np.squeeze(mask, axis=2),",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:331-361"
+ },
+ "707": {
+ "file_id": 66,
+ "content": "This code appears to be part of a function that damages a mask for a single object by randomly shifting it in numpy. The function takes a Boolean numpy array as input and returns the shifted version of the mask. It also includes parameters for scaling, rotation, and dilation, but these operations are not defined in this snippet.",
+ "type": "comment"
+ },
+ "708": {
+ "file_id": 66,
+ "content": " offset,\n order=0).astype('bool')[..., np.newaxis]\n return shifted_mask\ndef _scale_mask(mask, scale_amount=0.025):\n \"\"\"Damages a mask for a single object by randomly scaling it in numpy.\n Args:\n mask: Boolean numpy array of shape(height, width, 1).\n scale_amount: Float scalar, the maximum factor for random scaling.\n Returns:\n The scaled version of mask.\n \"\"\"\n nzy, nzx, _ = mask.nonzero()\n cy = 0.5 * (nzy.max() - nzy.min())\n cx = 0.5 * (nzx.max() - nzx.min())\n scale_factor = np.random.uniform(1.0 - scale_amount, 1.0 + scale_amount)\n shift = transform.SimilarityTransform(translation=[-cx, -cy])\n inv_shift = transform.SimilarityTransform(translation=[cx, cy])\n s = transform.SimilarityTransform(scale=[scale_factor, scale_factor])\n m = (shift + (s + inv_shift)).inverse\n scaled_mask = transform.warp(mask, m) > 0.5\n return scaled_mask\ndef _rotate_mask(mask, max_rot_degrees=3.0):\n \"\"\"Damages a mask for a single object by randomly rotating it in numpy.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:362-388"
+ },
+ "709": {
+ "file_id": 66,
+ "content": "The code contains three functions: _shift_mask, _scale_mask, and _rotate_mask. These functions are used to randomly manipulate a binary mask by shifting, scaling, or rotating it for a single object. The purpose is to damage the mask to enhance the robustness of the AI system against different poses or scales of the object.",
+ "type": "comment"
+ },
+ "710": {
+ "file_id": 66,
+ "content": " Args:\n mask: Boolean numpy array of shape(height, width, 1).\n max_rot_degrees: Float scalar, the maximum number of degrees to rotate.\n Returns:\n The scaled version of mask.\n \"\"\"\n cy = 0.5 * mask.shape[0]\n cx = 0.5 * mask.shape[1]\n rot_degrees = np.random.uniform(-max_rot_degrees, max_rot_degrees)\n shift = transform.SimilarityTransform(translation=[-cx, -cy])\n inv_shift = transform.SimilarityTransform(translation=[cx, cy])\n r = transform.SimilarityTransform(rotation=np.deg2rad(rot_degrees))\n m = (shift + (r + inv_shift)).inverse\n scaled_mask = transform.warp(mask, m) > 0.5\n return scaled_mask\nclass AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n def __init__(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def update(self, val, n=1):\n self.val = val\n self.sum += val * n",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:389-422"
+ },
+ "711": {
+ "file_id": 66,
+ "content": "This code defines a function that rotates and scales a binary mask. It first calculates the center coordinates of the mask, then generates a random rotation angle within a specified range, applies the transformation, and inverses it to get the final scaling transformation matrix. The result is a warped version of the mask where pixels above 0.5 are considered as true values. Additionally, there's an AverageMeter class that computes and stores average and current value for continuous metrics calculation.",
+ "type": "comment"
+ },
+ "712": {
+ "file_id": 66,
+ "content": " self.count += n\n self.avg = self.sum / self.count\nimport numpy as np\ndef label2colormap(label):\n m = label.astype(np.uint8)\n r, c = m.shape\n cmap = np.zeros((r, c, 3), dtype=np.uint8)\n cmap[:, :, 0] = (m & 1) << 7 | (m & 8) << 3 | (m & 64) >> 1\n cmap[:, :, 1] = (m & 2) << 6 | (m & 16) << 2 | (m & 128) >> 2\n cmap[:, :, 2] = (m & 4) << 5 | (m & 32) << 1\n return cmap\ndef torch2paddle(data):\n try:\n import torch\n if isinstance(data, dict):\n np_data = {}\n for k, v in data.items():\n np_data[k] = paddle.to_tensor(v.detach().numpy())\n return np_data\n else:\n return paddle.to_tensor(data.detach().numpy())\n except:\n pass\ndef fill_(tensor: Tensor, value):\n return tensor.set_value(paddle.full_like(tensor, value))\ndef zero_(tensor: Tensor):\n return tensor.set_value(paddle.zeros_like(tensor))\ndef float_(tensor: Tensor):\n return paddle.to_tensor(tensor, dtype='float32')\ndef long_(tensor: Tensor):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:423-466"
+ },
+ "713": {
+ "file_id": 66,
+ "content": "Code utilities for PaddleVideo: converts label to colormap, converts PyTorch data types to Paddle, fills tensor with a value, sets tensor value to zero, and casts tensor to float32 dtype.",
+ "type": "comment"
+ },
+ "714": {
+ "file_id": 66,
+ "content": " return paddle.to_tensor(tensor, dtype='int64')\ndef int_(tensor: Tensor):\n return paddle.to_tensor(tensor, dtype='int32')\ndef byte_(tensor: Tensor):\n return paddle.to_tensor(tensor, dtype='bool')\nclass ToPILImage(BaseTransform):\n def __init__(self, mode=None, keys=None):\n super(ToPILImage, self).__init__(keys)\n def _apply_image(self, pic):\n \"\"\"\n Args:\n pic (Tensor|np.ndarray): Image to be converted to PIL Image.\n Returns:\n PIL: Converted image.\n \"\"\"\n if not (isinstance(pic, paddle.Tensor) or isinstance(pic, np.ndarray)):\n raise TypeError('pic should be Tensor or ndarray. Got {}.'.format(\n type(pic)))\n elif isinstance(pic, paddle.Tensor):\n if pic.ndimension() not in {2, 3}:\n raise ValueError(\n 'pic should be 2/3 dimensional. Got {} dimensions.'.format(\n pic.ndimension()))\n elif pic.ndimension() == 2:\n # if 2D image, add channel dimension (CHW)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:467-500"
+ },
+ "715": {
+ "file_id": 66,
+ "content": "The code provides three functions for converting tensors to different data types: int64, int32, and bool. The class ToPILImage is used to convert images from Tensor or np.ndarray format to PIL Image. It checks the type of input pic and throws a TypeError if it's not a Tensor or ndarray. If pic has 2 or 3 dimensions, it adds a channel dimension for 2D images. If the number of dimensions is not 2 or 3, it raises a ValueError.",
+ "type": "comment"
+ },
+ "716": {
+ "file_id": 66,
+ "content": " pic = pic.unsqueeze(0)\n elif isinstance(pic, np.ndarray):\n if pic.ndim not in {2, 3}:\n raise ValueError(\n 'pic should be 2/3 dimensional. Got {} dimensions.'.format(\n pic.ndim))\n elif pic.ndim == 2:\n # if 2D image, add channel dimension (HWC)\n pic = np.expand_dims(pic, 2)\n npimg = pic\n if isinstance(pic, paddle.Tensor) and \"float\" in str(\n pic.numpy().dtype) and self.mode != 'F':\n pic = pic.mul(255).byte()\n if isinstance(pic, paddle.Tensor):\n npimg = np.transpose(pic.numpy(), (1, 2, 0))\n if not isinstance(npimg, np.ndarray):\n raise TypeError(\n 'Input pic must be a paddle.Tensor or NumPy ndarray, ' +\n 'not {}'.format(type(npimg)))\n if npimg.shape[2] == 1:\n expected_mode = None\n npimg = npimg[:, :, 0]\n if npimg.dtype == np.uint8:\n expected_mode = 'L'",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:501-529"
+ },
+ "717": {
+ "file_id": 66,
+ "content": "This code is checking the input \"pic\" and adjusting its format to be compatible with the function. It first checks if it's a tensor or ndarray, then ensures that the image is 2D or 3D (adding channels if necessary) before converting it into NumPy ndarray format. Finally, it checks the data type and mode to further adjust the \"pic\" as needed. If any issue arises during this process, it raises an error with a descriptive message.",
+ "type": "comment"
+ },
+ "718": {
+ "file_id": 66,
+ "content": " elif npimg.dtype == np.int16:\n expected_mode = 'I;16'\n elif npimg.dtype == np.int32:\n expected_mode = 'I'\n elif npimg.dtype == np.float32:\n expected_mode = 'F'\n if self.mode is not None and self.mode != expected_mode:\n raise ValueError(\n \"Incorrect self.mode ({}) supplied for input type {}. Should be {}\"\n .format(self.mode, np.dtype, expected_mode))\n self.mode = expected_mode\n elif npimg.shape[2] == 2:\n permitted_2_channel_modes = ['LA']\n if self.mode is not None and self.mode not in permitted_2_channel_modes:\n raise ValueError(\n \"Only self.modes {} are supported for 2D inputs\".format(\n permitted_2_channel_modes))\n if self.mode is None and npimg.dtype == np.uint8:\n self.mode = 'LA'\n elif npimg.shape[2] == 4:\n permitted_4_channel_modes = ['RGBA', 'CMYK', 'RGBX']",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:530-553"
+ },
+ "719": {
+ "file_id": 66,
+ "content": "This code is validating the input image's data type and dimensions to determine the appropriate mode for the image. It raises a ValueError if the supplied self.mode does not match the expected mode based on the input type, or if the number of channels in the image does not match permitted modes.",
+ "type": "comment"
+ },
+ "720": {
+ "file_id": 66,
+ "content": " if self.mode is not None and self.mode not in permitted_4_channel_modes:\n raise ValueError(\n \"Only self.modes {} are supported for 4D inputs\".format(\n permitted_4_channel_modes))\n if self.mode is None and npimg.dtype == np.uint8:\n self.mode = 'RGBA'\n else:\n permitted_3_channel_modes = ['RGB', 'YCbCr', 'HSV']\n if self.mode is not None and self.mode not in permitted_3_channel_modes:\n raise ValueError(\n \"Only self.modes {} are supported for 3D inputs\".format(\n permitted_3_channel_modes))\n if self.mode is None and npimg.dtype == np.uint8:\n self.mode = 'RGB'\n if self.mode is None:\n raise TypeError('Input type {} is not supported'.format(\n npimg.dtype))\n return Image.fromarray(npimg, mode=self.mode)\nclass Identity(nn.Layer):\n r\"\"\"A placeholder identity operator that is argument-insensitive.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:554-578"
+ },
+ "721": {
+ "file_id": 66,
+ "content": "The code snippet is part of a class function that checks the input image mode and data type to determine if it's compatible with the operation. If not, it raises an error or sets the mode accordingly. It also defines a placeholder identity operator class.",
+ "type": "comment"
+ },
+ "722": {
+ "file_id": 66,
+ "content": " Args:\n args: any argument (unused)\n kwargs: any keyword argument (unused)\n \"\"\"\n def __init__(self, *args, **kwargs):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\ndef convert(data: dict, to, dtype=None):\n assert isinstance(data, dict)\n input = {}\n for k, v in data.items():\n if 'paddle' == to:\n if isinstance(v, np.ndarray):\n if dtype is not None:\n input[k] = paddle.to_tensor(v.astype(dtype))\n else:\n input[k] = paddle.to_tensor(v)\n else:\n input[k] = v\n elif 'torch' == to:\n try:\n import torch\n if isinstance(v, np.ndarray):\n if dtype is not None:\n input[k] = torch.tensor(v.astype(dtype))\n else:\n input[k] = torch.tensor(v)\n else:\n input[k] = v\n except:\n pass",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:580-615"
+ },
+ "723": {
+ "file_id": 66,
+ "content": "This code defines a class \"Identity\" with an empty forward function and a convert function that converts dictionary data between Paddle and Torch formats. If 'paddle' is given as the to parameter, it converts numpy arrays in the input dictionary to Paddle tensors. If 'torch' is given, it tries to import torch and converts numpy arrays or leaves unchanged non-numpy elements in the input dictionary to Torch tensors. Dtype can be used to specify a specific data type for tensor conversion.",
+ "type": "comment"
+ },
+ "724": {
+ "file_id": 66,
+ "content": " else:\n if isinstance(v, np.ndarray):\n input[k] = v.astype(to)\n else:\n input[k] = v\n return input\ndef clip_grad_norm_(parameters: _tensor_or_tensors,\n max_norm: float,\n norm_type: float = 2.0,\n error_if_nonfinite: bool = False) -> paddle.Tensor:\n r\"\"\"Clips gradient norm of an iterable of parameters.\n The norm is computed over all gradients together, as if they were\n concatenated into a single vector. Gradients are modified in-place.\n Args:\n parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a\n single Tensor that will have gradients normalized\n max_norm (float or int): max norm of the gradients\n norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for\n infinity norm.\n error_if_nonfinite (bool): if True, an error is thrown if the total\n norm of the gradients from :attr:``parameters`` is ``nan``,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:616-640"
+ },
+ "725": {
+ "file_id": 66,
+ "content": "This function clips the gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. It takes arguments such as iterable of Tensors or a single Tensor that will have gradients normalized, max_norm (float or int) to set the maximum norm of the gradients, norm_type (float or int) for the type of used p-norm, and error_if_nonfinite (bool) to indicate whether an error should be thrown if total norm is nan.",
+ "type": "comment"
+ },
+ "726": {
+ "file_id": 66,
+ "content": " ``inf``, or ``-inf``. Default: False (will switch to True in the future)\n Returns:\n Total norm of the parameters (viewed as a single vector).\n \"\"\"\n import time\n if isinstance(parameters, paddle.Tensor):\n parameters = [parameters]\n parameters = [p for p in parameters if p.grad is not None]\n detached_grads = [p.grad.detach() for p in parameters]\n max_norm = float(max_norm)\n norm_type = float(norm_type)\n if len(parameters) == 0:\n return paddle.to_tensor(0.)\n # device = paddle.get_device() # parameters[0].grad.device\n if norm_type == inf:\n norms = [p.abs().max() for p in parameters]\n total_norm = norms[0] if len(norms) == 1 else paddle.max(\n paddle.stack(norms))\n else:\n # tik = time.time()\n total_norm = paddle.norm(\n paddle.stack([paddle.norm(g, norm_type) for g in detached_grads]),\n norm_type)\n # total_norm = paddle.norm(paddle.stack([paddle.sqrt(paddle.sum(g*g)) for g in detached_grads]), norm_type) # fixed.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:641-666"
+ },
+ "727": {
+ "file_id": 66,
+ "content": "This function calculates the total norm of parameters (viewed as a single vector) and handles cases where parameters are tensors. It first checks if the parameter is a tensor, then selects only those with non-null gradients, detaches their gradients, and applies different norm types based on the input. If the norm type is infinity, it calculates maximum absolute values for each parameter; otherwise, it calculates the p-norm using provided parameters and detached gradients.",
+ "type": "comment"
+ },
+ "728": {
+ "file_id": 66,
+ "content": " # print(time.time() - tik)\n if error_if_nonfinite and paddle.logical_or(total_norm.isnan(),\n total_norm.isinf()):\n raise RuntimeError(\n f'The total norm of order {norm_type} for gradients from '\n '`parameters` is non-finite, so it cannot be clipped. To disable '\n 'this error and scale the gradients by the non-finite norm anyway, '\n 'set `error_if_nonfinite=False`')\n clip_coef = max_norm / (total_norm + 1e-6)\n # Note: multiplying by the clamped coef is redundant when the coef is clamped to 1, but doing so\n # avoids a `if clip_coef < 1:` conditional which can require a CPU <=> device synchronization\n # when the gradients do not reside in CPU memory.\n clip_coef_clamped = paddle.clip(clip_coef, max=1.0)\n for i, p in enumerate(parameters):\n # p.set_value(paddle.multiply(p, clip_coef_clamped))\n p.grad.set_value(detached_grads[i] * clip_coef_clamped) # fixed",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:667-682"
+ },
+ "729": {
+ "file_id": 66,
+ "content": "This code checks if the total norm of gradients from `parameters` is non-finite. If it is, it raises a RuntimeError and suggests setting `error_if_nonfinite=False`. Then it calculates the clipping coefficient, performs a clip operation to ensure it's within the range (0, 1], and finally multiplies the gradients with this coefficient to scale them.",
+ "type": "comment"
+ },
+ "730": {
+ "file_id": 66,
+ "content": " # p.grad.detach().mul_(clip_coef_clamped\n return total_norm\n# def max(a: paddle.Tensor, axis=0, keepdim=True):\n# \"\"\"ndarray=numpy.array([[1, 2, 3, 4],\n# [4, 3, 2, 1],\n# [5, 6, 7, 8],\n# [8, 7, 6, 5]])\n# np.where(ndarray == np.max(ndarray))\n# (array([2, 3]), array([3, 0]))\n# ndarray[np.where(ndarray == np.max(ndarray))]\n# array([8, 8])\n# \"\"\"\n# max_ = a.max(axis).unsqueeze(-1)\n# index = paddle.argmax(a, axis=axis, keepdim=keepdim)\n# max_ = max_.numpy()\n# index = index.numpy()\n# # index = paddle.argmax(a, axis=axis, keepdim=keepdim)[-1].flatten()\n# return max_, index\ndef gather(tmp: paddle.Tensor, ind: paddle.Tensor):\n shape = tmp.shape\n tmp = paddle.to_tensor(tmp)\n ind = paddle.to_tensor(ind)\n if len(shape) == 2:\n b = shape[0]\n return concat([\n reshape(paddle.gather(tmp[i, :], ind[i, :]), [1, -1])\n for i in range(b)\n ],\n axis=0)\n elif len(shape) == 3:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:683-716"
+ },
+ "731": {
+ "file_id": 66,
+ "content": "This code defines a function `max()` that finds the maximum values in a tensor and their corresponding indices. It also includes a `gather()` function that performs tensor gathering based on provided indices. The functions use PaddlePaddle library for Tensor operations.",
+ "type": "comment"
+ },
+ "732": {
+ "file_id": 66,
+ "content": " out = []\n for i in range(tmp.shape[0]):\n _ = paddle.index_sample(tmp[i], ind[i])\n out.append(_)\n return paddle.to_tensor(out)\n elif len(shape) == 4:\n b, c, d = shape[:3]\n return concat([\n reshape(\n concat([\n reshape(\n concat([\n reshape(\n paddle.gather(tmp[i, j, k, :], ind[i, j, k, :]),\n [1, -1]) for k in range(d)\n ],\n axis=0), [1, d, -1]) for j in range(c)\n ],\n axis=0), [1, c, d, -1]) for i in range(b)\n ],\n axis=0)\n else:\n pass\n# These no_grad_* functions are necessary as wrappers around the parts of these\n# functions that use `with torch.no_grad()`. The JIT doesn't support context\n# managers, so these need to be implemented as builtins. Using these wrappers\n# lets us keep those builtins small and re-usable.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:717-745"
+ },
+ "733": {
+ "file_id": 66,
+ "content": "This function performs sampling and reshaping operations on tensors with different shapes. It handles cases where the tensor shape has 0, 1, or 4 dimensions. In case of a 4-dimensional tensor, it uses gather and concat functions to rearrange data according to the given indices. The no_grad_* functions are used as wrappers for parts that require `torch.no_grad()` context manager due to JIT's inability to handle context managers directly.",
+ "type": "comment"
+ },
+ "734": {
+ "file_id": 66,
+ "content": "def _no_grad_uniform_(tensor, a, b):\n with paddle.no_grad():\n tensor.set_value(paddle.uniform(tensor.shape, min=a, max=b))\n return tensor\ndef _no_grad_normal_(tensor, mean, std):\n with paddle.no_grad():\n tensor.set_value(paddle.normal(shape=tensor.shape, mean=mean, std=std))\n return tensor\ndef _no_grad_trunc_normal_(tensor, mean, std, a, b):\n from scipy import special\n # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf\n def norm_cdf(x):\n # Computes standard normal cumulative distribution function\n return (1. + math.erf(x / math.sqrt(2.))) / 2.\n if (mean < a - 2 * std) or (mean > b + 2 * std):\n warnings.warn(\n \"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. \"\n \"The distribution of values may be incorrect.\",\n stacklevel=2)\n with paddle.no_grad():\n # Values are generated by using a truncated uniform distribution and\n # then using the inverse CDF for the normal distribution.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:746-774"
+ },
+ "735": {
+ "file_id": 66,
+ "content": "This code defines three functions: `_no_grad_uniform_`, `_no_grad_normal_`, and `_no_grad_trunc_normal_`. These functions generate tensors with specific distributions while using gradient calculation, which helps in tensor initialization or randomization tasks. The first function generates uniformly distributed values within a defined range. The second function generates normally distributed values with specified mean and standard deviation. The third function generates truncated normal distribution values by combining uniform and normal distribution functions.",
+ "type": "comment"
+ },
+ "736": {
+ "file_id": 66,
+ "content": " # Get upper and lower cdf values\n l = norm_cdf((a - mean) / std)\n u = norm_cdf((b - mean) / std)\n # Uniformly fill tensor with values from [l, u], then translate to\n # [2l-1, 2u-1].\n tensor.set_value(\n paddle.uniform(tensor.shape, min=2 * l - 1, max=2 * u - 1))\n # tensor.uniform_(2 * l - 1, 2 * u - 1)\n # Use inverse cdf transform for normal distribution to get truncated\n # standard normal\n # tensor.erfinv_() # paddle 无\n tensor.set_value(special.erfinv(tensor))\n # Transform to proper mean, std\n # tensor.mul_(std * math.sqrt(2.))\n tensor.set_value(tensor.multiply(paddle.to_tensor(std * math.sqrt(2.))))\n tensor.add_(mean)\n # Clamp to ensure it's in the proper range\n tensor.clip_(min=a, max=b)\n return tensor\ndef _no_grad_fill_(tensor, val):\n with paddle.no_grad():\n tensor.set_value(paddle.full_like(tensor, fill_value=val))\n return tensor\ndef _no_grad_zero_(tensor):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:775-806"
+ },
+ "737": {
+ "file_id": 66,
+ "content": "This code snippet is used to generate random values within a specified range and then apply inverse cumulative distribution function (CDF) transforms to normalize the tensor. It uses PyTorch's `paddle` library for its operations. The result is then clamped between a minimum and maximum value, which are defined by the variables 'a' and 'b', respectively. This process ensures that the generated tensor falls within the desired range.",
+ "type": "comment"
+ },
+ "738": {
+ "file_id": 66,
+ "content": " with paddle.no_grad():\n tensor.set_value(paddle.zeros_like(tensor))\n return tensor\ndef calculate_gain(nonlinearity, param=None):\n r\"\"\"Return the recommended gain value for the given nonlinearity function.\n The values are as follows:\n ================= ====================================================\n nonlinearity gain\n ================= ====================================================\n Linear / Identity :math:`1`\n Conv{1,2,3}D :math:`1`\n Sigmoid :math:`1`\n Tanh :math:`\\frac{5}{3}`\n ReLU :math:`\\sqrt{2}`\n Leaky Relu :math:`\\sqrt{\\frac{2}{1 + \\text{negative\\_slope}^2}}`\n SELU :math:`\\frac{3}{4}`\n ================= ====================================================\n Args:\n nonlinearity: the non-linear function (`nn.functional` name)\n param: optional parameter for the non-linear function\n Examples:\n >>> gain = nn.init.calculate_gain('leaky_relu', 0.2) # leaky_relu with negative_slope=0.2",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:807-833"
+ },
+ "739": {
+ "file_id": 66,
+ "content": "This function calculates the recommended gain value for a given nonlinearity function. The gain values depend on the function used, with different values assigned to functions like Linear/Identity (1), Sigmoid (1), Tanh (5/3), ReLU (sqrt(2)), Leaky Relu (sqrt((2/(1 + negative_slope^2))), SELU (3/4). The function takes nonlinearity and optional param as arguments, and returns the gain value.",
+ "type": "comment"
+ },
+ "740": {
+ "file_id": 66,
+ "content": " \"\"\"\n linear_fns = [\n 'linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d',\n 'conv_transpose2d', 'conv_transpose3d'\n ]\n if nonlinearity in linear_fns or nonlinearity == 'sigmoid':\n return 1\n elif nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if param is None:\n negative_slope = 0.01\n elif not isinstance(param, bool) and isinstance(\n param, int) or isinstance(param, float):\n # True/False are instances of int, hence check above\n negative_slope = param\n else:\n raise ValueError(\n \"negative_slope {} not a valid number\".format(param))\n return math.sqrt(2.0 / (1 + negative_slope**2))\n elif nonlinearity == 'selu':\n return 3.0 / 4 # Value found empirically (https://github.com/pytorch/pytorch/pull/50664)\n else:\n raise ValueError(\"Unsupported nonlinearity {}\".format(nonlinearity))",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:834-859"
+ },
+ "741": {
+ "file_id": 66,
+ "content": "This code defines a function to map different nonlinearities (e.g., linear, sigmoid, tanh) to corresponding numerical values or exceptions when an unsupported nonlinearity is provided. It handles cases like linear, sigmoid, tanh, relu, leaky_relu, and selu, providing the appropriate values for each.",
+ "type": "comment"
+ },
+ "742": {
+ "file_id": 66,
+ "content": "def uniform_(tensor: Tensor, a: float = 0., b: float = 1.) -> Tensor:\n r\"\"\"Fills the input Tensor with values drawn from the uniform\n distribution :math:`\\mathcal{U}(a, b)`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n a: the lower bound of the uniform distribution\n b: the upper bound of the uniform distribution\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.uniform_(w)\n \"\"\"\n return _no_grad_uniform_(tensor, a, b)\ndef normal_(tensor: Tensor, mean: float = 0., std: float = 1.) -> Tensor:\n r\"\"\"Fills the input Tensor with values drawn from the normal\n distribution :math:`\\mathcal{N}(\\text{mean}, \\text{std}^2)`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n mean: the mean of the normal distribution\n std: the standard deviation of the normal distribution\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.normal_(w)\n \"\"\"\n return _no_grad_normal_(tensor, mean, std)\ndef trunc_normal_(tensor: Tensor,\n mean: float = 0.,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:862-895"
+ },
+ "743": {
+ "file_id": 66,
+ "content": "These code snippets define functions to initialize a tensor with values drawn from a uniform or normal distribution. The `uniform_` function fills the input tensor with values from a uniform distribution, while `normal_` initializes it with values from a normal distribution. These functions can be used for various tasks such as initializing weights in a neural network.",
+ "type": "comment"
+ },
+ "744": {
+ "file_id": 66,
+ "content": " std: float = 1.,\n a: float = -2.,\n b: float = 2.) -> Tensor:\n r\"\"\"Fills the input Tensor with values drawn from a truncated\n normal distribution. The values are effectively drawn from the\n normal distribution :math:`\\mathcal{N}(\\text{mean}, \\text{std}^2)`\n with values outside :math:`[a, b]` redrawn until they are within\n the bounds. The method used for generating the random values works\n best when :math:`a \\leq \\text{mean} \\leq b`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n mean: the mean of the normal distribution\n std: the standard deviation of the normal distribution\n a: the minimum cutoff value\n b: the maximum cutoff value\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.trunc_normal_(w)\n \"\"\"\n return _no_grad_trunc_normal_(tensor, mean, std, a, b)\ndef constant_(tensor: Tensor, val: float) -> Tensor:\n r\"\"\"Fills the input Tensor with the value :math:`\\text{val}`.\n Args:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:896-923"
+ },
+ "745": {
+ "file_id": 66,
+ "content": "This code snippet defines a function `trunc_normal_` that fills the input Tensor with values drawn from a truncated normal distribution. The values are within the range [a, b] and the method works best when a <= mean <= b. Additionally, it includes a separate function `constant_` which fills the input Tensor with a constant value val.",
+ "type": "comment"
+ },
+ "746": {
+ "file_id": 66,
+ "content": " tensor: an n-dimensional `torch.Tensor`\n val: the value to fill the tensor with\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.constant_(w, 0.3)\n \"\"\"\n return _no_grad_fill_(tensor, val)\ndef ones_(tensor: Tensor) -> Tensor:\n r\"\"\"Fills the input Tensor with the scalar value `1`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.ones_(w)\n \"\"\"\n return _no_grad_fill_(tensor, 1.)\ndef zeros_(tensor: Tensor) -> Tensor:\n r\"\"\"Fills the input Tensor with the scalar value `0`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.zeros_(w)\n \"\"\"\n return _no_grad_zero_(tensor)\ndef eye_(tensor):\n r\"\"\"Fills the 2-dimensional input `Tensor` with the identity\n matrix. Preserves the identity of the inputs in `Linear` layers, where as\n many inputs are preserved as possible.\n Args:\n tensor: a 2-dimensional `torch.Tensor`",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:924-966"
+ },
+ "747": {
+ "file_id": 66,
+ "content": "These methods fill a tensor with specific values or an identity matrix for Linear layers. The `constant_()`, `ones_()`, and `zeros_()` functions fill the input tensor with constant, ones, or zeros respectively. The `eye_()` function fills a 2-dimensional tensor with an identity matrix while preserving the identities of inputs in Linear layers.",
+ "type": "comment"
+ },
+ "748": {
+ "file_id": 66,
+ "content": " Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.eye_(w)\n \"\"\"\n if tensor.ndimension() != 2:\n raise ValueError(\"Only tensors with 2 dimensions are supported\")\n with paddle.no_grad():\n tensor.set_value(paddle.eye(*tensor.shape))\n return tensor\ndef dirac_(tensor, groups=1):\n r\"\"\"Fills the {3, 4, 5}-dimensional input `Tensor` with the Dirac\n delta function. Preserves the identity of the inputs in `Convolutional`\n layers, where as many input channels are preserved as possible. In case\n of groups>1, each group of channels preserves identity\n Args:\n tensor: a {3, 4, 5}-dimensional `torch.Tensor`\n groups (optional): number of groups in the conv layer (default: 1)\n Examples:\n >>> w = torch.empty(3, 16, 5, 5)\n >>> nn.init.dirac_(w)\n >>> w = torch.empty(3, 24, 5, 5)\n >>> nn.init.dirac_(w, 3)\n \"\"\"\n dimensions = tensor.ndimension()\n if dimensions not in [3, 4, 5]:\n raise ValueError(\n \"Only tensors with 3, 4, or 5 dimensions are supported\")",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:968-998"
+ },
+ "749": {
+ "file_id": 66,
+ "content": "The provided code defines two functions, `eye_` and `dirac_`, that initialize a tensor with specific values. The `eye_` function fills the 2D tensor with an identity matrix, while the `dirac_` function fills a 3D, 4D or 5D tensor with Dirac delta functions. It also takes an optional argument for groups in case of Convolutional layers. Both functions require the input tensor to have specific dimensions and raise a ValueError if not satisfied.",
+ "type": "comment"
+ },
+ "750": {
+ "file_id": 66,
+ "content": " sizes = tensor.shape\n if sizes[0] % groups != 0:\n raise ValueError('dim 0 must be divisible by groups')\n out_chans_per_grp = sizes[0] // groups\n min_dim = min(out_chans_per_grp, sizes[1])\n with paddle.no_grad():\n tensor.zero_()\n for g in range(groups):\n for d in range(min_dim):\n if dimensions == 3: # Temporal convolution\n tensor[g * out_chans_per_grp + d, d,\n tensor.shape[2] // 2] = 1\n elif dimensions == 4: # Spatial convolution\n tensor[g * out_chans_per_grp + d, d, tensor.shape[2] // 2,\n tensor.shape[3] // 2] = 1\n else: # Volumetric convolution\n tensor[g * out_chans_per_grp + d, d, tensor.shape[2] // 2,\n tensor.shape[3] // 2, tensor.shape[4] // 2] = 1\n return tensor\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = tensor.dim()\n if dimensions < 2:\n raise ValueError(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1000-1028"
+ },
+ "751": {
+ "file_id": 66,
+ "content": "This function initializes a tensor with ones in specific positions based on the provided dimensions (3 for temporal convolution, 4 for spatial convolution, and 5 for volumetric convolution). It checks if dim 0 is divisible by groups and raises an error if not. Then it calculates out_chans_per_grp and min_dim, and finally initializes the tensor using no_grad context manager.",
+ "type": "comment"
+ },
+ "752": {
+ "file_id": 66,
+ "content": " \"Fan in and fan out can not be computed for tensor with fewer than 2 dimensions\"\n )\n num_input_fmaps = tensor.shape[1] # .size(1)\n num_output_fmaps = tensor.shape[0] # .size(0)\n receptive_field_size = 1\n if tensor.dim() > 2:\n for s in tensor.shape[2:]:\n receptive_field_size *= s # fixed\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef LongTensor(x):\n return paddle.to_tensor(x, dtype='int64')\ndef IntTensor(x):\n return paddle.to_tensor(x, dtype='int32')\ndef xavier_uniform_(tensor: Tensor, gain: float = 1.) -> Tensor:\n r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Understanding the difficulty of training deep feedforward\n neural networks` - Glorot, X. & Bengio, Y. (2010), using a uniform\n distribution. The resulting tensor will have values sampled from\n :math:`\\mathcal{U}(-a, a)` where\n .. math::\n a = \\text{gain} \\times \\sqrt{\\frac{6}{\\text{fan\\_in} + \\text{fan\\_out}}}",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1029-1060"
+ },
+ "753": {
+ "file_id": 66,
+ "content": "Function to calculate fan_in and fan_out for tensor with dimensions greater than 2, compute the gain factor for Xavier uniform initialization, fill the input Tensor with values from a uniform distribution according to Glorot & Bengio (2010) method.",
+ "type": "comment"
+ },
+ "754": {
+ "file_id": 66,
+ "content": " Also known as Glorot initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n gain: an optional scaling factor\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))\n \"\"\"\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n std = gain * math.sqrt(2.0 / float(fan_in + fan_out))\n a = math.sqrt(3.0) * std # Calculate uniform bounds from standard deviation\n return _no_grad_uniform_(tensor, -a, a)\ndef xavier_normal_(tensor: Tensor, gain: float = 1.) -> Tensor:\n r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Understanding the difficulty of training deep feedforward\n neural networks` - Glorot, X. & Bengio, Y. (2010), using a normal\n distribution. The resulting tensor will have values sampled from\n :math:`\\mathcal{N}(0, \\text{std}^2)` where\n .. math::\n \\text{std} = \\text{gain} \\times \\sqrt{\\frac{2}{\\text{fan\\_in} + \\text{fan\\_out}}}\n Also known as Glorot initialization.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1062-1089"
+ },
+ "755": {
+ "file_id": 66,
+ "content": "This code snippet is for initializing a Tensor with values from a normal distribution, using the Xavier/Glorot initialization method. It calculates the standard deviation based on the fan-in and fan-out of the tensor and applies it to uniformly fill the tensor within specified bounds.",
+ "type": "comment"
+ },
+ "756": {
+ "file_id": 66,
+ "content": " Args:\n tensor: an n-dimensional `torch.Tensor`\n gain: an optional scaling factor\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.xavier_normal_(w)\n \"\"\"\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n std = gain * math.sqrt(2.0 / float(fan_in + fan_out))\n return _no_grad_normal_(tensor, 0., std)\ndef _calculate_correct_fan(tensor, mode):\n mode = mode.lower()\n valid_modes = ['fan_in', 'fan_out']\n if mode not in valid_modes:\n raise ValueError(\"Mode {} not supported, please use one of {}\".format(\n mode, valid_modes))\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n return fan_in if mode == 'fan_in' else fan_out\ndef kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu'):\n r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Delving deep into rectifiers: Surpassing human-level\n performance on ImageNet classification` - He, K. et al. (2015), using a\n uniform distribution. The resulting tensor will have values sampled from",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1091-1120"
+ },
+ "757": {
+ "file_id": 66,
+ "content": "This code snippet is from the PyTorch library and provides functions for initializing tensors with Xavier normal distribution. It includes the `kaiming_uniform_` function that takes a tensor, gain factor, and optional parameters for fan-in/fan-out calculation or nonlinearity. The `_calculate_fan_in_and_fan_out` and `_calculate_correct_fan` functions help calculate the appropriate fan values based on input arguments.",
+ "type": "comment"
+ },
+ "758": {
+ "file_id": 66,
+ "content": " :math:`\\mathcal{U}(-\\text{bound}, \\text{bound})` where\n .. math::\n \\text{bound} = \\text{gain} \\times \\sqrt{\\frac{3}{\\text{fan\\_mode}}}\n Also known as He initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n a: the negative slope of the rectifier used after this layer (only\n used with ``'leaky_relu'``)\n mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``\n preserves the magnitude of the variance of the weights in the\n forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the\n backwards pass.\n nonlinearity: the non-linear function (`nn.functional` name),\n recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')\n \"\"\"\n fan = _calculate_correct_fan(tensor, mode)\n gain = calculate_gain(nonlinearity, a)\n std = gain / math.sqrt(fan)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1121-1145"
+ },
+ "759": {
+ "file_id": 66,
+ "content": "This function initializes a tensor with Kaiming uniform initialization, setting the standard deviation of the normal distribution to be equal to gain multiplied by the square root of 3 divided by fan mode. It's used for He initialization and applies nonlinearity based on the specified parameter.",
+ "type": "comment"
+ },
+ "760": {
+ "file_id": 66,
+ "content": " bound = math.sqrt(\n 3.0) * std # Calculate uniform bounds from standard deviation\n with paddle.no_grad():\n tensor.set_value(paddle.uniform(tensor.shape, min=-bound, max=bound))\n return tensor\ndef kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu'):\n r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Delving deep into rectifiers: Surpassing human-level\n performance on ImageNet classification` - He, K. et al. (2015), using a\n normal distribution. The resulting tensor will have values sampled from\n :math:`\\mathcal{N}(0, \\text{std}^2)` where\n .. math::\n \\text{std} = \\frac{\\text{gain}}{\\sqrt{\\text{fan\\_mode}}}\n Also known as He initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n a: the negative slope of the rectifier used after this layer (only\n used with ``'leaky_relu'``)\n mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``\n preserves the magnitude of the variance of the weights in the",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1146-1170"
+ },
+ "761": {
+ "file_id": 66,
+ "content": "This code initializes a tensor using the Kaiming normal method. It fills the input tensor with values sampled from a normal distribution, where std is calculated based on gain and fan_mode (fan_in by default). This initialization method is often used in neural networks to improve performance.",
+ "type": "comment"
+ },
+ "762": {
+ "file_id": 66,
+ "content": " forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the\n backwards pass.\n nonlinearity: the non-linear function (`nn.functional` name),\n recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).\n Examples:\n >>> w = torch.empty(3, 5)\n >>> kaiming_normal_(w, mode='fan_out', nonlinearity='relu')\n \"\"\"\n fan = _calculate_correct_fan(tensor, mode)\n gain = calculate_gain(nonlinearity, a)\n std = gain / math.sqrt(fan)\n with paddle.no_grad():\n tensor.set_value(paddle.normal(shape=tensor.shape, mean=0, std=std))\n return tensor\ndef orthogonal_(tensor, gain=1):\n r\"\"\"Fills the input `Tensor` with a (semi) orthogonal matrix, as\n described in `Exact solutions to the nonlinear dynamics of learning in deep\n linear neural networks` - Saxe, A. et al. (2013). The input tensor must have\n at least 2 dimensions, and for tensors with more than 2 dimensions the\n trailing dimensions are flattened.\n Args:\n tensor: an n-dimensional `torch.Tensor`, where :math:`n \\geq 2`",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1171-1196"
+ },
+ "763": {
+ "file_id": 66,
+ "content": "These functions fill a tensor with either a (semi) orthogonal matrix or initialize weights using Kaiming normal distribution. The 'fan_out' and 'nonlinearity' parameters are used for the initialization process. These functions are inspired by research papers, one focusing on orthogonal matrices in deep linear neural networks and another on Kaiming normal distribution for weight initialization.",
+ "type": "comment"
+ },
+ "764": {
+ "file_id": 66,
+ "content": " gain: optional scaling factor\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.orthogonal_(w)\n \"\"\"\n if tensor.ndimension() < 2:\n raise ValueError(\"Only tensors with 2 or more dimensions are supported\")\n rows = tensor.shape[0] # .size(0)\n cols = tensor.numel() // rows\n flattened = tensor.new(rows, cols).normal_(0, 1)\n if rows < cols:\n flattened.t_()\n # Compute the qr factorization\n q, r = paddle.to_tensor(np.linalg.qr(flattened.numpy()))\n # q, r = torch.qr(flattened)\n # Make Q uniform according to https://arxiv.org/pdf/math-ph/0609050.pdf\n d = paddle.diag(r, 0)\n ph = d.sign()\n q *= ph\n if rows < cols:\n q.t_()\n with paddle.no_grad():\n tensor.view_as(q).copy_(q)\n tensor.mul_(gain)\n return tensor\ndef sparse_(tensor, sparsity, std=0.01):\n r\"\"\"Fills the 2D input `Tensor` as a sparse matrix, where the\n non-zero elements will be drawn from the normal distribution\n :math:`\\mathcal{N}(0, 0.01)`, as described in `Deep learning via",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1197-1233"
+ },
+ "765": {
+ "file_id": 66,
+ "content": "This function initializes a 2D tensor with values drawn from the standard normal distribution, ensuring that at least a certain sparsity level is maintained. It uses QR factorization and scales the result by a given gain factor if specified.",
+ "type": "comment"
+ },
+ "766": {
+ "file_id": 66,
+ "content": " Hessian-free optimization` - Martens, J. (2010).\n Args:\n tensor: an n-dimensional `torch.Tensor`\n sparsity: The fraction of elements in each column to be set to zero\n std: the standard deviation of the normal distribution used to generate\n the non-zero values\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.sparse_(w, sparsity=0.1)\n \"\"\"\n if tensor.ndimension() != 2:\n raise ValueError(\"Only tensors with 2 dimensions are supported\")\n rows, cols = tensor.shape\n num_zeros = int(math.ceil(sparsity * rows))\n with paddle.no_grad():\n tensor.normal_(0, std)\n for col_idx in range(cols):\n row_indices = paddle.randperm(rows)\n zero_indices = row_indices[:num_zeros]\n tensor[zero_indices, col_idx] = 0\n return tensor\n# for backward compatibility\ndef _make_deprecate(meth):\n new_name = meth.__name__\n old_name = new_name[:-1]\n def deprecated_init(*args, **kwargs):\n warnings.warn(\n \"nn.init.{} is now deprecated in favor of nn.init.{}.\".format(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1234-1268"
+ },
+ "767": {
+ "file_id": 66,
+ "content": "This code initializes a 2D torch.Tensor with a specified sparsity and standard deviation by setting some elements to zero while keeping others non-zero. It checks for tensor dimensions, normalizes values, assigns zeroes based on the input sparsity, and is compatible with both PyTorch and PaddlePaddle.",
+ "type": "comment"
+ },
+ "768": {
+ "file_id": 66,
+ "content": " old_name, new_name),\n stacklevel=2)\n return meth(*args, **kwargs)\n deprecated_init.__doc__ = r\"\"\"\n {old_name}(...)\n .. warning::\n This method is now deprecated in favor of :func:`torch.nn.init.{new_name}`.\n See :func:`~torch.nn.init.{new_name}` for details.\"\"\".format(\n old_name=old_name, new_name=new_name)\n deprecated_init.__name__ = old_name\n return deprecated_init\n# uniform = _make_deprecate(uniform_)\n# normal = _make_deprecate(normal_)\n# constant = _make_deprecate(constant_)\n# eye = _make_deprecate(eye_)\n# dirac = _make_deprecate(dirac_)\n# xavier_uniform = _make_deprecate(xavier_uniform_)\n# xavier_normal = _make_deprecate(xavier_normal_)\n# kaiming_uniform = _make_deprecate(kaiming_uniform_)\n# kaiming_normal = _make_deprecate(kaiming_normal_)\n# orthogonal = _make_deprecate(orthogonal_)\n# sparse = _make_deprecate(sparse_)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/manet_utils.py:1269-1295"
+ },
+ "769": {
+ "file_id": 66,
+ "content": "This code defines several deprecated initialization methods and creates their corresponding non-deprecated alternatives. The _make_deprecate function wraps the old functions with a warning that they are deprecated in favor of new Torch.nn.init functions, redirecting users to the new functions for more information.",
+ "type": "comment"
+ },
+ "770": {
+ "file_id": 67,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/precise_bn.py",
+ "type": "filepath"
+ },
+ "771": {
+ "file_id": 67,
+ "content": "The precise_bn.py file in PaddlePaddle's EIVideo module contains a function called do_preciseBN, which recomputes batch normalization stats for improved accuracy by running the model multiple times with input data from the data_loader, updating BN layers with running averages for normalization.",
+ "type": "summary"
+ },
+ "772": {
+ "file_id": 67,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport itertools\nfrom EIVideo.paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n\"\"\"\nImplement precise bn, which is useful for improving accuracy.\n\"\"\"\n@paddle.no_grad() # speed up and save CUDA memory\ndef do_preciseBN(model, data_loader, parallel, num_iters=200):\n \"\"\"\n Recompute and update the batch norm stats to make them more precise. During\n training both BN stats and the weight are changing after every iteration, so",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/precise_bn.py:1-30"
+ },
+ "773": {
+ "file_id": 67,
+ "content": "The code provided is part of the PaddlePaddle framework for video applications, specifically the EIVideo module. This precise_bn.py file contains a function called do_preciseBN that recomputes and updates batch norm stats to improve accuracy. It does so by running the model with input data from the data_loader multiple times (num_iters) to make BN statistics more precise. The code also includes an import for paddle, itertools, and EIVideo's paddlevideo module.",
+ "type": "comment"
+ },
+ "774": {
+ "file_id": 67,
+ "content": " the running average can not precisely reflect the actual stats of the\n current model.\n In this function, the BN stats are recomputed with fixed weights, to make\n the running average more precise. Specifically, it computes the true average\n of per-batch mean/variance instead of the running average.\n This is useful to improve validation accuracy.\n Args:\n model: the model whose bn stats will be recomputed\n data_loader: an iterator. Produce data as input to the model\n num_iters: number of iterations to compute the stats.\n Return:\n the model with precise mean and variance in bn layers.\n \"\"\"\n bn_layers_list = [\n m for m in model.sublayers()\n if any((isinstance(m, bn_type)\n for bn_type in (paddle.nn.BatchNorm1D, paddle.nn.BatchNorm2D,\n paddle.nn.BatchNorm3D))) and m.training\n ]\n if len(bn_layers_list) == 0:\n return\n # moving_mean=moving_mean*momentum+batch_mean*(1.−momentum)\n # we set momentum=0. to get the true mean and variance during forward",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/precise_bn.py:31-54"
+ },
+ "775": {
+ "file_id": 67,
+ "content": "This code recomputes the batch normalization (BN) statistics with fixed weights for a given model, improving validation accuracy. It computes true average of per-batch mean/variance instead of running average. The code targets specific BN layers in the model and is applied when there are no such layers or if training is not enabled.",
+ "type": "comment"
+ },
+ "776": {
+ "file_id": 67,
+ "content": " momentum_actual = [bn._momentum for bn in bn_layers_list]\n for bn in bn_layers_list:\n bn._momentum = 0.\n running_mean = [paddle.zeros_like(bn._mean)\n for bn in bn_layers_list] #pre-ignore\n running_var = [paddle.zeros_like(bn._variance) for bn in bn_layers_list]\n ind = -1\n for ind, data in enumerate(itertools.islice(data_loader, num_iters)):\n logger.info(\"doing precise BN {} / {}...\".format(ind + 1, num_iters))\n if parallel:\n model._layers.train_step(data)\n else:\n model.train_step(data)\n for i, bn in enumerate(bn_layers_list):\n # Accumulates the bn stats.\n running_mean[i] += (bn._mean - running_mean[i]) / (ind + 1)\n running_var[i] += (bn._variance - running_var[i]) / (ind + 1)\n assert ind == num_iters - 1, (\n \"update_bn_stats is meant to run for {} iterations, but the batch_sampler stops at {} iterations.\"\n .format(num_iters, ind))\n # Sets the precise bn stats.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/precise_bn.py:55-80"
+ },
+ "777": {
+ "file_id": 67,
+ "content": "This code initializes the momentum of batch normalization (BN) layers to 0 and creates lists for running mean and variance. It then trains a model for a specified number of iterations, updating the BN statistics by accumulating the difference between current and running mean/variance. Finally, it asserts that the correct number of iterations were performed.",
+ "type": "comment"
+ },
+ "778": {
+ "file_id": 67,
+ "content": " for i, bn in enumerate(bn_layers_list):\n bn._mean.set_value(running_mean[i])\n bn._variance.set_value(running_var[i])\n bn._momentum = momentum_actual[i]",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/precise_bn.py:81-84"
+ },
+ "779": {
+ "file_id": 67,
+ "content": "This code is iterating through a list of batch normalization (BN) layers, setting their mean and variance values from a separate list, and updating their momentum value. This could be part of a model's training process where it updates the BN layers with running averages for normalization.",
+ "type": "comment"
+ },
+ "780": {
+ "file_id": 68,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/profiler.py",
+ "type": "filepath"
+ },
+ "781": {
+ "file_id": 68,
+ "content": "This code initializes global profiling variables and defines the ProfilerOptions class for operator-level timing using PaddlePaddle's profiler. It also stops the profiler, checks for exit conditions, and increments _profiler_step_id.",
+ "type": "summary"
+ },
+ "782": {
+ "file_id": 68,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport sys\nimport paddle\n# A global variable to record the number of calling times for profiler\n# functions. It is used to specify the tracing range of training steps.\n_profiler_step_id = 0\n# A global variable to avoid parsing from string every time.\n_profiler_options = None\nclass ProfilerOptions(object):\n \"\"\"\n Use a string to initialize a ProfilerOptions.\n The string should be in the format: \"key1=value1;key2=value;key3=value3\".",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/profiler.py:1-29"
+ },
+ "783": {
+ "file_id": 68,
+ "content": "This code is setting up a global variable to record the number of calling times for profiler functions and another global variable to avoid parsing from string every time. It also defines the ProfilerOptions class, which can be initialized using a string in the format \"key1=value1;key2=value;key3=value3\". This indicates that the code is part of PaddleVideo's EIVideo application and is related to profiling options and step ID management.",
+ "type": "comment"
+ },
+ "784": {
+ "file_id": 68,
+ "content": " For example:\n \"profile_path=model.profile\"\n \"batch_range=[50, 60]; profile_path=model.profile\"\n \"batch_range=[50, 60]; tracer_option=OpDetail; profile_path=model.profile\"\n ProfilerOptions supports following key-value pair:\n batch_range - a integer list, e.g. [100, 110].\n state - a string, the optional values are 'CPU', 'GPU' or 'All'.\n sorted_key - a string, the optional values are 'calls', 'total',\n 'max', 'min' or 'ave.\n tracer_option - a string, the optional values are 'Default', 'OpDetail',\n 'AllOpDetail'.\n profile_path - a string, the path to save the serialized profile data,\n which can be used to generate a timeline.\n exit_on_finished - a boolean.\n \"\"\"\n def __init__(self, options_str):\n assert isinstance(options_str, str)\n self._options = {\n 'batch_range': [10, 20],\n 'state': 'All',\n 'sorted_key': 'total',",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/profiler.py:30-52"
+ },
+ "785": {
+ "file_id": 68,
+ "content": "The code defines a class \"ProfilerOptions\" which takes in an options string and initializes its attributes. Options can include batch_range, state (CPU/GPU/All), sorted_key (calls/total/max/min/ave), tracer_option (Default/OpDetail/AllOpDetail), profile_path for storing serialized data, and exit_on_finished boolean flag.",
+ "type": "comment"
+ },
+ "786": {
+ "file_id": 68,
+ "content": " 'tracer_option': 'Default',\n 'profile_path': '/tmp/profile',\n 'exit_on_finished': True\n }\n self._parse_from_string(options_str)\n def _parse_from_string(self, options_str):\n for kv in options_str.replace(' ', '').split(';'):\n key, value = kv.split('=')\n if key == 'batch_range':\n value_list = value.replace('[', '').replace(']', '').split(',')\n value_list = list(map(int, value_list))\n if len(value_list) >= 2 and value_list[0] >= 0 and value_list[\n 1] > value_list[0]:\n self._options[key] = value_list\n elif key == 'exit_on_finished':\n self._options[key] = value.lower() in (\"yes\", \"true\", \"t\", \"1\")\n elif key in [\n 'state', 'sorted_key', 'tracer_option', 'profile_path'\n ]:\n self._options[key] = value\n def __getitem__(self, name):\n if self._options.get(name, None) is None:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/profiler.py:53-76"
+ },
+ "787": {
+ "file_id": 68,
+ "content": "Class for parsing profile options from a string. It stores the batch range, tracer option, exit on finished status, state, sorted key, and profile path as options. The _parse_from_string function sets the values based on specific conditions: if the 'batch_range' is valid, 'exit_on_finished' is set to True if the value matches \"yes\", \"true\", \"t\", or \"1\", and other options are directly assigned from the string. If an option name is not found in the string, it returns None.",
+ "type": "comment"
+ },
+ "788": {
+ "file_id": 68,
+ "content": " raise ValueError(\n \"ProfilerOptions does not have an option named %s.\" % name)\n return self._options[name]\ndef add_profiler_step(options_str=None):\n \"\"\"\n Enable the operator-level timing using PaddlePaddle's profiler.\n The profiler uses a independent variable to count the profiler steps.\n One call of this function is treated as a profiler step.\n Args:\n profiler_options - a string to initialize the ProfilerOptions.\n Default is None, and the profiler is disabled.\n \"\"\"\n if options_str is None:\n return\n global _profiler_step_id\n global _profiler_options\n if _profiler_options is None:\n _profiler_options = ProfilerOptions(options_str)\n if _profiler_step_id == _profiler_options['batch_range'][0]:\n paddle.utils.profiler.start_profiler(_profiler_options['state'],\n _profiler_options['tracer_option'])\n elif _profiler_step_id == _profiler_options['batch_range'][1]:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/profiler.py:77-104"
+ },
+ "789": {
+ "file_id": 68,
+ "content": "This function enables the operator-level timing using PaddlePaddle's profiler. It initializes the ProfilerOptions with a provided string and increments the global profiler step id. If the current step matches the start or end of the batch range in the options, it starts or stops the profiler respectively.",
+ "type": "comment"
+ },
+ "790": {
+ "file_id": 68,
+ "content": " paddle.utils.profiler.stop_profiler(_profiler_options['sorted_key'],\n _profiler_options['profile_path'])\n if _profiler_options['exit_on_finished']:\n sys.exit(0)\n _profiler_step_id += 1",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/profiler.py:105-110"
+ },
+ "791": {
+ "file_id": 68,
+ "content": "This code snippet stops the profiler, checks if it should exit on finished, and increments _profiler_step_id.",
+ "type": "comment"
+ },
+ "792": {
+ "file_id": 69,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py",
+ "type": "filepath"
+ },
+ "793": {
+ "file_id": 69,
+ "content": "This code initializes a logger, defines functions for logging metrics such as loss and learning rate during training. It also creates a class for tracking various metrics with update method and logs batch metrics at specified batch IDs in log_batch function, while formatting log string with colors for clarity in video processing tasks.",
+ "type": "summary"
+ },
+ "794": {
+ "file_id": 69,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections import OrderedDict\nimport paddle\nfrom .logger import coloring, get_logger\nlogger = get_logger(\"paddlevideo\")\n__all__ = ['AverageMeter', 'build_record', 'log_batch', 'log_epoch']\ndef build_record(cfg):\n framework_type = cfg.get('framework', '')\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"lr\", AverageMeter('lr', 'f', need_avg=False)),\n ]\n if 'Recognizer1D' in framework_type: #TODO: required specify str in framework",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py:1-32"
+ },
+ "795": {
+ "file_id": 69,
+ "content": "This code imports necessary libraries and defines functions for logging metrics such as loss, learning rate during training. It also initializes a logger with the name \"paddlevideo\" and specifies the available classes or functions that can be accessed from this file. The build_record function takes a configuration file and creates an ordered dictionary of metrics to record based on the framework type specified in the configuration file.",
+ "type": "comment"
+ },
+ "796": {
+ "file_id": 69,
+ "content": " record_list.append((\"hit_at_one\", AverageMeter(\"hit_at_one\", '.5f')))\n record_list.append((\"perr\", AverageMeter(\"perr\", '.5f')))\n record_list.append((\"gap\", AverageMeter(\"gap\", '.5f')))\n elif 'Recognizer' in framework_type:\n record_list.append((\"top1\", AverageMeter(\"top1\", '.5f')))\n record_list.append((\"top5\", AverageMeter(\"top5\", '.5f')))\n elif 'FastRCNN' in framework_type:\n record_list.append(\n (\"recall@thr=0.5\", AverageMeter(\"recall@thr=0.5\", '.5f')))\n record_list.append(\n (\"prec@thr=0.5\", AverageMeter(\"prec@thr=0.5\", '.5f')))\n record_list.append((\"recall@top3\", AverageMeter(\"recall@top3\", '.5f')))\n record_list.append((\"prec@top3\", AverageMeter(\"prec@top3\", '.5f')))\n record_list.append((\"recall@top5\", AverageMeter(\"recall@top5\", '.5f')))\n record_list.append((\"prec@top5\", AverageMeter(\"prec@top5\", '.5f')))\n record_list.append((\"mAP@0.5IOU\", AverageMeter(\"mAP@0.5IOU\", '.5f')))\n elif 'DepthEstimator' in cfg.framework:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py:33-49"
+ },
+ "797": {
+ "file_id": 69,
+ "content": "This code is part of a function that handles recording different metrics for various framework types. It appends specific metric names and instances of the AverageMeter class to the record_list depending on the framework type. If 'Recognizer' is in the framework type, it records 'top1' and 'top5' metrics. If 'FastRCNN' is present, it records a series of recall and precision metrics along with mAP@0.5IOU. The function continues with more conditions for different framework types after this snippet.",
+ "type": "comment"
+ },
+ "798": {
+ "file_id": 69,
+ "content": " record_list.append((\"abs_rel\", AverageMeter(\"abs_rel\", '.5f')))\n record_list.append((\"sq_rel\", AverageMeter(\"sq_rel\", '.5f')))\n record_list.append((\"rmse\", AverageMeter(\"rmse\", '.5f')))\n record_list.append((\"rmse_log\", AverageMeter(\"rmse_log\", '.5f')))\n record_list.append((\"a1\", AverageMeter(\"a1\", '.5f')))\n record_list.append((\"a2\", AverageMeter(\"a2\", '.5f')))\n record_list.append((\"a3\", AverageMeter(\"a3\", '.5f')))\n record_list.append((\"losses_day\", AverageMeter(\"losses_day\", '.5f')))\n record_list.append(\n (\"losses_night\", AverageMeter(\"losses_night\", '.5f')))\n record_list.append((\"batch_time\", AverageMeter('batch_cost', '.5f')))\n record_list.append((\"reader_time\", AverageMeter('reader_cost', '.5f')))\n record_list = OrderedDict(record_list)\n return record_list\nclass AverageMeter(object):\n \"\"\"\n Computes and stores the average and current value\n \"\"\"\n def __init__(self, name='', fmt='f', need_avg=True):\n self.name = name",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py:50-72"
+ },
+ "799": {
+ "file_id": 69,
+ "content": "This code defines a list of metrics to be tracked and an AverageMeter class that computes and stores the average and current value for each metric. The list includes various metrics like \"abs_rel\", \"sq_rel\", \"rmse\", \"rmse_log\", \"a1\", \"a2\", \"a3\", \"losses_day\", \"losses_night\", \"batch_time\", and \"reader_time\". The list is then converted to an OrderedDict.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/70.json b/docs/data/70.json
new file mode 100644
index 000000000..eefb779eb
--- /dev/null
+++ b/docs/data/70.json
@@ -0,0 +1,550 @@
+{
+ "7000": {
+ "file_id": 514,
+ "content": " cnt = 0\n for d in slice(-window_size[0]), slice(-window_size[0],\n -shift_size[0]), slice(\n -shift_size[0], None):\n for h in slice(-window_size[1]), slice(-window_size[1],\n -shift_size[1]), slice(\n -shift_size[1], None):\n for w in slice(-window_size[2]), slice(-window_size[2],\n -shift_size[2]), slice(\n -shift_size[2], None):\n img_mask[:, d, h, w, :] = cnt\n cnt += 1\n mask_windows = window_partition(img_mask,\n window_size) # nW, ws[0]*ws[1]*ws[2], 1\n mask_windows = mask_windows.squeeze(-1) # nW, ws[0]*ws[1]*ws[2]\n attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)\n # attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:427-443"
+ },
+ "7001": {
+ "file_id": 514,
+ "content": "This code generates an attention mask for a Swin Transformer model. It iterates through various dimensions (d, h, w) within the window size and shift size, assigning incremental values to each position in the img_mask tensor. The resulting img_mask is then partitioned into non-overlapping windows and squeezed along the last dimension to create mask_windows. Finally, attn_mask is created by subtracting the expanded version of mask_windows from itself, effectively creating a binary mask where values are either 0 or -100.",
+ "type": "comment"
+ },
+ "7002": {
+ "file_id": 514,
+ "content": " huns = -100.0 * paddle.ones_like(attn_mask)\n attn_mask = huns * (attn_mask != 0).astype(\"float32\")\n return attn_mask\nclass BasicLayer(nn.Layer):\n \"\"\" A basic Swin Transformer layer for one stage.\n Args:\n dim (int): Number of feature channels\n depth (int): Depths of this stage.\n num_heads (int): Number of attention head.\n window_size (tuple[int]): Local window size. Default: (1,7,7).\n mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.\n qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True\n qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.\n drop (float, optional): Dropout rate. Default: 0.0\n attn_drop (float, optional): Attention dropout rate. Default: 0.0\n drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0\n norm_layer (nn.Layer, optional): Normalization layer. Default: nn.LayerNorm\n ",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:444-464"
+ },
+ "7003": {
+ "file_id": 514,
+ "content": "The code defines a Swin Transformer layer for one stage in a neural network. The BasicLayer class takes various arguments such as feature channel dimensions, depth, number of heads, local window size, etc. It also includes an MLP (Multi-Layer Perceptron) with a specified ratio, and provides options to add learnable bias, scale factors, dropout rates, stochastic depth rate, and a normalization layer for each input. This basic layer can be utilized in the Swin Transformer architecture for feature extraction and classification tasks.",
+ "type": "comment"
+ },
+ "7004": {
+ "file_id": 514,
+ "content": " downsample (nn.Layer | None, optional): Downsample layer at the end of the layer. Default: None\n \"\"\"\n def __init__(self,\n dim,\n depth,\n num_heads,\n window_size=(1, 7, 7),\n mlp_ratio=4.,\n qkv_bias=False,\n qk_scale=None,\n drop=0.,\n attn_drop=0.,\n drop_path=0.,\n norm_layer=nn.LayerNorm,\n downsample=None,\n use_checkpoint=False):\n super().__init__()\n self.window_size = window_size\n self.shift_size = tuple(i // 2 for i in window_size)\n self.depth = depth\n self.use_checkpoint = use_checkpoint\n # build blocks\n self.blocks = nn.LayerList([\n SwinTransformerBlock3D(\n dim=dim,\n num_heads=num_heads,\n window_size=window_size,\n shift_size=(0, 0, 0) if (i % 2 == 0) else self.shift_size,\n mlp_ratio=mlp_ratio,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:464-493"
+ },
+ "7005": {
+ "file_id": 514,
+ "content": "This code defines a 3D Swin Transformer block with optional downsampling layer at the end. It takes parameters such as dim, depth, num_heads, window size, mlp ratio, etc., and initializes an instance of the class SwinTransformerBlock3D for each block in a LayerList. The window size is set to (1, 7, 7) by default and the shift size is determined based on whether the current index is even or odd.",
+ "type": "comment"
+ },
+ "7006": {
+ "file_id": 514,
+ "content": " qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n drop=drop,\n attn_drop=attn_drop,\n drop_path=drop_path[i]\n if isinstance(drop_path, list) else drop_path,\n norm_layer=norm_layer,\n use_checkpoint=use_checkpoint,\n ) for i in range(depth)\n ])\n self.downsample = downsample\n if self.downsample is not None:\n self.downsample = downsample(dim=dim, norm_layer=norm_layer)\n def forward(self, x):\n \"\"\" Forward function.\n Args:\n x: Input feature, tensor size (B, C, D, H, W).\n \"\"\"\n # calculate attention mask for SW-MSA\n B = paddle.shape(x)[0]\n _, C, D, H, W = x.shape\n window_size, shift_size = get_window_size((D, H, W), self.window_size,\n self.shift_size)\n # x = rearrange(x, 'b c d h w -> b d h w c')\n x = x.transpose([0, 2, 3, 4, 1])\n Dp = int(np.ceil(D / window_size[0])) * window_size[0]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:494-522"
+ },
+ "7007": {
+ "file_id": 514,
+ "content": "This code defines a Swin Transformer block for the PaddleVideo library. It takes input dimensions and creates multiple linear layers for self-attention, followed by a downsampling operation if needed. The forward function calculates an attention mask based on window size and shifts before rearranging the input tensor.",
+ "type": "comment"
+ },
+ "7008": {
+ "file_id": 514,
+ "content": " Hp = int(np.ceil(H / window_size[1])) * window_size[1]\n Wp = int(np.ceil(W / window_size[2])) * window_size[2]\n attn_mask = compute_mask(Dp, Hp, Wp, window_size, shift_size)\n for blk in self.blocks:\n x = blk(x, attn_mask)\n x = x.reshape([B, D, H, W, C])\n if self.downsample is not None:\n x = self.downsample(x)\n x = x.transpose([0, 4, 1, 2, 3])\n return x\nclass PatchEmbed3D(nn.Layer):\n \"\"\" Video to Patch Embedding.\n Args:\n patch_size (int): Patch token size. Default: (2,4,4).\n in_chans (int): Number of input video channels. Default: 3.\n embed_dim (int): Number of linear projection output channels. Default: 96.\n norm_layer (nn.Layer, optional): Normalization layer. Default: None\n \"\"\"\n def __init__(self,\n patch_size=(2, 4, 4),\n in_chans=3,\n embed_dim=96,\n norm_layer=None):\n super().__init__()\n self.patch_size = patch_size",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:523-551"
+ },
+ "7009": {
+ "file_id": 514,
+ "content": "This code implements a PatchEmbed3D class, which embeds input video frames into patches for use in the Swin Transformer model. It takes the input video frames, divides them into non-overlapping patches, and performs linear projections on the patches to obtain embeddings. The patch size, number of input channels, and embedding dimension are configurable parameters.",
+ "type": "comment"
+ },
+ "7010": {
+ "file_id": 514,
+ "content": " self.in_chans = in_chans\n self.embed_dim = embed_dim\n self.proj = nn.Conv3D(in_chans,\n embed_dim,\n kernel_size=patch_size,\n stride=patch_size)\n if norm_layer is not None:\n self.norm = norm_layer(embed_dim)\n else:\n self.norm = None\n def forward(self, x):\n _, _, D, H, W = x.shape\n if W % self.patch_size[2] != 0:\n x = F.pad(\n x, (0, self.patch_size[2] - W % self.patch_size[2], 0, 0, 0, 0),\n data_format='NCDHW')\n if H % self.patch_size[1] != 0:\n x = F.pad(\n x, (0, 0, 0, self.patch_size[1] - H % self.patch_size[1], 0, 0),\n data_format='NCDHW')\n if D % self.patch_size[0] != 0:\n x = F.pad(\n x, (0, 0, 0, 0, 0, self.patch_size[0] - D % self.patch_size[0]),\n data_format='NCDHW')\n x = self.proj(x) # B C D Wh Ww\n if self.norm is not None:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:553-581"
+ },
+ "7011": {
+ "file_id": 514,
+ "content": "This code is for the Swin Transformer backbone in PaddleVideo. It initializes the module with input channels (in_chans), embed dim, and patch size. If a norm layer is provided, it also initializes the normalization layer (norm). The forward function pads the input according to the dimensions and applies a convolution operation for feature extraction. If a normalization layer was initialized, it performs normalization on the features before returning them.",
+ "type": "comment"
+ },
+ "7012": {
+ "file_id": 514,
+ "content": " D, Wh, Ww = x.shape[2], x.shape[3], x.shape[4]\n x = x.flatten(2).transpose([0, 2, 1])\n x = self.norm(x)\n x = x.transpose([0, 2, 1]).reshape([-1, self.embed_dim, D, Wh, Ww])\n return x\n@BACKBONES.register()\nclass SwinTransformer3D(nn.Layer):\n \"\"\" Swin Transformer backbone.\n A Paddle impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` -\n https://arxiv.org/pdf/2103.14030\n Args:\n patch_size (int | tuple(int)): Patch size. Default: (4,4,4).\n in_chans (int): Number of input image channels. Default: 3.\n embed_dim (int): Number of linear projection output channels. Default: 96.\n depths (tuple[int]): Depths of each Swin Transformer stage.\n num_heads (tuple[int]): Number of attention head of each stage.\n window_size (int): Window size. Default: 7.\n mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.\n qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: Truee",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:582-604"
+ },
+ "7013": {
+ "file_id": 514,
+ "content": "This code defines the Swin Transformer 3D backbone for Paddle Video. It takes an input tensor and performs normalization, transposition, and reshaping operations before returning the processed tensor. The class also registers with BACKBONES to be recognized as a valid backbone model.",
+ "type": "comment"
+ },
+ "7014": {
+ "file_id": 514,
+ "content": " qk_scale (float): Override default qk scale of head_dim ** -0.5 if set.\n drop_rate (float): Dropout rate.\n attn_drop_rate (float): Attention dropout rate. Default: 0.\n drop_path_rate (float): Stochastic depth rate. Default: 0.2.\n norm_layer: Normalization layer. Default: nn.LayerNorm.\n patch_norm (bool): If True, add normalization after patch embedding. Default: False.\n frozen_stages (int): Stages to be frozen (stop grad and set eval mode).\n -1 means not freezing any parameters.\n \"\"\"\n def __init__(self,\n pretrained=None,\n patch_size=(4, 4, 4),\n in_chans=3,\n embed_dim=96,\n depths=[2, 2, 6, 2],\n num_heads=[3, 6, 12, 24],\n window_size=(2, 7, 7),\n mlp_ratio=4.,\n qkv_bias=True,\n qk_scale=None,\n drop_rate=0.,\n attn_drop_rate=0.,\n drop_path_rate=0.2,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:605-627"
+ },
+ "7015": {
+ "file_id": 514,
+ "content": "This code defines the initialization parameters for the SWIN Transformer model in PaddleVideo. Parameters include pretrained weights, patch size, input channels, embedding dimension, depths of each stage, number of heads per stage, window size, MLP ratio, qkv_bias, qk scale, drop rate, attn drop rate, and stochastic depth rate. The normalization layer and whether to freeze any stages can also be specified during initialization.",
+ "type": "comment"
+ },
+ "7016": {
+ "file_id": 514,
+ "content": " norm_layer=nn.LayerNorm,\n patch_norm=False,\n frozen_stages=-1,\n use_checkpoint=False):\n super().__init__()\n self.pretrained = pretrained\n self.num_layers = len(depths)\n self.embed_dim = embed_dim\n self.patch_norm = patch_norm\n self.frozen_stages = frozen_stages\n self.window_size = window_size\n self.patch_size = patch_size\n # split image into non-overlapping patches\n self.patch_embed = PatchEmbed3D(\n patch_size=patch_size,\n in_chans=in_chans,\n embed_dim=embed_dim,\n norm_layer=norm_layer if self.patch_norm else None)\n self.pos_drop = nn.Dropout(p=drop_rate)\n # stochastic depth\n dpr = [\n x.item() for x in paddle.linspace(0, drop_path_rate, sum(depths))\n ] # stochastic depth decay rule\n # build layers\n self.layers = nn.LayerList()\n for i_layer in range(self.num_layers):\n layer = BasicLayer(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:628-659"
+ },
+ "7017": {
+ "file_id": 514,
+ "content": "The code initializes a Swin Transformer model with specified parameters, including depths, embed dimension, patch size, window size, and input channels. It creates the patch embedding layer and position dropout layer. Stochastic depth is applied using a decay rule. The layers are built using BasicLayer instances for each layer in the specified number of layers.",
+ "type": "comment"
+ },
+ "7018": {
+ "file_id": 514,
+ "content": " dim=int(embed_dim * 2**i_layer),\n depth=depths[i_layer],\n num_heads=num_heads[i_layer],\n window_size=window_size,\n mlp_ratio=mlp_ratio,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n drop=drop_rate,\n attn_drop=attn_drop_rate,\n drop_path=dpr[sum(depths[:i_layer]):sum(depths[:i_layer + 1])],\n norm_layer=norm_layer,\n downsample=PatchMerging\n if i_layer < self.num_layers - 1 else None,\n use_checkpoint=use_checkpoint)\n self.layers.append(layer)\n self.num_features = int(embed_dim * 2**(self.num_layers - 1))\n # add a norm layer for each output\n self.norm = norm_layer(self.num_features)\n self._freeze_stages()\n def _freeze_stages(self):\n if self.frozen_stages >= 0:\n self.patch_embed.eval()\n for param in self.patch_embed.parameters():\n param.stop_gradient = True",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:660-687"
+ },
+ "7019": {
+ "file_id": 514,
+ "content": "This code initializes a Swin Transformer backbone with specified parameters and adds a norm layer for each output. It also includes a function to freeze certain stages of the model if desired.",
+ "type": "comment"
+ },
+ "7020": {
+ "file_id": 514,
+ "content": " if self.frozen_stages >= 1:\n self.pos_drop.eval()\n for i in range(0, self.frozen_stages):\n m = self.layers[i]\n m.eval()\n for param in m.parameters():\n param.stop_gradient = True\n def _init_fn(self, m):\n if isinstance(m, nn.Linear):\n trunc_normal_(m.weight, std=0.02)\n if m.bias is not None:\n zeros_(m.bias)\n elif isinstance(m, nn.LayerNorm):\n zeros_(m.bias)\n ones_(m.weight)\n def init_weights(self):\n \"\"\"Initialize the weights in backbone.\n Args:\n pretrained (str, optional): Path to pre-trained weights.\n Defaults to None.\n \"\"\"\n \"\"\"First init model's weight\"\"\"\n self.apply(self._init_fn)\n \"\"\"Second, if provide pretrained ckpt, load it\"\"\"\n if isinstance(\n self.pretrained, str\n ) and self.pretrained.strip() != \"\": # load pretrained weights\n load_ckpt(self, self.pretrained)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:689-720"
+ },
+ "7021": {
+ "file_id": 514,
+ "content": "This code is part of a backbone model's initialization. It first applies an initializer function to the layers, then checks if pretrained weights are provided and loads them if available. The frozen_stages variable determines how many stages of the model should be frozen (set to eval mode) during inference.",
+ "type": "comment"
+ },
+ "7022": {
+ "file_id": 514,
+ "content": " elif self.pretrained is None or self.pretrained.strip() == \"\":\n pass\n else:\n raise NotImplementedError\n def forward(self, x):\n \"\"\"Forward function.\"\"\"\n x = self.patch_embed(x)\n x = self.pos_drop(x)\n for layer in self.layers:\n x = layer(x)\n x = x.transpose([0, 2, 3, 4, 1])\n x = self.norm(x)\n x = x.transpose([0, 4, 1, 2, 3])\n return x\n def train(self, mode=True):\n \"\"\"Convert the model into training mode while keep layers freezed.\"\"\"\n super(SwinTransformer3D, self).train(mode)\n self._freeze_stages()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/swin_transformer.py:721-742"
+ },
+ "7023": {
+ "file_id": 514,
+ "content": "If pretrained is None or empty, do nothing. Else, raise NotImplementedError. Forward function processes input through patch embedding and positional dropout, iterates over layers, transposes dimensions, normalizes, and returns output. Train mode keeps layers unfrozen by calling the superclass method and freezing stages.",
+ "type": "comment"
+ },
+ "7024": {
+ "file_id": 515,
+ "content": "/paddlevideo/modeling/backbones/toshift_vit.py",
+ "type": "filepath"
+ },
+ "7025": {
+ "file_id": 515,
+ "content": "The code defines a VisionTransformer class, ToShiftVIT model and TokenShiftVisionTransformer for image processing with attention blocks, positional embeddings, dropout and normalization layers. It also supports pretrained checkpoints.",
+ "type": "summary"
+ },
+ "7026": {
+ "file_id": 515,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections.abc import Callable\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import Constant\nfrom ...utils import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import trunc_normal_\n__all__ = ['VisionTransformer']\nzeros_ = Constant(value=0.)\nones_ = Constant(value=1.)\ndef to_2tuple(x):\n return tuple([x] * 2)\ndef drop_path(x, drop_prob=0., training=False):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:1-37"
+ },
+ "7027": {
+ "file_id": 515,
+ "content": "The code defines a class for VisionTransformer backbones and imports necessary libraries. It includes functions like `to_2tuple` and `drop_path` for processing input data and implementing drop path operation, respectively. The code also handles initialization of zero and one constants.",
+ "type": "comment"
+ },
+ "7028": {
+ "file_id": 515,
+ "content": " \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...\n # issuecomment-532968956 ...\n See discussion: https://github.com/tensorflow/tpu/issues/494\n \"\"\"\n if drop_prob == 0. or not training:\n return x\n keep_prob = paddle.to_tensor(1 - drop_prob)\n shape = (paddle.shape(x)[0], ) + (1, ) * (x.ndim - 1)\n random_tensor = keep_prob + paddle.rand(shape).astype(x.dtype)\n random_tensor = paddle.floor(random_tensor) # binarize\n output = x.divide(keep_prob) * random_tensor\n return output\nclass DropPath(nn.Layer):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n \"\"\"\n def __init__(self, drop_prob=None):\n super(DropPath, self).__init__()\n self.drop_prob = drop_prob\n def forward(self, x):\n return drop_path(x, self.drop_prob, self.training)\nclass Identity(nn.Layer):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:38-65"
+ },
+ "7029": {
+ "file_id": 515,
+ "content": "Code implements Drop Paths (Stochastic Depth) for residual blocks. The function applies dropout probabilistically, and the class `DropPath` handles it during forward pass. `Identity` class serves as an identity mapping.",
+ "type": "comment"
+ },
+ "7030": {
+ "file_id": 515,
+ "content": " def __init__(self):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\nclass Mlp(nn.Layer):\n def __init__(self,\n in_features,\n hidden_features=None,\n out_features=None,\n act_layer=nn.GELU,\n drop=0.0):\n super().__init__()\n out_features = out_features or in_features\n hidden_features = hidden_features or in_features\n self.fc1 = nn.Linear(in_features, hidden_features)\n self.act = act_layer()\n self.fc2 = nn.Linear(hidden_features, out_features)\n self.drop = nn.Dropout(drop)\n def forward(self, x):\n x = self.fc1(x)\n x = self.act(x)\n x = self.drop(x)\n x = self.fc2(x)\n x = self.drop(x)\n return x\nclass Attention(nn.Layer):\n def __init__(self,\n dim,\n num_heads=8,\n qkv_bias=False,\n qk_scale=None,\n attn_drop=0.0,\n proj_drop=0.0):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:66-104"
+ },
+ "7031": {
+ "file_id": 515,
+ "content": "This code defines three classes: Identity, Mlp, and Attention. Identity is a simple class that returns its input unchanged. Mlp stands for Multi-Layer Perceptron and defines a feedforward neural network layer with optional hidden layers. The Attention class implements a self-attention mechanism commonly used in transformer models. It initializes the necessary parameters and applies dropout to the input and output of the attention operation.",
+ "type": "comment"
+ },
+ "7032": {
+ "file_id": 515,
+ "content": " super().__init__()\n self.num_heads = num_heads\n head_dim = dim // num_heads\n self.scale = qk_scale or head_dim**-0.5\n self.qkv = nn.Linear(dim, dim * 3, bias_attr=qkv_bias)\n self.proj = nn.Linear(dim, dim)\n self.proj_drop = nn.Dropout(proj_drop)\n self.attn_drop = nn.Dropout(attn_drop)\n def forward(self, x):\n N, C = x.shape[1:]\n qkv = self.qkv(x).reshape(\n (-1, N, 3, self.num_heads, C // self.num_heads)).transpose(\n (2, 0, 3, 1, 4))\n q, k, v = qkv[0], qkv[1], qkv[2]\n attn = (q.matmul(k.transpose((0, 1, 3, 2)))) * self.scale\n attn = nn.functional.softmax(attn, axis=-1)\n attn = self.attn_drop(attn)\n x = (attn.matmul(v)).transpose((0, 2, 1, 3)).reshape((-1, N, C))\n x = self.proj(x)\n x = self.proj_drop(x)\n return x\nclass Block(nn.Layer):\n def __init__(self,\n dim,\n num_heads,\n mlp_ratio=4.0,\n qkv_bias=False,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:105-138"
+ },
+ "7033": {
+ "file_id": 515,
+ "content": "The code defines a class representing a self-attention module, with parameters like dimension (dim), number of heads (num_heads), and optional bias for the QKV linear layer (qkv_bias). The class initializes these attributes and defines its forward function to compute attention scores and output.",
+ "type": "comment"
+ },
+ "7034": {
+ "file_id": 515,
+ "content": " qk_scale=None,\n drop=0.0,\n attn_drop=0.0,\n drop_path=0.1,\n act_layer=nn.GELU,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n num_segments = 8,\n fold_div = 4):\n #attention_type='divided_space_time',\n super().__init__()\n self.n_seg = num_segments #ckk\n self.foldP_div = fold_div #ckk\n #self.attention_type = attention_type\n if isinstance(norm_layer, str):\n self.norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:139-164"
+ },
+ "7035": {
+ "file_id": 515,
+ "content": "The code above initializes an object with multiple parameters such as num_segments, fold_div, norm_layer, and attention_type. It also creates a norm1 layer based on the type of norm_layer provided (either a string or a Callable). If norm_layer is a string, it uses eval() to call the specified class, otherwise if it's a Callable, it directly initializes the layer with that function.",
+ "type": "comment"
+ },
+ "7036": {
+ "file_id": 515,
+ "content": " attn_drop=attn_drop,\n proj_drop=drop)\n # Temporal Attention Parameters\n '''\n if self.attention_type == 'divided_space_time':\n if isinstance(norm_layer, str):\n self.temporal_norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.temporal_norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.temporal_attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop)\n self.temporal_fc = nn.Linear(dim, dim)\n '''\n # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:165-186"
+ },
+ "7037": {
+ "file_id": 515,
+ "content": "This code initializes the Temporal Attention parameters for the model. If the attention_type is 'divided_space_time', it creates a temporal normalization layer and an attention layer, as well as a fully connected layer for the temporal branch of the model. Drop paths are used for stochastic depth to reduce overfitting.",
+ "type": "comment"
+ },
+ "7038": {
+ "file_id": 515,
+ "content": " self.drop_path = DropPath(drop_path) if drop_path > 0. else Identity()\n if isinstance(norm_layer, str):\n self.norm2 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.norm2 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = Mlp(in_features=dim,\n hidden_features=mlp_hidden_dim,\n act_layer=act_layer,\n drop=drop)\n # token_shift\n def shuift_tk(self, x):\n t = self.n_seg\n bt, n, c = x.shape\n b = bt // t\n x = x.reshape([b, t, n, c]) #B T N C\n fold = c // self.foldP_div\n out = paddle.zeros_like(x)\n out.stop_gradient = True\n # print(\"#### fold \", fold)\n # print(out.shape)\n # print(x[:, 1:, 0, :fold].unsqueeze(2).shape)\n # print(out[:, :-1, 0:1, :fold].shape)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:187-213"
+ },
+ "7039": {
+ "file_id": 515,
+ "content": "This code initializes a ToShift ViT model. It creates a drop path layer, normalization layer, and MLP based on the given parameters. The `shuift_tk` function performs token shifting by reshaping the input, creating a mask with stop gradient attribute, and element-wise adding it to the original input. This helps in improving the model's performance by reducing the effect of irrelevant tokens during training.",
+ "type": "comment"
+ },
+ "7040": {
+ "file_id": 515,
+ "content": " # exit(0)\n out[:, :-1, 0, :fold] = x[:, 1:, 0, :fold] # shift left\n out[:, 1:, 0, fold:2*fold] = x[:,:-1:, 0, fold:2*fold]\n out[:, :, 1:, :2*fold] = x[:, :, 1:, :2*fold]\n out[:, :, :, 2*fold:] = x[:, :, :, 2*fold:]\n return out.reshape([bt, n, c])\n def forward(self, x):\n x = self.shuift_tk(x)\n x = x + self.drop_path(self.attn(self.norm1(x)))\n x = self.shuift_tk(x)\n x = x + self.drop_path(self.mlp(self.norm2(x)))\n return x\nclass PatchEmbed(nn.Layer):\n \"\"\" Image to Patch Embedding\n \"\"\"\n def __init__(self,\n img_size=224,\n patch_size=16,\n in_channels=3,\n embed_dim=768):\n super().__init__()\n img_size = to_2tuple(img_size)\n patch_size = to_2tuple(patch_size)\n num_patches = (img_size[1] // patch_size[1]) * (img_size[0] //\n patch_size[0])\n self.img_size = img_size\n self.patch_size = patch_size",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:214-245"
+ },
+ "7041": {
+ "file_id": 515,
+ "content": "This code defines a \"ToshiftVIT\" class, which appears to be a custom backbone for a Vision Transformer model. It includes a forward function that applies shift and drop path operations on the input, as well as a PatchEmbed class for image-to-patch embedding. The ToshiftVIT class also has an unknown \"shuift_tk\" function that seems to be used in the forward pass.",
+ "type": "comment"
+ },
+ "7042": {
+ "file_id": 515,
+ "content": " self.num_patches = num_patches\n self.proj = nn.Conv2D(in_channels,\n embed_dim,\n kernel_size=patch_size,\n stride=patch_size)\n def forward(self, x):\n B, C, T, H, W = x.shape\n assert H == self.img_size[0] and W == self.img_size[1], \\\n f\"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]}).\"\n x = x.transpose((0, 2, 1, 3, 4))\n x = x.reshape([-1, C, H, W])\n x = self.proj(x)\n W = x.shape[-1]\n x = x.flatten(2).transpose((0, 2, 1))\n return x, T, W\n@BACKBONES.register()\nclass TokenShiftVisionTransformer(nn.Layer):\n \"\"\" Vision Transformer with support for patch input\n \"\"\"\n def __init__(self,\n pretrained=None,\n img_size=224,\n patch_size=16,\n in_channels=3,\n embed_dim=768,\n depth=12,\n num_heads=12,\n mlp_ratio=4,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:246-278"
+ },
+ "7043": {
+ "file_id": 515,
+ "content": "This code defines a TokenShiftVisionTransformer class, which is a type of Vision Transformer model that supports patch input. The class has an initialization function where it sets the number of patches and creates a projection layer. It also includes a forward function for processing input data through the model. The assert statement ensures the input image size matches the expected model dimensions. The @BACKBONES.register() decorator registers the model with other backbones in the codebase.",
+ "type": "comment"
+ },
+ "7044": {
+ "file_id": 515,
+ "content": " qkv_bias=False,\n qk_scale=None,\n drop_rate=0,\n attn_drop_rate=0.,\n drop_path_rate=0.1,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n num_seg=8,\n attention_type='divided_space_time',\n **args):\n super().__init__()\n self.pretrained = pretrained\n self.num_seg = num_seg\n self.attention_type = attention_type\n self.num_features = self.embed_dim = embed_dim\n self.patch_embed = PatchEmbed(img_size=img_size,\n patch_size=patch_size,\n in_channels=in_channels,\n embed_dim=embed_dim)\n num_patches = self.patch_embed.num_patches\n # Positional Embeddings\n self.cls_token = self.create_parameter(shape=(1, 1, embed_dim),\n default_initializer=zeros_)\n self.pos_embed = self.create_parameter(shape=(1, num_patches + 1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:279-305"
+ },
+ "7045": {
+ "file_id": 515,
+ "content": "This code is initializing a class for the Toshift ViT backbone. It sets parameters such as pretrained, num_seg, attention_type, embed_dim, and others. It creates PatchEmbed and positional embeddings (cls_token and pos_embed). The code also calculates the number of patches.",
+ "type": "comment"
+ },
+ "7046": {
+ "file_id": 515,
+ "content": " embed_dim),\n default_initializer=zeros_)\n self.pos_drop = nn.Dropout(p=drop_rate)\n if self.attention_type != 'space_only':\n self.time_embed = self.create_parameter(shape=(1, num_seg,\n embed_dim),\n default_initializer=zeros_)\n self.time_drop = nn.Dropout(p=drop_rate)\n self.add_parameter(\"pos_embed\", self.pos_embed)\n self.add_parameter(\"cls_token\", self.cls_token)\n dpr = np.linspace(0, drop_path_rate, depth)\n self.blocks = nn.LayerList([\n Block(dim=embed_dim,\n num_heads=num_heads,\n mlp_ratio=mlp_ratio,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n drop=drop_rate,\n attn_drop=attn_drop_rate,\n drop_path=dpr[i],\n norm_layer=norm_layer,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:306-330"
+ },
+ "7047": {
+ "file_id": 515,
+ "content": "This code initializes and sets up the parameters for a Transformer-based backbone model. It creates positional embeddings, dropout layers, and a list of transformer blocks with specified dimensions, numbers of heads, ratios, biases, scale factors, drop rates, attn drop rates, and drop path rates. These parameters are used to build the network architecture for processing data in downstream tasks.",
+ "type": "comment"
+ },
+ "7048": {
+ "file_id": 515,
+ "content": " epsilon=epsilon,\n num_segments= self.num_seg\n ) for i in range(depth)\n #attention_type=self.attention_type\n ])\n self.norm = eval(norm_layer)(embed_dim, epsilon=epsilon)\n def init_weights(self):\n \"\"\"First init model's weight\"\"\"\n trunc_normal_(self.pos_embed, std=0.02)\n trunc_normal_(self.cls_token, std=0.02)\n self.apply(self._init_fn)\n if self.attention_type == 'divided_space_time':\n i = 0\n for m in self.blocks.sublayers(include_self=True):\n m_str = str(m)\n if 'Block' in m_str:\n if i > 0:\n zeros_(m.temporal_fc.weight)\n zeros_(m.temporal_fc.bias)\n i += 1\n \"\"\"Second, if provide pretrained ckpt, load it\"\"\"\n if isinstance(\n self.pretrained, str\n ) and self.pretrained.strip() != \"\": # load pretrained weights\n load_ckpt(self,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:331-360"
+ },
+ "7049": {
+ "file_id": 515,
+ "content": "The code initializes a Toshift_VIT model with the specified number of segments and depth. It sets the attention type to 'divided_space_time' for certain blocks. The model's weights are then initialized using truncated normal distribution and the provided function, and any temporal FC layers in the respective block are set to zero. If a pretrained checkpoint is provided, it will be loaded.",
+ "type": "comment"
+ },
+ "7050": {
+ "file_id": 515,
+ "content": " self.pretrained,\n num_patches=self.patch_embed.num_patches,\n num_seg=self.num_seg,\n attention_type=self.attention_type)\n def _init_fn(self, m):\n if isinstance(m, nn.Linear):\n trunc_normal_(m.weight)\n if m.bias is not None:\n zeros_(m.bias)\n elif isinstance(m, nn.LayerNorm):\n ones_(m.weight)\n zeros_(m.bias)\n def forward_features(self, x):\n # B = x.shape[0]\n B = paddle.shape(x)[0]\n x, T, W = self.patch_embed(x) # [BT,nH*nW,F]\n cls_tokens = self.cls_token.expand((B * T, -1, -1)) # [1,1,F]->[BT,1,F]\n x = paddle.concat((cls_tokens, x), axis=1)\n pos_interp = (x.shape[1] != self.pos_embed.shape[1])\n if pos_interp:\n pos_embed = self.pos_embed\n cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)\n other_pos_embed = pos_embed[0, 1:, :].unsqueeze(0).transpose(\n (0, 2, 1))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:361-386"
+ },
+ "7051": {
+ "file_id": 515,
+ "content": "The code initializes a TOShiftViT model, defines an initialization function for the layers, and defines a forward_features function to process input feature maps. The function takes the number of patches from the patch embedding layer, expands the class token, concatenates it with the features, and applies positional embeddings if needed.",
+ "type": "comment"
+ },
+ "7052": {
+ "file_id": 515,
+ "content": " P = int(other_pos_embed.shape[2]**0.5)\n H = x.shape[1] // W\n other_pos_embed = other_pos_embed.reshape([1, x.shape[2], P, P])\n new_pos_embed = F.interpolate(other_pos_embed,\n size=(H, W),\n mode='nearest')\n new_pos_embed = new_pos_embed.flatten(2)\n new_pos_embed = new_pos_embed.transpose((0, 2, 1))\n new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed),\n axis=1)\n x = x + new_pos_embed\n else:\n x = x + self.pos_embed\n x = self.pos_drop(x)\n # Attention blocks\n for blk in self.blocks:\n x = blk(x)\n x = self.norm(x)\n return x[:, 0] # [B, embed_dim] -> [B*T, embed_dim]\n def forward(self, x):\n x = self.forward_features(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/toshift_vit.py:387-413"
+ },
+ "7053": {
+ "file_id": 515,
+ "content": "This function reshapes and interpolates a positional embedding, adding it to the input if specified. It then performs positional dropout before passing through attention blocks and normalization layers, finally returning the forward pass of features.",
+ "type": "comment"
+ },
+ "7054": {
+ "file_id": 516,
+ "content": "/paddlevideo/modeling/backbones/transnetv2.py",
+ "type": "filepath"
+ },
+ "7055": {
+ "file_id": 516,
+ "content": "OctConv3D is a configurable 3D convolutional layer in TransNetV2's backbone, utilizing features such as max pooling and SDDCNNV2 blocks for shot transition detection. ConvNextV2 applies feature extraction and pooling, while the code defines models using Linear and ConvexCombinationRegularization layers for classification tasks.",
+ "type": "summary"
+ },
+ "7056": {
+ "file_id": 516,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as functional\nimport random\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\nclass OctConv3D(nn.Layer):\n def __init__(self, in_filters, filters, kernel_size=3, dilation_rate=(1, 1, 1), alpha=0.25,\n use_bias=True, kernel_initializer=nn.initializer.KaimingNormal()):\n super(OctConv3D, self).__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:1-28"
+ },
+ "7057": {
+ "file_id": 516,
+ "content": "This code defines a 3D convolutional neural network layer called OctConv3D. It takes input and output channels, kernel size, dilation rate, alpha (for octave pooling), use_bias flag, and initializer as parameters for creating the layer. This layer can be used in other models by utilizing the BACKBONES registry.",
+ "type": "comment"
+ },
+ "7058": {
+ "file_id": 516,
+ "content": " self.low_channels = int(filters * alpha)\n self.high_channels = filters - self.low_channels\n self.high_to_high = nn.Conv3D(in_filters, self.high_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),\n weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.high_to_low = nn.Conv3D(self.high_channels, self.low_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),\n weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=False)\n self.low_to_high = nn.Conv3D(in_filters, self.high_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:30-43"
+ },
+ "7059": {
+ "file_id": 516,
+ "content": "Defines a 3D Convolutional network with interleaved low and high-resolution paths. Low-to-high and high-to-low convolutions are performed to maintain spatial resolution while reducing dimensionality for the TransNetV2 backbone model.",
+ "type": "comment"
+ },
+ "7060": {
+ "file_id": 516,
+ "content": " weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=False)\n self.low_to_low = nn.Conv3D(self.high_channels, self.low_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),\n weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.upsampler = nn.Upsample(size=(1, 2, 2), data_format='NCDHW')\n self.downsampler = nn.AvgPool3D(kernel_size=(1, 2, 2), stride=(1, 2, 2), padding=(0, 1, 1))\n @staticmethod\n def pad_to(tensor, target_shape):\n shape = tensor.shape\n padding = [[0, tar - curr] for curr, tar in zip(shape, target_shape)]\n return functional.pad(tensor, padding, \"CONSTANT\", data_format='NCDHW')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:44-58"
+ },
+ "7061": {
+ "file_id": 516,
+ "content": "This code defines a TransNetV2 backbone model for video analysis. It includes convolutional layers, an upsampler, and downsampler to process input data. The `pad_to` function pads the tensor with zeros to match a target shape, useful for maintaining consistent dimensions throughout the model.",
+ "type": "comment"
+ },
+ "7062": {
+ "file_id": 516,
+ "content": " @staticmethod\n def crop_to(tensor, target_width, target_height):\n return tensor[:, :, :target_height, :target_width]\n def forward(self, inputs):\n low_inputs, high_inputs = inputs\n high_to_high = self.high_to_high(high_inputs)\n high_to_low = self.high_to_low(self.downsampler(high_inputs))\n low_to_high = self.upsampler(self.low_to_high(low_inputs))\n low_to_low = self.low_to_low(low_inputs)\n high_output = high_to_high[:, :, :, :low_to_high.shape[3], :low_to_high.shape[4]] + low_to_high\n low_output = low_to_low + high_to_low[:, :, :, :low_to_low.shape[3], :low_to_low.shape[4]]\n return low_output, high_output\nclass Conv3DConfigurable(nn.Layer):\n def __init__(self,\n in_filters,\n filters,\n dilation_rate,\n separable=True,\n octave=False,\n use_bias=True):\n super(Conv3DConfigurable, self).__init__()\n assert not (separable and octave)\n if separable:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:60-90"
+ },
+ "7063": {
+ "file_id": 516,
+ "content": "The code defines a forward function that takes inputs and performs high-to-high, high-to-low, low-to-high, and low-to-low transformations. It also includes a Conv3DConfigurable class with parameters for in_filters, filters, dilation_rate, separable, octave, and use_bias. The code asserts that separable and octave cannot both be True.",
+ "type": "comment"
+ },
+ "7064": {
+ "file_id": 516,
+ "content": " conv1 = nn.Conv3D(in_filters, 2 * filters, kernel_size=(1, 3, 3),\n dilation=(1, 1, 1), padding=(0, 1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.KaimingNormal()),\n bias_attr=False)\n conv2 = nn.Conv3D(2 * filters, filters, kernel_size=(3, 1, 1),\n dilation=(dilation_rate, 1, 1), padding=(dilation_rate, 0, 0),\n weight_attr=ParamAttr(initializer=nn.initializer.KaimingNormal()),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.layers = nn.LayerList([conv1, conv2])\n elif octave:\n conv = OctConv3D(in_filters, filters, kernel_size=3, dilation_rate=(dilation_rate, 1, 1),\n use_bias=use_bias,\n kernel_initializer=nn.initializer.KaimingNormal())",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:91-104"
+ },
+ "7065": {
+ "file_id": 516,
+ "content": "The code initializes a Conv3D layer and an optional octave convolution layer for the TransNetV2 backbone. The Conv3D layers apply 3x3 kernel with varying dilation rates, while the optional OctConv3D layer has a 3x1x1 kernel and dilation rate (dilation_rate, 1, 1). The layers are added to a LayerList for further processing in the network.",
+ "type": "comment"
+ },
+ "7066": {
+ "file_id": 516,
+ "content": " self.layers = [conv]\n else:\n conv = nn.Conv3D(in_filters, filters, kernel_size=3,\n dilation=(dilation_rate, 1, 1), padding=(dilation_rate, 1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.KaimingNormal()),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.layers = nn.LayerList([conv])\n def forward(self, inputs):\n x = inputs\n for layer in self.layers:\n x = layer(x)\n return x\nclass DilatedDCNNV2(nn.Layer):\n def __init__(self,\n in_filters,\n filters,\n batch_norm=True,\n activation=None,\n octave_conv=False):\n super(DilatedDCNNV2, self).__init__()\n assert not (octave_conv and batch_norm)\n self.Conv3D_1 = Conv3DConfigurable(in_filters, filters, 1, use_bias=not batch_norm, octave=octave_conv)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:105-131"
+ },
+ "7067": {
+ "file_id": 516,
+ "content": "This code defines a neural network backbone called TransnetV2, which consists of convolutional layers. The Conv3DConfigurable class is used to configure the layers with specified input and output filters, kernel size, dilation rate, padding, and whether to use bias or batch normalization. The DilatedDCNNV2 class extends this concept by allowing the choice between octave convolution and batch normalization. Both classes inherit from nn.Layer and have a forward method for processing inputs.",
+ "type": "comment"
+ },
+ "7068": {
+ "file_id": 516,
+ "content": " self.Conv3D_2 = Conv3DConfigurable(in_filters, filters, 2, use_bias=not batch_norm, octave=octave_conv)\n self.Conv3D_4 = Conv3DConfigurable(in_filters, filters, 4, use_bias=not batch_norm, octave=octave_conv)\n self.Conv3D_8 = Conv3DConfigurable(in_filters, filters, 8, use_bias=not batch_norm, octave=octave_conv)\n self.octave = octave_conv\n self.bn = nn.BatchNorm3D(filters * 4, momentum=0.99, epsilon=1e-03,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n ) if batch_norm else None\n self.activation = activation\n def forward(self, inputs):\n conv1 = self.Conv3D_1(inputs)\n conv2 = self.Conv3D_2(inputs)\n conv3 = self.Conv3D_4(inputs)\n conv4 = self.Conv3D_8(inputs)\n # shape of convi[j]/convi is [B, 3, T, H, W], concat in channel dimension\n if self.octave:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:132-150"
+ },
+ "7069": {
+ "file_id": 516,
+ "content": "This code defines a TransNetV2 model, which uses multiple Conv3D layers to process input data. The model includes configurable convolution layers with different filter sizes (2, 4, and 8), batch normalization, and activation functions. The forward method applies these layers to the inputs and concatenates their outputs along the channel dimension.",
+ "type": "comment"
+ },
+ "7070": {
+ "file_id": 516,
+ "content": " x = [paddle.concat([conv1[0], conv2[0], conv3[0], conv4[0]], axis=1),\n paddle.concat([conv1[1], conv2[1], conv3[1], conv4[1]], axis=1)]\n else:\n x = paddle.concat([conv1, conv2, conv3, conv4], axis=1)\n if self.bn is not None:\n x = self.bn(x)\n if self.activation is not None:\n if self.octave:\n x = [self.activation(x[0]), self.activation(x[1])]\n else:\n x = self.activation(x)\n return x\nclass StackedDDCNNV2(nn.Layer):\n def __init__(self,\n in_filters,\n n_blocks,\n filters,\n shortcut=True,\n use_octave_conv=False,\n pool_type=\"avg\",\n stochastic_depth_drop_prob=0.0):\n super(StackedDDCNNV2, self).__init__()\n assert pool_type == \"max\" or pool_type == \"avg\"\n if use_octave_conv and pool_type == \"max\":\n print(\"WARN: Octave convolution was designed with average pooling, not max pooling.\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:151-179"
+ },
+ "7071": {
+ "file_id": 516,
+ "content": "The code defines a StackedDDCNNV2 class that is a type of neural network layer. It takes in parameters such as number of input filters, number of blocks, and output filters. The class uses convolutions with optional batch normalization and activation functions. The convolutions can be either octave or non-octave depending on the parameter setting. The pooling type is either max or average pooling, and there is a stochastic depth drop probability parameter.",
+ "type": "comment"
+ },
+ "7072": {
+ "file_id": 516,
+ "content": " self.shortcut = shortcut\n self.DDCNN = nn.LayerList([\n DilatedDCNNV2(in_filters if i == 1 else filters * 4, filters, octave_conv=use_octave_conv,\n activation=functional.relu if i != n_blocks else None) for i in range(1, n_blocks + 1)\n ])\n self.pool = nn.MaxPool3D(kernel_size=(1, 2, 2)) if pool_type == \"max\" else nn.AvgPool3D(kernel_size=(1, 2, 2))\n self.octave = use_octave_conv\n self.stochastic_depth_drop_prob = stochastic_depth_drop_prob\n def forward(self, inputs):\n x = inputs\n shortcut = None\n if self.octave:\n x = [self.pool(x), x]\n for block in self.DDCNN:\n x = block(x)\n if shortcut is None:\n shortcut = x\n # shape of x[i] is [B, 3, T, H, W], concat in channel dimension\n if self.octave:\n x = paddle.concat([x[0], self.pool(x[1])], axis=1)\n x = functional.relu(x)\n if self.shortcut is not None:\n if self.stochastic_depth_drop_prob != 0.:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:181-207"
+ },
+ "7073": {
+ "file_id": 516,
+ "content": "Initializes backbone layers and sets parameters. Applies octave convolution if use_octave_conv is True, and performs max or avg pooling depending on pool_type. Stochastic depth is applied with probability stochastic_depth_drop_prob. Forward pass applies blocks of DDCNNV2, concatenates and applies ReLU activation.",
+ "type": "comment"
+ },
+ "7074": {
+ "file_id": 516,
+ "content": " if self.training:\n if random.random() < self.stochastic_depth_drop_prob:\n x = shortcut\n else:\n x = x + shortcut\n else:\n x = (1 - self.stochastic_depth_drop_prob) * x + shortcut\n else:\n x += shortcut\n if not self.octave:\n x = self.pool(x)\n return x\nclass ResNetBlock(nn.Layer):\n def __init__(self, in_filters, filters, strides=(1, 1)):\n super(ResNetBlock, self).__init__()\n self.conv1 = nn.Conv2D(in_filters, filters, kernel_size=(3, 3), stride=strides, padding=(1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.bn1 = nn.BatchNorm2D(filters,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:208-232"
+ },
+ "7075": {
+ "file_id": 516,
+ "content": "This code defines a ResNetBlock class that consists of Conv2D layer and BatchNorm2D layer. The stochastic depth is applied during training by randomly dropping connections with a specified probability, while in non-octave cases, it applies pooling to the output.",
+ "type": "comment"
+ },
+ "7076": {
+ "file_id": 516,
+ "content": " self.conv2 = nn.Conv2D(filters, filters, kernel_size=(3, 3), padding=(1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.bn2 = nn.BatchNorm2D(filters,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))\n def forward(self, inputs):\n x = self.conv1(inputs)\n x = self.bn1(x)\n x = functional.relu(x)\n x = self.conv2(x)\n x = self.bn2(x)\n shortcut = inputs\n x += shortcut\n return functional.relu(x)\nclass ResNetFeatures(nn.Layer):\n def __init__(self, in_filters=3,\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]):\n super(ResNetFeatures, self).__init__()\n self.conv1 = nn.Conv2D(in_channels=in_filters, out_channels=64, kernel_size=(7, 7),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:234-260"
+ },
+ "7077": {
+ "file_id": 516,
+ "content": "The code defines a Conv2D layer and BatchNorm2D layer in the `TransNetV2` class, followed by a forward function that applies these layers in sequence. The ResNetFeatures class initializes a Conv2D layer for extracting features from input images. Both classes are part of an object-oriented model architecture.",
+ "type": "comment"
+ },
+ "7078": {
+ "file_id": 516,
+ "content": " stride=(2, 2), padding=(3, 3),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.bn1 = nn.BatchNorm2D(num_features=64, momentum=0.99, epsilon=1e-03,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n )\n self.max_pool = nn.MaxPool2D(kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n self.layer2a = ResNetBlock(64, 64)\n self.layer2b = ResNetBlock(64, 64)\n self.mean = paddle.to_tensor(mean)\n self.std = paddle.to_tensor(std)\n def forward(self, inputs):\n shape = inputs.shape\n x = paddle.reshape(inputs, [shape[0] * shape[2], shape[1], shape[3], shape[4]])\n x = (x - self.mean) / self.std\n x = self.conv1(x)\n x = self.bn1(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:261-282"
+ },
+ "7079": {
+ "file_id": 516,
+ "content": "This code is for TransNetV2 backbone model initialization. It includes a convolution layer with padding, batch normalization, max pooling, and ResNetBlocks (layer2a, layer2b). The forward function performs normalization, reshaping, convolution, and batch normalization on the input.",
+ "type": "comment"
+ },
+ "7080": {
+ "file_id": 516,
+ "content": " x = functional.relu(x)\n x = self.max_pool(x)\n x = self.layer2a(x)\n x = self.layer2b(x)\n new_shape = x.shape\n x = paddle.reshape(x, [shape[0], new_shape[1], shape[2], new_shape[2], new_shape[3]])\n return x\nclass FrameSimilarity(nn.Layer):\n def __init__(self,\n in_filters,\n similarity_dim=128,\n lookup_window=101,\n output_dim=128,\n stop_gradient=False,\n use_bias=False):\n super(FrameSimilarity, self).__init__()\n self.projection = nn.Linear(in_filters, similarity_dim,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=use_bias)\n self.fc = nn.Linear(lookup_window, output_dim,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:283-307"
+ },
+ "7081": {
+ "file_id": 516,
+ "content": "This code defines a class \"FrameSimilarity\" that takes in filters, similarity dimension, lookup window, output dimension, stop_gradient flag, and use_bias as parameters. It initializes the layer with a projection linear layer and an fc linear layer. The projection layer maps input features to a specified similarity dimension using XavierUniform initialization. The fc layer maps the lookup window to the output dimension, using XavierUniform initialization for weights and Constant initialization for biases.",
+ "type": "comment"
+ },
+ "7082": {
+ "file_id": 516,
+ "content": " self.lookup_window = lookup_window\n self.stop_gradient = stop_gradient\n assert lookup_window % 2 == 1, \"`lookup_window` must be odd integer\"\n def forward(self, inputs):\n x = paddle.concat([paddle.mean(x, axis=[3, 4]) for x in inputs], axis=1)\n x = paddle.transpose(x, (0, 2, 1))\n if self.stop_gradient:\n x = x.stop_gradient\n x = self.projection(x)\n x = functional.normalize(x, p=2, axis=2)\n batch_size = paddle.slice(x.shape, starts=[0], ends=[1], axes=[0]) if x.shape[0] == -1 else x.shape[0]\n time_window = x.shape[1]\n similarities = paddle.bmm(x, x.transpose([0, 2, 1])) # [batch_size, time_window, time_window]\n similarities_padded = functional.pad(similarities,\n [(self.lookup_window - 1) // 2, (self.lookup_window - 1) // 2],\n data_format='NCL')\n batch_indices = paddle.arange(0, batch_size).reshape([batch_size, 1, 1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:309-330"
+ },
+ "7083": {
+ "file_id": 516,
+ "content": "The code initializes a TransNetV2 model with lookup window and stop_gradient options. It then calculates similarities between time windows using batch mean, transpose, projection, and normalization. Finally, it pads the similarities for further calculations.",
+ "type": "comment"
+ },
+ "7084": {
+ "file_id": 516,
+ "content": " batch_indices = paddle.tile(batch_indices, [1, time_window, self.lookup_window])\n time_indices = paddle.arange(0, time_window).reshape([1, time_window, 1])\n time_indices = paddle.tile(time_indices, [batch_size, 1, self.lookup_window])\n lookup_indices = paddle.arange(0, self.lookup_window).reshape([1, 1, self.lookup_window])\n lookup_indices = paddle.tile(lookup_indices, [batch_size, time_window, 1]) + time_indices\n indices = paddle.stack([batch_indices, time_indices, lookup_indices], -1)\n similarities = paddle.gather_nd(similarities_padded, indices)\n return functional.relu(self.fc(similarities))\nclass ConvexCombinationRegularization(nn.Layer):\n def __init__(self, in_filters, filters=32, delta_scale=10., loss_weight=0.01):\n super(ConvexCombinationRegularization, self).__init__()\n self.projection = nn.Conv3D(in_filters, filters, kernel_size=1, dilation=1, padding=(0, 0, 0),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:331-346"
+ },
+ "7085": {
+ "file_id": 516,
+ "content": "This code is calculating the indices for gathering similarities from a padded tensor. It tiles and stacks batch, time, and lookup indices to create an array of valid indices. Then it uses these indices to gather similarities from the padded tensor and applies ReLU activation on top of an FC layer to return the output. The ConvexCombinationRegularization class initializes a projection layer with specified parameters.",
+ "type": "comment"
+ },
+ "7086": {
+ "file_id": 516,
+ "content": " bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))\n self.features = nn.Conv3D((filters * 3), filters * 2,\n kernel_size=(3, 3, 3), dilation=1, padding=(1, 1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))\n self.dense = nn.Linear(64, 1, weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()), bias_attr=True)\n self.loss = nn.SmoothL1Loss(reduction='none')\n self.delta_scale = delta_scale\n self.loss_weight = loss_weight\n def forward(self, image_inputs, feature_inputs):\n x = feature_inputs\n x = self.projection(x)\n x = functional.relu(x)\n batch_size = x.shape[0]\n window_size = x.shape[2]\n first_frame = paddle.tile(x[:, :, :1], [1, 1, window_size, 1, 1])\n last_frame = paddle.tile(x[:, :, -1:], [1, 1, window_size, 1, 1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:347-364"
+ },
+ "7087": {
+ "file_id": 516,
+ "content": "This code defines a Conv3D model for the TransNetV2 backbone. It has a projection layer, relu activation, and takes in image_inputs and feature_inputs. The forward function processes these inputs, extracting the first and last frame windows.",
+ "type": "comment"
+ },
+ "7088": {
+ "file_id": 516,
+ "content": " x = paddle.concat([x, first_frame, last_frame], 1)\n x = self.features(x)\n x = functional.relu(x)\n x = paddle.mean(x, axis=[3, 4])\n x = paddle.transpose(x, (0, 2, 1))\n alpha = self.dense(x)\n alpha = paddle.transpose(alpha, (0, 2, 1))\n first_img = paddle.tile(image_inputs[:, :, :1], [1, 1, window_size, 1, 1])\n last_img = paddle.tile(image_inputs[:, :, -1:], [1, 1, window_size, 1, 1])\n alpha_ = functional.sigmoid(alpha)\n alpha_ = paddle.reshape(alpha_, [batch_size, 1, window_size, 1, 1])\n predictions_ = (alpha_ * first_img + (1 - alpha_) * last_img)\n loss_ = self.loss(label=image_inputs / self.delta_scale, input=predictions_ / self.delta_scale)\n loss_ = self.loss_weight * paddle.mean(loss_)\n return alpha, loss_\nclass ColorHistograms(nn.Layer):\n def __init__(self,\n lookup_window=101,\n output_dim=None):\n super(ColorHistograms, self).__init__()\n self.fc = nn.Linear(lookup_window, output_dim,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:365-390"
+ },
+ "7089": {
+ "file_id": 516,
+ "content": "This code is part of the TransnetV2 model in PaddleVideo. It concatenates frames, processes them through layers, and calculates alpha values for first and last images. It then combines these images based on the calculated alphas and performs loss calculation using a loss function. The ColorHistograms layer is initialized with a linear transformation layer.",
+ "type": "comment"
+ },
+ "7090": {
+ "file_id": 516,
+ "content": " weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.))) if output_dim is not None else None\n self.lookup_window = lookup_window\n assert lookup_window % 2 == 1, \"`lookup_window` must be odd integer\"\n def compute_color_histograms(self, frames):\n frames = frames.astype('int32')\n def get_bin(frames):\n # returns 0 .. 511\n R, G, B = frames[:, :, 0], frames[:, :, 1], frames[:, :, 2]\n R, G, B = R // 32, G // 32, B // 32\n return (R * 64) + (G * 8) + B\n batch_size = paddle.slice(frames.shape, starts=[0], ends=[1], axes=[0]) if frames.shape[0] == -1 else frames.shape[0]\n time_window, height, width, no_channels = frames.shape[1:]\n assert no_channels == 3 or no_channels == 6\n if no_channels == 3:\n frames_flatten = frames.reshape([-1, height * width, 3])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:391-411"
+ },
+ "7091": {
+ "file_id": 516,
+ "content": "This code defines a function to compute color histograms of frames. It first converts the frame values to int32, then defines a function get_bin which extracts and scales RGB values. The batch size is extracted from the frames shape, and the frames are flattened into a 3-dimensional array if the number of channels is 3 or 6.",
+ "type": "comment"
+ },
+ "7092": {
+ "file_id": 516,
+ "content": " else:\n frames_flatten = frames.reshape([-1, height * width * 2, 3])\n binned_values = get_bin(frames_flatten)\n frame_bin_prefix = (paddle.arange(0, batch_size * time_window) * 512).reshape([-1, 1])\n binned_values = (binned_values + frame_bin_prefix).reshape([-1, 1])\n histograms = paddle.zeros_like(frame_bin_prefix, dtype='int32').tile([512]).reshape([-1])\n histograms = histograms.scatter_nd_add(binned_values, paddle.ones_like(binned_values, dtype='int32').reshape([-1]))\n histograms = histograms.reshape([batch_size, time_window, 512]).astype('float32')\n histograms_normalized = functional.normalize(histograms, p=2, axis=2)\n return histograms_normalized\n def forward(self, inputs):\n x = self.compute_color_histograms(inputs)\n batch_size = paddle.slice(x.shape, starts=[0], ends=[1], axes=[0]) if x.shape[0] == -1 else x.shape[0]\n time_window = x.shape[1]\n similarities = paddle.bmm(x, x.transpose([0, 2, 1])) # [batch_size, time_window, time_window]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:412-429"
+ },
+ "7093": {
+ "file_id": 516,
+ "content": "This code computes color histograms for each frame in a video and then calculates similarities between frames using batch matrix multiplication. It first checks the input shape to determine whether it should extract only the batch size and time window or use the total shape. It then reshapes and bins the frame values, normalizes the histograms, and finally computes the similarity matrix for each frame pair. The purpose is likely for video sequence analysis or comparison.",
+ "type": "comment"
+ },
+ "7094": {
+ "file_id": 516,
+ "content": " similarities_padded = functional.pad(similarities,\n [(self.lookup_window - 1) // 2, (self.lookup_window - 1) // 2],\n data_format='NCL')\n batch_indices = paddle.arange(0, batch_size).reshape([batch_size, 1, 1])\n batch_indices = paddle.tile(batch_indices, [1, time_window, self.lookup_window])\n time_indices = paddle.arange(0, time_window).reshape([1, time_window, 1])\n time_indices = paddle.tile(time_indices, [batch_size, 1, self.lookup_window])\n lookup_indices = paddle.arange(0, self.lookup_window).reshape([1, 1, self.lookup_window])\n lookup_indices = paddle.tile(lookup_indices, [batch_size, time_window, 1]) + time_indices\n indices = paddle.stack([batch_indices, time_indices, lookup_indices], -1)\n similarities = paddle.gather_nd(similarities_padded, indices)\n if self.fc is not None:\n return functional.relu(self.fc(similarities))\n return similarities",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:430-446"
+ },
+ "7095": {
+ "file_id": 516,
+ "content": "This code performs lookup on a padded tensor using gathered indices from batch, time window, and lookup window. It then applies an optional fully connected layer with ReLU activation function if present.",
+ "type": "comment"
+ },
+ "7096": {
+ "file_id": 516,
+ "content": "@BACKBONES.register()\nclass TransNetV2(nn.Layer):\n \"\"\"TransNetV2 model from\n `\"TransNet V2: An effective deep network architecture for fast shot transition detection\" `_\n \"\"\"\n def __init__(self,\n F=16, L=3, S=2, D=1024,\n use_many_hot_targets=True,\n use_frame_similarity=True,\n use_color_histograms=True,\n use_mean_pooling=False,\n dropout_rate=0.5,\n use_convex_comb_reg=False,\n use_resnet_features=False,\n use_resnet_like_top=False,\n frame_similarity_on_last_layer=False,\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]):\n super(TransNetV2, self).__init__()\n self.mean = np.array(mean, np.float32).reshape([1, 3, 1, 1]) * 255\n self.std = np.array(std, np.float32).reshape([1, 3, 1, 1]) * 255\n self.use_resnet_features = use_resnet_features\n s",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:449-473"
+ },
+ "7097": {
+ "file_id": 516,
+ "content": "The code defines the TransNetV2 model, a deep network architecture for shot transition detection. It has multiple input sources and various options to use or not use different features and operations. The mean and std are provided as initialization parameters to standardize the input data.",
+ "type": "comment"
+ },
+ "7098": {
+ "file_id": 516,
+ "content": "elf.resnet_layers = ResNetFeatures(in_filters=3, mean=self.mean, std=self.std) if self.use_resnet_features else None\n self.resnet_like_top = use_resnet_like_top\n if self.resnet_like_top:\n self.resnet_like_top_conv = nn.Conv3D(64 if self.use_resnet_features else 3, 32, kernel_size=(3, 7, 7),\n stride=(1, 2, 2),\n padding=(1, 3, 3),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.resnet_like_top_bn = nn.BatchNorm3D(32, momentum=0.99, epsilon=1e-03,\n weight_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:473-484"
+ },
+ "7099": {
+ "file_id": 516,
+ "content": "Code snippet is from PaddleVideo's TransNetV2 model. It checks if use_resnet_features is True and if so, initializes resnet_layers with ResNetFeatures. If resnet_like_top is also True, it then initializes resnet_like_top_conv and resnet_like_top_bn for ResNet-like top layers with specified parameters.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/71.json b/docs/data/71.json
new file mode 100644
index 000000000..cbe809f0f
--- /dev/null
+++ b/docs/data/71.json
@@ -0,0 +1,548 @@
+{
+ "7100": {
+ "file_id": 516,
+ "content": " self.resnet_like_top_max_pool = nn.MaxPool3D(kernel_size=(1, 3, 3), stride=(1, 2, 2),\n padding=(0, 1, 1))\n if self.resnet_like_top:\n in_filters = 32\n elif self.use_resnet_features:\n in_filters = 64\n else:\n in_filters = 3\n self.SDDCNN = nn.LayerList(\n [StackedDDCNNV2(in_filters=in_filters, n_blocks=S, filters=F,\n stochastic_depth_drop_prob=0.)] +\n [StackedDDCNNV2(in_filters=(F * 2 ** (i - 1)) * 4, n_blocks=S, filters=F * 2 ** i) for i in range(1, L)]\n )\n self.frame_sim_layer = FrameSimilarity(\n sum([(F * 2 ** i) * 4 for i in range(L)]), lookup_window=101, output_dim=128, similarity_dim=128,\n use_bias=True\n ) if use_frame_similarity else None\n self.color_hist_layer = ColorHistograms(\n lookup_window=101, output_dim=128\n ) if use_color_histograms else None\n self.dropout = nn.Dropout(dropout_rate) if dropout_rate is not None else None",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:485-508"
+ },
+ "7101": {
+ "file_id": 516,
+ "content": "This code initializes the model components of a TransNetv2 backbone. It sets up max pooling, creates a LayerList for SDDCNNV2 blocks, initializes frame similarity and color histogram layers based on flags, and includes dropout layer if needed.",
+ "type": "comment"
+ },
+ "7102": {
+ "file_id": 516,
+ "content": " output_dim = ((F * 2 ** (L - 1)) * 4) * 3 * 6 # 3x6 for spatial dimensions\n if use_frame_similarity: output_dim += 128\n if use_color_histograms: output_dim += 128\n self.use_mean_pooling = use_mean_pooling\n self.has_downsample = False\n if self.use_resnet_features or self.resnet_like_top or self.use_mean_pooling:\n self.has_downsample = True\n self.fc1 = nn.Linear(512 if self.has_downsample else output_dim, D,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n )\n self.frame_similarity_on_last_layer = frame_similarity_on_last_layer\n self.cls_layer1 = nn.Linear(1152 if self.frame_similarity_on_last_layer else D, 1,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:510-526"
+ },
+ "7103": {
+ "file_id": 516,
+ "content": "This code initializes a neural network model with a linear layer (`self.fc1`) that takes an input dimension of 512 if certain conditions are met, otherwise it takes the output_dim calculated earlier. The layer has D output dimensions and uses Xavier uniform initialization for weights and constant initialization for biases. Additionally, there's another linear layer (`self.cls_layer1`) with 1 output dimension that is initialized with Xavier uniform initialization for weights and a constant value of 0 for biases. It takes an input dimension of either 1152 or D based on whether frame similarity is added to the last layer or not.",
+ "type": "comment"
+ },
+ "7104": {
+ "file_id": 516,
+ "content": " )\n self.cls_layer2 = nn.Linear(1152 if self.frame_similarity_on_last_layer else D, 1,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n ) if use_many_hot_targets else None\n self.convex_comb_reg = ConvexCombinationRegularization(\n in_filters=(F * 2 ** (L - 1) * 4)) if use_convex_comb_reg else None\n def forward(self, inputs):\n assert list(inputs.shape[2:]) == [27, 48, 3] and inputs.dtype == paddle.float32, \\\n \"incorrect input type and/or shape\"\n out_dict = {}\n # shape [B, T, H, W, 3] to shape [B, 3, T, H, W]\n x = inputs.transpose([0, 4, 1, 2, 3])\n if self.use_resnet_features:\n x = self.resnet_layers(x)\n else:\n x = x / 255.\n inputs = inputs.clip(min=0).astype('uint8')\n if self.resnet_like_top:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:527-548"
+ },
+ "7105": {
+ "file_id": 516,
+ "content": "The code defines a model with two layers, a Linear layer and ConvexCombinationRegularization, depending on the use_many_hot_targets and use_convex_comb_reg parameters. The Linear layer has 1 output for each frame, unless frame_similarity_on_last_layer is set, in which case it has D outputs. If use_many_hot_targets is False, the layer is None. The forward function receives inputs of shape [B, T, H, W, 3] and performs transpose, resnet_features processing (if use_resnet_features=True), and normalization to apply the model layers. It also clips the input values between 0 and 255 before applying the regularization if use_convex_comb_reg is True.",
+ "type": "comment"
+ },
+ "7106": {
+ "file_id": 516,
+ "content": " x = self.resnet_like_top_conv(x)\n x = self.resnet_like_top_bn(x)\n x = self.resnet_like_top_max_pool(x)\n block_features = []\n for block in self.SDDCNN:\n x = block(x)\n block_features.append(x)\n if self.convex_comb_reg is not None:\n out_dict[\"alphas\"], out_dict[\"comb_reg_loss\"] = self.convex_comb_reg(inputs.transpose([0, 4, 1, 2, 3]), x)\n if self.use_mean_pooling:\n x = paddle.mean(x, axis=[3, 4])\n x = x.transpose([0, 2, 1])\n else:\n x = x.transpose([0, 2, 3, 4, 1])\n x = x.reshape([x.shape[0], x.shape[1], x.shape[2]*x.shape[3]*x.shape[4]])\n if self.frame_sim_layer is not None:\n x = paddle.concat([self.frame_sim_layer(block_features), x], 2)\n if self.color_hist_layer is not None:\n x = paddle.concat([self.color_hist_layer(inputs), x], 2)\n x = self.fc1(x)\n x = functional.relu(x)\n if self.dropout is not None:\n x = self.dropout(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:549-571"
+ },
+ "7107": {
+ "file_id": 516,
+ "content": "This code performs feature extraction and pooling operations for a ConvNextV2 backbone model. It applies residual blocks, top convolutions, batch normalization, and max pooling to the input. Then it calculates convex combination regression if required. The code either applies mean pooling or 3D reshaping based on the use_mean_pooling flag. Finally, it concatenates frame similarity layer outputs and color histogram layer outputs before performing fully connected layer calculations and applying relu activation and dropout if necessary.",
+ "type": "comment"
+ },
+ "7108": {
+ "file_id": 516,
+ "content": " if self.frame_sim_layer is not None and self.frame_similarity_on_last_layer:\n x = paddle.concat([self.frame_sim_layer(block_features), x], 2)\n one_hot = self.cls_layer1(x)\n if self.cls_layer2 is not None:\n out_dict[\"many_hot\"] = self.cls_layer2(x)\n if len(out_dict) > 0:\n return one_hot, out_dict\n return one_hot",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/transnetv2.py:572-581"
+ },
+ "7109": {
+ "file_id": 516,
+ "content": "This code checks if the frame similarity layer and classifier layers are not None, then performs a concatenation operation on block features and x. It applies the classifier layer to the resulting output and optionally applies another classifier layer. The function returns one_hot and an optional out_dict if they exist.",
+ "type": "comment"
+ },
+ "7110": {
+ "file_id": 517,
+ "content": "/paddlevideo/modeling/backbones/vit.py",
+ "type": "filepath"
+ },
+ "7111": {
+ "file_id": 517,
+ "content": "The PaddleVideo code offers video processing functions, including a VisionTransformer class. It initializes and applies the model using parameters, transformations, and blocks while setting up components for future use.",
+ "type": "summary"
+ },
+ "7112": {
+ "file_id": 517,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections.abc import Callable\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import Constant\nfrom ...utils import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import trunc_normal_\n__all__ = ['VisionTransformer']\nzeros_ = Constant(value=0.)\nones_ = Constant(value=1.)\ndef to_2tuple(x):\n return tuple([x] * 2)\ndef drop_path(x, drop_prob=0., training=False):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:1-37"
+ },
+ "7113": {
+ "file_id": 517,
+ "content": "This code snippet is from the PaddleVideo library and contains a copyright notice, license information, and several helper functions. The VisionTransformer class will be defined later in the file, which serves as a backbone model for video processing tasks. The code defines constants for zero and one values, a function to convert a single value into a tuple of length 2 (to_2tuple), and a drop path function that applies dropout to inputs with a specified probability during training.",
+ "type": "comment"
+ },
+ "7114": {
+ "file_id": 517,
+ "content": " \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...\n # issuecomment-532968956 ...\n See discussion: https://github.com/tensorflow/tpu/issues/494\n \"\"\"\n if drop_prob == 0. or not training:\n return x\n keep_prob = paddle.to_tensor(1 - drop_prob, dtype=x.dtype)\n shape = (paddle.shape(x)[0], ) + (1, ) * (x.ndim - 1)\n random_tensor = keep_prob + paddle.rand(shape).astype(x.dtype)\n random_tensor = paddle.floor(random_tensor) # binarize\n output = x.divide(keep_prob) * random_tensor\n return output\nclass DropPath(nn.Layer):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n \"\"\"\n def __init__(self, drop_prob=None):\n super(DropPath, self).__init__()\n self.drop_prob = drop_prob\n def forward(self, x):\n return drop_path(x, self.drop_prob, self.training)\nclass Identity(nn.Layer):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:38-65"
+ },
+ "7115": {
+ "file_id": 517,
+ "content": "This code defines three classes: \"DropPath\", \"Identity\". The DropPath class implements dropout paths (Stochastic Depth) for each sample in the main path of residual blocks. It takes a single parameter, 'drop_prob', to control the probability of dropping out features. If 'drop_prob' is 0 or not training, it returns the input unchanged. The Identity class simply returns its input without any transformation.",
+ "type": "comment"
+ },
+ "7116": {
+ "file_id": 517,
+ "content": " def __init__(self):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\nclass Mlp(nn.Layer):\n def __init__(self,\n in_features,\n hidden_features=None,\n out_features=None,\n act_layer=nn.GELU,\n drop=0.0):\n super().__init__()\n out_features = out_features or in_features\n hidden_features = hidden_features or in_features\n self.fc1 = nn.Linear(in_features, hidden_features)\n self.act = act_layer()\n self.fc2 = nn.Linear(hidden_features, out_features)\n self.drop = nn.Dropout(drop)\n def forward(self, x):\n x = self.fc1(x)\n x = self.act(x)\n x = self.drop(x)\n x = self.fc2(x)\n x = self.drop(x)\n return x\nclass Attention(nn.Layer):\n def __init__(self,\n dim,\n num_heads=8,\n qkv_bias=False,\n qk_scale=None,\n attn_drop=0.0,\n proj_drop=0.0):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:66-104"
+ },
+ "7117": {
+ "file_id": 517,
+ "content": "The code defines three classes: Identity, Mlp, and Attention. Identity is a simple class that returns its input unchanged. Mlp stands for Multilayer Perceptron, and it's a feed-forward neural network layer. Attention is a class for implementing attention mechanisms in the model. Both Mlp and Attention classes take inputs and return outputs after applying their respective operations.",
+ "type": "comment"
+ },
+ "7118": {
+ "file_id": 517,
+ "content": " super().__init__()\n self.num_heads = num_heads\n head_dim = dim // num_heads\n self.scale = qk_scale or head_dim**-0.5\n self.qkv = nn.Linear(dim, dim * 3, bias_attr=qkv_bias)\n self.proj = nn.Linear(dim, dim)\n self.proj_drop = nn.Dropout(proj_drop)\n self.attn_drop = nn.Dropout(attn_drop)\n def forward(self, x):\n N, C = x.shape[1:]\n qkv = self.qkv(x).reshape(\n (-1, N, 3, self.num_heads, C // self.num_heads)).transpose(\n (2, 0, 3, 1, 4))\n q, k, v = qkv[0], qkv[1], qkv[2]\n attn = (q.matmul(k.transpose((0, 1, 3, 2)))) * self.scale\n attn = nn.functional.softmax(attn, axis=-1)\n attn = self.attn_drop(attn)\n x = (attn.matmul(v)).transpose((0, 2, 1, 3)).reshape((-1, N, C))\n x = self.proj(x)\n x = self.proj_drop(x)\n return x\nclass Block(nn.Layer):\n def __init__(self,\n dim,\n num_heads,\n mlp_ratio=4.0,\n qkv_bias=False,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:105-138"
+ },
+ "7119": {
+ "file_id": 517,
+ "content": "This code initializes a multi-head attention layer, and defines the forward pass. It reshapes input into query (Q), key (K), and value (V) matrices, calculates attention scores, applies dropout, and reconstructs output using residual connections and layer normalization. The `Block` class is also defined for building a Vision Transformer model.",
+ "type": "comment"
+ },
+ "7120": {
+ "file_id": 517,
+ "content": " qk_scale=None,\n drop=0.0,\n attn_drop=0.0,\n drop_path=0.1,\n act_layer=nn.GELU,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n attention_type='divided_space_time'):\n super().__init__()\n self.attention_type = attention_type\n if isinstance(norm_layer, str):\n self.norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop)\n # Temporal Attention Parameters\n if self.attention_type == 'divided_space_time':",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:139-166"
+ },
+ "7121": {
+ "file_id": 517,
+ "content": "This function is initializing a backbone model with specified parameters. It takes in arguments like attention_type, norm_layer, and others to define the model's layers, including its attention layer. If norm_layer is a string, it uses the given string as the normalization layer; if it's a Callable, it uses that function as the normalization layer. The code also checks if the attention type is 'divided_space_time'.",
+ "type": "comment"
+ },
+ "7122": {
+ "file_id": 517,
+ "content": " if isinstance(norm_layer, str):\n self.temporal_norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.temporal_norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.temporal_attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop)\n self.temporal_fc = nn.Linear(dim, dim)\n # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here\n self.drop_path = DropPath(drop_path) if drop_path > 0. else Identity()\n if isinstance(norm_layer, str):\n self.norm2 = eval(norm_layer)(dim, epsilon=epsilon)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:167-185"
+ },
+ "7123": {
+ "file_id": 517,
+ "content": "This code initializes the temporal normalization layer and attention mechanism for a Vision Transformer backbone. It also creates a linear layer and drop path, based on provided configurations. The norm_layer parameter can be a string representing the desired normalization layer or a Callable object. If not a valid type, it raises a TypeError.",
+ "type": "comment"
+ },
+ "7124": {
+ "file_id": 517,
+ "content": " elif isinstance(norm_layer, Callable):\n self.norm2 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = Mlp(in_features=dim,\n hidden_features=mlp_hidden_dim,\n act_layer=act_layer,\n drop=drop)\n def forward(self, x, B, T, W):\n num_spatial_tokens = (x.shape[1] - 1) // T\n H = num_spatial_tokens // W\n if self.attention_type in ['space_only', 'joint_space_time']:\n x = x + self.drop_path(self.attn(self.norm1(x)))\n x = x + self.drop_path(self.mlp(self.norm2(x)))\n return x\n elif self.attention_type == 'divided_space_time':\n ########## Temporal ##########\n xt = x[:, 1:, :]\n _, _, _, _t, _m = B, H, W, T, xt.shape[-1]\n xt = xt.reshape([-1, _t, _m])\n res_temporal = self.drop_path(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:186-210"
+ },
+ "7125": {
+ "file_id": 517,
+ "content": "The code defines a class and its forward method. It sets the normalization layer, calculates the number of spatial tokens, checks the attention type, applies normalization and MLP layers to the input, and performs divided space-time attention.",
+ "type": "comment"
+ },
+ "7126": {
+ "file_id": 517,
+ "content": " self.temporal_attn(self.temporal_norm1(xt)))\n _, _h, _w, _t, _m = B, H, W, T, res_temporal.shape[-1]\n res_temporal = res_temporal.reshape([-1, _h * _w * _t, _m])\n res_temporal = self.temporal_fc(res_temporal)\n xt = x[:, 1:, :] + res_temporal\n ########## Spatial ##########\n init_cls_token = x[:, 0, :].unsqueeze(1)\n cls_token = init_cls_token.tile((1, T, 1))\n _b, _t, _m = cls_token.shape\n cls_token = cls_token.reshape([-1, _m]).unsqueeze(1)\n xs = xt\n _, _h, _w, _t, _m = B, H, W, T, xs.shape[-1]\n xs = xs.reshape([-1, _h, _w, _t, _m]).transpose(\n (0, 3, 1, 2, 4)).reshape([-1, _h * _w, _m])\n xs = paddle.concat((cls_token, xs), axis=1)\n res_spatial = self.drop_path(self.attn(self.norm1(xs)))\n # Taking care of CLS token\n cls_token = res_spatial[:, 0, :]\n _, _t, _m = B, T, cls_token.shape[-1]\n cls_token = cls_token.reshape([-1, _t, _m])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:211-235"
+ },
+ "7127": {
+ "file_id": 517,
+ "content": "This code performs spatial attention in the Vision Transformer model. It creates a cls_token, reshapes the input, concatenates it with the cls_token, and then passes it through a drop path and an attention layer. Finally, it extracts the cls_token for further use.",
+ "type": "comment"
+ },
+ "7128": {
+ "file_id": 517,
+ "content": " # averaging for every frame\n cls_token = paddle.mean(cls_token, axis=1, keepdim=True)\n res_spatial = res_spatial[:, 1:, :]\n _, _t, _h, _w, _m = B, T, H, W, res_spatial.shape[-1]\n res_spatial = res_spatial.reshape([-1, _t, _h, _w, _m]).transpose(\n (0, 2, 3, 1, 4)).reshape([-1, _h * _w * _t, _m])\n res = res_spatial\n x = xt\n x = paddle.concat((init_cls_token, x), axis=1) + paddle.concat(\n (cls_token, res), axis=1)\n # Mlp\n x = x + self.drop_path(self.mlp(self.norm2(x)))\n return x\n else:\n raise NotImplementedError\nclass PatchEmbed(nn.Layer):\n \"\"\" Image to Patch Embedding\n \"\"\"\n def __init__(self,\n img_size=224,\n patch_size=16,\n in_channels=3,\n embed_dim=768):\n super().__init__()\n img_size = to_2tuple(img_size)\n patch_size = to_2tuple(patch_size)\n num_patches = (img_size[1] // patch_size[1]) * (img_size[0] //",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:236-267"
+ },
+ "7129": {
+ "file_id": 517,
+ "content": "This code performs averaging across frames, reshapes the spatial features, concatenates initial class token and input sequence, adds a drop path and MLP layer, and returns the output. It also defines PatchEmbed for image to patch embedding.",
+ "type": "comment"
+ },
+ "7130": {
+ "file_id": 517,
+ "content": " patch_size[0])\n self.img_size = img_size\n self.patch_size = patch_size\n self.num_patches = num_patches\n self.proj = nn.Conv2D(in_channels,\n embed_dim,\n kernel_size=patch_size,\n stride=patch_size)\n def forward(self, x):\n B, C, T, H, W = x.shape\n assert H == self.img_size[0] and W == self.img_size[1], \\\n f\"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]}).\"\n x = x.transpose((0, 2, 1, 3, 4))\n x = x.reshape([-1, C, H, W])\n x = self.proj(x)\n W = x.shape[-1]\n x = x.flatten(2).transpose((0, 2, 1))\n return x, T, W\n@BACKBONES.register()\nclass VisionTransformer(nn.Layer):\n \"\"\" Vision Transformer with support for patch input\n \"\"\"\n def __init__(self,\n pretrained=None,\n img_size=224,\n patch_size=16,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:268-298"
+ },
+ "7131": {
+ "file_id": 517,
+ "content": "The code defines a VisionTransformer class that takes input patches of an image. It initializes the model parameters such as img_size, patch_size and num_patches. The forward function performs the transformation by projecting the input into embedding space using a convolutional layer. If the input image size does not match the expected model size, it raises an assertion error. This class is registered with BACKBONES for future use.",
+ "type": "comment"
+ },
+ "7132": {
+ "file_id": 517,
+ "content": " in_channels=3,\n embed_dim=768,\n depth=12,\n num_heads=12,\n mlp_ratio=4,\n qkv_bias=False,\n qk_scale=None,\n drop_rate=0.,\n attn_drop_rate=0.,\n drop_path_rate=0.1,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n num_seg=8,\n attention_type='divided_space_time',\n **args):\n super().__init__()\n self.pretrained = pretrained\n self.num_seg = num_seg\n self.attention_type = attention_type\n self.num_features = self.embed_dim = embed_dim\n self.patch_embed = PatchEmbed(img_size=img_size,\n patch_size=patch_size,\n in_channels=in_channels,\n embed_dim=embed_dim)\n num_patches = self.patch_embed.num_patches\n # Positional Embeddings\n self.cls_token = self.create_parameter(shape=(1, 1, embed_dim),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:299-327"
+ },
+ "7133": {
+ "file_id": 517,
+ "content": "This code initializes a Vision Transformer (ViT) backbone model with specified parameters such as input dimensions, embedding dimension, depth, number of heads, mlp ratio, and attention type. The code sets up the patch embedding layer, creates a class token, and defines the number of patches based on the input size provided.",
+ "type": "comment"
+ },
+ "7134": {
+ "file_id": 517,
+ "content": " default_initializer=zeros_)\n self.pos_embed = self.create_parameter(shape=(1, num_patches + 1,\n embed_dim),\n default_initializer=zeros_)\n self.pos_drop = nn.Dropout(p=drop_rate)\n if self.attention_type != 'space_only':\n self.time_embed = self.create_parameter(shape=(1, num_seg,\n embed_dim),\n default_initializer=zeros_)\n self.time_drop = nn.Dropout(p=drop_rate)\n self.add_parameter(\"pos_embed\", self.pos_embed)\n self.add_parameter(\"cls_token\", self.cls_token)\n dpr = np.linspace(0, drop_path_rate, depth)\n self.blocks = nn.LayerList([\n Block(dim=embed_dim,\n num_heads=num_heads,\n mlp_ratio=mlp_ratio,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:328-350"
+ },
+ "7135": {
+ "file_id": 517,
+ "content": "This code initializes various components of a vision transformer model, including positional embeddings (pos_embed), classification token (cls_token), and dropout layers (pos_drop, time_drop). It also creates a LayerList of blocks with specified dimensions and parameters.",
+ "type": "comment"
+ },
+ "7136": {
+ "file_id": 517,
+ "content": " drop=drop_rate,\n attn_drop=attn_drop_rate,\n drop_path=dpr[i],\n norm_layer=norm_layer,\n epsilon=epsilon,\n attention_type=self.attention_type) for i in range(depth)\n ])\n self.norm = eval(norm_layer)(embed_dim, epsilon=epsilon)\n def init_weights(self):\n \"\"\"First init model's weight\"\"\"\n trunc_normal_(self.pos_embed, std=0.02)\n trunc_normal_(self.cls_token, std=0.02)\n self.apply(self._init_fn)\n if self.attention_type == 'divided_space_time':\n i = 0\n for m in self.blocks.sublayers(include_self=True):\n m_str = str(m)\n if 'Block' in m_str:\n if i > 0:\n zeros_(m.temporal_fc.weight)\n zeros_(m.temporal_fc.bias)\n i += 1\n \"\"\"Second, if provide pretrained ckpt, load it\"\"\"\n if isinstance(\n self.pretrained, str\n ) and self.pretrained.strip() != \"\": # load pretrained weights",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:351-379"
+ },
+ "7137": {
+ "file_id": 517,
+ "content": "This code initializes a Vision Transformer (ViT) model. It creates a series of blocks with specified dimensions, applies normalization layers, and initializes the weight values using truncated normal distribution. Additionally, if pre-trained weights are provided, it loads them into the model.",
+ "type": "comment"
+ },
+ "7138": {
+ "file_id": 517,
+ "content": " load_ckpt(self,\n self.pretrained,\n num_patches=self.patch_embed.num_patches,\n num_seg=self.num_seg,\n attention_type=self.attention_type)\n def _init_fn(self, m):\n if isinstance(m, nn.Linear):\n trunc_normal_(m.weight)\n if m.bias is not None:\n zeros_(m.bias)\n elif isinstance(m, nn.LayerNorm):\n ones_(m.weight)\n zeros_(m.bias)\n def forward_features(self, x):\n # B = x.shape[0]\n B = paddle.shape(x)[0]\n x, T, W = self.patch_embed(x) # [BT,nH*nW,F]\n cls_tokens = self.cls_token.expand((B * T, -1, -1)) # [1,1,F]->[BT,1,F]\n x = paddle.concat((cls_tokens, x), axis=1)\n pos_interp = (x.shape[1] != self.pos_embed.shape[1])\n if pos_interp:\n pos_embed = self.pos_embed\n cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)\n other_pos_embed = pos_embed[0, 1:, :].unsqueeze(0).transpose(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:380-405"
+ },
+ "7139": {
+ "file_id": 517,
+ "content": "This code initializes the forward function of a Vision Transformer (ViT) model. It extracts features from input images, adds positional embeddings, and handles batch size changes. The trunc_normal_ and zeros_ functions are used to initialize weights and biases for layers like Linear and LayerNorm, respectively.",
+ "type": "comment"
+ },
+ "7140": {
+ "file_id": 517,
+ "content": " (0, 2, 1))\n P = int(other_pos_embed.shape[2]**0.5)\n H = x.shape[1] // W\n other_pos_embed = other_pos_embed.reshape([1, x.shape[2], P, P])\n new_pos_embed = F.interpolate(other_pos_embed,\n size=(H, W),\n mode='nearest')\n new_pos_embed = new_pos_embed.flatten(2)\n new_pos_embed = new_pos_embed.transpose((0, 2, 1))\n new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed),\n axis=1)\n x = x + new_pos_embed\n else:\n x = x + self.pos_embed\n x = self.pos_drop(x)\n # Time Embeddings\n if self.attention_type != 'space_only':\n cls_tokens = x[:B, 0, :].unsqueeze(1) if B > 0 else x.split(\n T)[0].index_select(paddle.to_tensor([0]), axis=1)\n x = x[:, 1:]\n _, _n, _m = x.shape\n _t = T\n x = x.reshape([-1, _t, _n, _m]).transpose(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:406-430"
+ },
+ "7141": {
+ "file_id": 517,
+ "content": "The code is applying relative position embeddings to the input features (x) for a vision transformer model. It first checks if a specific flag is set, then interpolates other position embeddings based on the size of the input and adds them to class position embeddings. If the flag is not set, it simply adds the position embeddings from the model. Afterward, the code applies time embeddings if the attention type is not \"space_only\".",
+ "type": "comment"
+ },
+ "7142": {
+ "file_id": 517,
+ "content": " (0, 2, 1, 3)).reshape([-1, _t, _m])\n # Resizing time embeddings in case they don't match\n time_interp = (T != self.time_embed.shape[1])\n if time_interp: # T' != T\n time_embed = self.time_embed.transpose((0, 2, 1)).unsqueeze(0)\n new_time_embed = F.interpolate(time_embed,\n size=(T, x.shape[-1]),\n mode='nearest').squeeze(0)\n new_time_embed = new_time_embed.transpose((0, 2, 1))\n x = x + new_time_embed\n else:\n x = x + self.time_embed\n x = self.time_drop(x)\n _, _t, _m = x.shape\n x = x.reshape([-1, W * W * T, _m])\n x = paddle.concat((cls_tokens, x), axis=1)\n # Attention blocks\n for blk in self.blocks:\n x = blk(x, B, T, W)\n # Predictions for space-only baseline\n if self.attention_type == 'space_only':\n _, _n, _m = x.shape",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:431-455"
+ },
+ "7143": {
+ "file_id": 517,
+ "content": "This code performs time embeddings resizing and adds them to the input feature maps. It then flattens the tensor, concatenates class tokens, processes through attention blocks, and finally, for space-only attention type, it makes predictions.",
+ "type": "comment"
+ },
+ "7144": {
+ "file_id": 517,
+ "content": " _t = T\n x = x.reshape([-1, _t, _n, _m])\n x = paddle.mean(x, 1) # averaging predictions for every frame\n x = self.norm(x)\n return x[:, 0] # [B, embed_dim]\n def forward(self, x):\n x = self.forward_features(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit.py:456-465"
+ },
+ "7145": {
+ "file_id": 517,
+ "content": "This code snippet is part of a Vision Transformer (ViT) model implementation. The function averages predictions for every frame and applies normalization before returning the embeddings for each image in the input sequence.",
+ "type": "comment"
+ },
+ "7146": {
+ "file_id": 518,
+ "content": "/paddlevideo/modeling/backbones/vit_tweaks.py",
+ "type": "filepath"
+ },
+ "7147": {
+ "file_id": 518,
+ "content": "The PaddleVideo library's backbones code introduces the VisionTransformer_tweaks model with weight initialization, stochastic depth, spatial attention in ViT models, and transformer configurations. It is a time-based feature modification model that computes space-only predictions through attention blocks.",
+ "type": "summary"
+ },
+ "7148": {
+ "file_id": 518,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections.abc import Callable\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle import ParamAttr\nfrom paddle.nn.initializer import Constant\nfrom paddle.regularizer import L2Decay\nfrom ...utils import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import trunc_normal_\n__all__ = ['VisionTransformer_tweaks']\nzeros_ = Constant(value=0.)\nones_ = Constant(value=1.)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:1-32"
+ },
+ "7149": {
+ "file_id": 518,
+ "content": "This code is from the PaddleVideo library's backbones module and defines the VisionTransformer_tweaks model. It imports necessary libraries, sets constant values, and includes function definitions for weight initialization and regularizers. The BACKBONES registry is also defined to categorize the model type.",
+ "type": "comment"
+ },
+ "7150": {
+ "file_id": 518,
+ "content": "def to_2tuple(x):\n return tuple([x] * 2)\ndef rand_bbox(size, lam):\n \"\"\" rand_bbox \"\"\"\n w = size[2]\n h = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(w * cut_rat)\n cut_h = np.int(h * cut_rat)\n # uniform\n cx = np.random.randint(w)\n cy = np.random.randint(h)\n bbx1 = np.clip(cx - cut_w // 2, 0, w)\n bby1 = np.clip(cy - cut_h // 2, 0, h)\n bbx2 = np.clip(cx + cut_w // 2, 0, w)\n bby2 = np.clip(cy + cut_h // 2, 0, h)\n return bbx1, bby1, bbx2, bby2\ndef drop_path(x, drop_prob=0., training=False):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...\n # issuecomment-532968956 ...\n See discussion: https://github.com/tensorflow/tpu/issues/494\n \"\"\"\n if drop_prob == 0. or not training:\n return x\n keep_prob = paddle.to_tensor(1 - drop_prob)\n shape = (paddle.shape(x)[0], ) + (1, ) * (x.ndim - 1)\n random_tensor = keep_prob + paddle.rand(shape, dtype=x.dtype)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:35-69"
+ },
+ "7151": {
+ "file_id": 518,
+ "content": "The code defines three functions. The \"to_2tuple\" function takes an input and returns a tuple with the same value repeated twice. The \"rand_bbox\" function generates random bounding box coordinates within the size of an image, given a specified probability. The \"drop_path\" function applies stochastic depth (dropout) to each sample in the main path of residual blocks with a specified dropout rate.",
+ "type": "comment"
+ },
+ "7152": {
+ "file_id": 518,
+ "content": " random_tensor = paddle.floor(random_tensor) # binarize\n output = x.divide(keep_prob) * random_tensor\n return output\nclass DropPath(nn.Layer):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n \"\"\"\n def __init__(self, drop_prob=None):\n super(DropPath, self).__init__()\n self.drop_prob = drop_prob\n def forward(self, x):\n return drop_path(x, self.drop_prob, self.training)\nclass Identity(nn.Layer):\n def __init__(self):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\nclass Mlp(nn.Layer):\n def __init__(self,\n in_features,\n hidden_features=None,\n out_features=None,\n act_layer=nn.GELU,\n drop=0.,\n wd_bias=True,\n lr_mult=1.0):\n super().__init__()\n out_features = out_features or in_features\n hidden_features = hidden_features or in_features\n self.fc1 = nn.Linear(in_features, hidden_features)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:70-107"
+ },
+ "7153": {
+ "file_id": 518,
+ "content": "This code defines a class called `Mlp` which is a fully connected layer with a middle layer and an output layer. It also includes an activation function (GELU by default) and a dropout layer (with drop probability specified). The class `DropPath` applies drop paths to stochastically mask layers during training, while the `Identity` class simply returns its input unchanged.",
+ "type": "comment"
+ },
+ "7154": {
+ "file_id": 518,
+ "content": " self.act = act_layer()\n self.fc2 = nn.Linear(hidden_features, out_features)\n self.drop = nn.Dropout(drop)\n def forward(self, x):\n x = self.fc1(x)\n x = self.act(x)\n x = self.drop(x)\n x = self.fc2(x)\n x = self.drop(x)\n return x\nclass Attention(nn.Layer):\n def __init__(self,\n dim,\n num_heads=8,\n qkv_bias=False,\n qk_scale=None,\n attn_drop=0.,\n proj_drop=0.,\n wd_bias=True,\n lr_mult=1.0):\n super().__init__()\n self.num_heads = num_heads\n head_dim = dim // num_heads\n self.scale = qk_scale or head_dim**-0.5\n self.qkv = nn.Linear(dim, dim * 3, bias_attr=qkv_bias)\n self.proj = nn.Linear(dim, dim)\n self.proj_drop = nn.Dropout(proj_drop)\n self.attn_drop = nn.Dropout(attn_drop)\n def forward(self, x):\n N, C = x.shape[1:]\n qkv = self.qkv(x).reshape(\n (-1, N, 3, self.num_heads, C // self.num_heads)).transpose(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:108-144"
+ },
+ "7155": {
+ "file_id": 518,
+ "content": "The code defines a neural network layer called \"Attention\" with several components including a Linear layer for the query-key-value (QKV) transform, and separate Dropout layers for the attention and projection operations. The forward function performs the multi-head self-attention operation on the input tensor x, reshaping it to apply the QKV transform, and then applying dropout for both the attention and projection steps before returning the result. This layer is commonly used in transformer models for processing sequential data.",
+ "type": "comment"
+ },
+ "7156": {
+ "file_id": 518,
+ "content": " (2, 0, 3, 1, 4))\n q, k, v = qkv[0], qkv[1], qkv[2]\n attn = (q.matmul(k.transpose((0, 1, 3, 2)))) * self.scale\n attn = nn.functional.softmax(attn, axis=-1)\n attn = self.attn_drop(attn)\n x = (attn.matmul(v)).transpose((0, 2, 1, 3)).reshape((-1, N, C))\n x = self.proj(x)\n x = self.proj_drop(x)\n return x\nclass Block(nn.Layer):\n def __init__(self,\n dim,\n num_heads,\n mlp_ratio=4.0,\n qkv_bias=False,\n qk_scale=None,\n drop=0.0,\n attn_drop=0.0,\n drop_path=0.1,\n act_layer=nn.GELU,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n attention_type='divided_space_time',\n wd_bias=True,\n lr_mult=1.0):\n super().__init__()\n self.attention_type = attention_type\n if isinstance(norm_layer, str):\n self.norm1 = eval(norm_layer)(dim, epsilon=epsilon)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:145-178"
+ },
+ "7157": {
+ "file_id": 518,
+ "content": "This code defines a `Block` class that implements an attention mechanism using query-key value (QKV) decomposition. The block also includes a multi-layer perceptron (MLP) layer and supports different attention types. The input dimensions, number of heads in the attention mechanism, and other parameters are passed to the constructor.",
+ "type": "comment"
+ },
+ "7158": {
+ "file_id": 518,
+ "content": " elif isinstance(norm_layer, Callable):\n self.norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop,\n wd_bias=wd_bias,\n lr_mult=lr_mult)\n # Temporal Attention Parameters\n if self.attention_type == 'divided_space_time':\n if isinstance(norm_layer, str):\n self.temporal_norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.temporal_norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:179-202"
+ },
+ "7159": {
+ "file_id": 518,
+ "content": "This code checks the type of norm_layer and creates an instance of either a str or a paddle.nn.layer.Layer class for self.norm1. If no temporal attention is required, it raises a TypeError if norm_layer is neither a str nor a Callable. If divided space time attention is selected, it checks the type of norm_layer again and creates an instance of either a str or a paddle.nn.layer.Layer class for self.temporal_norm1.",
+ "type": "comment"
+ },
+ "7160": {
+ "file_id": 518,
+ "content": " self.temporal_attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop,\n wd_bias=wd_bias,\n lr_mult=lr_mult)\n self.temporal_fc = nn.Linear(dim, dim)\n # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here\n self.drop_path = DropPath(drop_path) if drop_path > 0. else Identity()\n if isinstance(norm_layer, str):\n self.norm2 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.norm2 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:203-221"
+ },
+ "7161": {
+ "file_id": 518,
+ "content": "This code initializes the temporal attention module, a linear layer for temporal features, and a drop path for stochastic depth. It also handles norm_layer initialization according to its type.",
+ "type": "comment"
+ },
+ "7162": {
+ "file_id": 518,
+ "content": " mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = Mlp(in_features=dim,\n hidden_features=mlp_hidden_dim,\n act_layer=act_layer,\n drop=drop,\n wd_bias=wd_bias,\n lr_mult=lr_mult)\n def forward(self, x, B, T, W):\n num_spatial_tokens = (x.shape[1] - 1) // T\n H = num_spatial_tokens // W\n if self.attention_type in ['space_only', 'joint_space_time']:\n x = paddle.add(x, self.drop_path(self.attn(self.norm1(x))))\n x = paddle.add(x, self.drop_path(self.mlp(self.norm2(x))))\n return x\n elif self.attention_type == 'divided_space_time':\n ########## Temporal ##########\n xt = x[:, 1:, :]\n _, _, _, _t, _m = B, H, W, T, xt.shape[-1]\n xt = xt.reshape([-1, _t, _m])\n res_temporal = self.drop_path(\n self.temporal_attn(self.temporal_norm1(xt)))\n _, _h, _w, _t, _m = B, H, W, T, res_temporal.shape[-1]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:223-247"
+ },
+ "7163": {
+ "file_id": 518,
+ "content": "Code defines a backbone for Vision Transformer (ViT) with tweaks and handles the forward pass. The MLP layer is added, and attention type can be space-only, joint space-time or divided space-time. In divided space-time, it also includes temporal attention.",
+ "type": "comment"
+ },
+ "7164": {
+ "file_id": 518,
+ "content": " res_temporal = res_temporal.reshape([-1, _h * _w * _t, _m])\n res_temporal = self.temporal_fc(res_temporal)\n xt = paddle.add(x[:, 1:, :], res_temporal)\n ########## Spatial ##########\n init_cls_token = x[:, 0, :].unsqueeze(1)\n cls_token = init_cls_token.tile((1, T, 1))\n _b, _t, _m = cls_token.shape\n cls_token = cls_token.reshape([-1, _m]).unsqueeze(1)\n xs = xt\n _, _h, _w, _t, _m = B, H, W, T, xs.shape[-1]\n xs = xs.reshape([-1, _h, _w, _t, _m]).transpose(\n (0, 3, 1, 2, 4)).reshape([-1, _h * _w, _m])\n xs = paddle.concat((cls_token, xs), axis=1)\n res_spatial = self.drop_path(self.attn(self.norm1(xs)))\n # Taking care of CLS token\n cls_token = res_spatial[:, 0, :]\n _, _t, _m = B, T, cls_token.shape[-1]\n cls_token = cls_token.reshape([-1, _t, _m])\n # averaging for every frame\n cls_token = paddle.mean(cls_token, axis=1, keepdim=True)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:248-271"
+ },
+ "7165": {
+ "file_id": 518,
+ "content": "This code performs spatial attention in a ViT model. It reshapes the input, concatenates the class token with the reshaped input, applies normalization and self-attention, and finally averages the class tokens for each frame to obtain a contextual representation.",
+ "type": "comment"
+ },
+ "7166": {
+ "file_id": 518,
+ "content": " res_spatial = res_spatial[:, 1:, :]\n _, _t, _h, _w, _m = B, T, H, W, res_spatial.shape[-1]\n res_spatial = res_spatial.reshape([-1, _t, _h, _w, _m]).transpose(\n (0, 2, 3, 1, 4)).reshape([-1, _h * _w * _t, _m])\n res = res_spatial\n x = xt\n x = paddle.add(paddle.concat((init_cls_token, x), axis=1),\n paddle.concat((cls_token, res), axis=1))\n # Mlp\n x = paddle.add(x, self.drop_path(self.mlp(self.norm2(x))))\n return x\n else:\n raise NotImplementedError\nclass PatchEmbed(nn.Layer):\n \"\"\" Image to Patch Embedding\n \"\"\"\n def __init__(self,\n img_size=224,\n patch_size=16,\n in_channels=3,\n embed_dim=768,\n wd_bias=True,\n lr_mult=1.0):\n super().__init__()\n img_size = to_2tuple(img_size)\n patch_size = to_2tuple(patch_size)\n num_patches = (img_size[1] // patch_size[1]) * (img_size[0] //",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:273-302"
+ },
+ "7167": {
+ "file_id": 518,
+ "content": "This code is from the PaddleVideo library and defines a PatchEmbed class for image to patch embedding. It takes in parameters such as img_size, patch_size, in_channels, embed_dim, wd_bias, and lr_mult. The class performs image to patch embedding by dividing the input image into patches of specified size and flattening them into a 2D feature map. The code also includes a NotImplementedError for certain conditions, suggesting that some parts may not be fully implemented yet.",
+ "type": "comment"
+ },
+ "7168": {
+ "file_id": 518,
+ "content": " patch_size[0])\n self.img_size = img_size\n self.patch_size = patch_size\n self.num_patches = num_patches\n self.proj = nn.Conv2D(in_channels,\n embed_dim,\n kernel_size=patch_size,\n stride=patch_size)\n def forward(self, x):\n B, C, T, H, W = x.shape\n assert H == self.img_size[0] and W == self.img_size[1], \\\n f\"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]}).\"\n x = x.transpose((0, 2, 1, 3, 4)) # [B,T,C,H,W]\n x = x.reshape([-1, C, H, W]) # [BT,C,H,W]\n x = self.proj(x) # [BT,F,nH,nW]\n W = x.shape[-1]\n x = x.flatten(2).transpose((0, 2, 1)) # [BT,F,nHnW]\n return x, T, W\n@BACKBONES.register()\nclass VisionTransformer_tweaks(nn.Layer):\n \"\"\" Vision Transformer with support for patch input\n \"\"\"\n def __init__(self,\n pretrained=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:303-331"
+ },
+ "7169": {
+ "file_id": 518,
+ "content": "This code defines a VisionTransformer with patch input. The model takes an image of size img_size and divides it into patches of size patch_size, extracting features from each patch using the Conv2D layer. The forward method reshapes the input and passes it through the projection convolution. It then flattens the output and returns the result along with the number of patches (T) and the total number of image pixels (W).",
+ "type": "comment"
+ },
+ "7170": {
+ "file_id": 518,
+ "content": " img_size=224,\n patch_size=16,\n in_channels=3,\n embed_dim=768,\n depth=12,\n num_heads=12,\n mlp_ratio=4,\n qkv_bias=False,\n qk_scale=None,\n drop_rate=0.,\n attn_drop_rate=0.,\n drop_path_rate=0.1,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n num_seg=8,\n attention_type='divided_space_time',\n wd_bias=True,\n lr_mult_list=[1.0, 1.0, 1.0, 1.0, 1.0],\n **args):\n super().__init__()\n self.pretrained = pretrained\n self.num_seg = num_seg\n self.attention_type = attention_type\n self.lr_mult_list = lr_mult_list\n self.num_features = self.embed_dim = embed_dim\n self.patch_embed = PatchEmbed(img_size=img_size,\n patch_size=patch_size,\n in_channels=in_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:332-360"
+ },
+ "7171": {
+ "file_id": 518,
+ "content": "This code initializes a ViT (Vision Transformer) model with specified dimensions and parameters. It uses the PatchEmbed class to embed input images, sets the number of segments for attention, and defines the learning rate multipliers for each stage of the model. It also specifies whether to use pre-trained weights.",
+ "type": "comment"
+ },
+ "7172": {
+ "file_id": 518,
+ "content": " embed_dim=embed_dim,\n wd_bias=wd_bias,\n lr_mult=self.lr_mult_list[0])\n num_patches = self.patch_embed.num_patches\n # Positional Embeddings\n self.cls_token = self.create_parameter(\n shape=(1, 1, embed_dim),\n default_initializer=zeros_,\n attr=ParamAttr(regularizer=L2Decay(0.0)))\n self.pos_embed = self.create_parameter(\n shape=(1, num_patches + 1, embed_dim),\n default_initializer=zeros_,\n attr=ParamAttr(regularizer=L2Decay(0.0)))\n self.pos_drop = nn.Dropout(p=drop_rate)\n if self.attention_type != 'space_only':\n self.time_embed = self.create_parameter(\n shape=(1, num_seg, embed_dim),\n default_initializer=zeros_,\n attr=ParamAttr(regularizer=L2Decay(0.0)))\n self.time_drop = nn.Dropout(p=drop_rate)\n self.add_parameter(\"pos_embed\", self.pos_embed)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:361-384"
+ },
+ "7173": {
+ "file_id": 518,
+ "content": "This code initializes the positional and time embeddings for a transformer model. It creates a cls_token, pos_embed and optionally time_embed with specified dimensions and regularizers. It also adds dropout layers for positional and temporal features, if needed.",
+ "type": "comment"
+ },
+ "7174": {
+ "file_id": 518,
+ "content": " self.add_parameter(\"cls_token\", self.cls_token)\n dpr = np.linspace(0, drop_path_rate, depth)\n self.blocks = nn.LayerList([\n Block(dim=embed_dim,\n num_heads=num_heads,\n mlp_ratio=mlp_ratio,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n drop=drop_rate,\n attn_drop=attn_drop_rate,\n drop_path=dpr[i],\n norm_layer=norm_layer,\n epsilon=epsilon,\n attention_type=self.attention_type,\n wd_bias=wd_bias,\n lr_mult=self.lr_mult_list[(i // 4) + 1]) for i in range(depth)\n ])\n self.norm = eval(norm_layer)(embed_dim, epsilon=epsilon)\n def init_weights(self):\n \"\"\"First init model's weight\"\"\"\n trunc_normal_(self.pos_embed, std=0.02)\n trunc_normal_(self.cls_token, std=0.02)\n self.apply(self._init_fn)\n if self.attention_type == 'divided_space_time':\n i = 0",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:385-414"
+ },
+ "7175": {
+ "file_id": 518,
+ "content": "The code initializes a transformer model with blocks, adds parameters for position and classification tokens, creates a layer list of blocks with varying drop paths and attention types, and applies weight initialization to the positional embeddings, classification token, and layers.",
+ "type": "comment"
+ },
+ "7176": {
+ "file_id": 518,
+ "content": " for m in self.blocks.sublayers(include_self=True):\n m_str = str(m)\n if 'Block' in m_str:\n if i > 0:\n zeros_(m.temporal_fc.weight)\n zeros_(m.temporal_fc.bias)\n i += 1\n \"\"\"Second, if provide pretrained ckpt, load it\"\"\"\n if isinstance(\n self.pretrained, str\n ) and self.pretrained.strip() != \"\": # load pretrained weights\n load_ckpt(self,\n self.pretrained,\n num_patches=self.patch_embed.num_patches,\n num_seg=self.num_seg,\n attention_type=self.attention_type)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n pass\n else:\n raise NotImplementedError\n def _init_fn(self, m):\n if isinstance(m, nn.Linear):\n trunc_normal_(m.weight)\n if m.bias is not None:\n zeros_(m.bias)\n elif isinstance(m, nn.LayerNorm):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:415-441"
+ },
+ "7177": {
+ "file_id": 518,
+ "content": "Initializing the backbone network by iterating through each sublayer, setting temporal_fc weight and bias to zeros if it's a Block type. If pretrained weights are provided, load them after checking the input. Else, continue with no change or raise an error for unsupported inputs. Initialize the network parameters using truncated normal distribution for Linear layers and setting bias of LayerNorm layers to zero.",
+ "type": "comment"
+ },
+ "7178": {
+ "file_id": 518,
+ "content": " ones_(m.weight)\n zeros_(m.bias)\n def forward_features(self, x):\n # B = x.shape[0]\n B = paddle.shape(x)[0]\n x, T, W = self.patch_embed(x) # [BT,nH*nW,F]\n cls_tokens = self.cls_token.expand((B * T, -1, -1)) # [1,1,F]->[BT,1,F]\n x = paddle.concat((cls_tokens, x), axis=1)\n pos_interp = (x.shape[1] != self.pos_embed.shape[1])\n if pos_interp:\n pos_embed = self.pos_embed\n cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)\n other_pos_embed = pos_embed[0, 1:, :].unsqueeze(0).transpose(\n (0, 2, 1))\n P = int(other_pos_embed.shape[2]**0.5)\n H = x.shape[1] // W\n other_pos_embed = other_pos_embed.reshape([1, x.shape[2], P, P])\n new_pos_embed = F.interpolate(other_pos_embed,\n size=(H, W),\n mode='nearest')\n new_pos_embed = new_pos_embed.flatten(2)\n new_pos_embed = new_pos_embed.transpose((0, 2, 1))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:442-464"
+ },
+ "7179": {
+ "file_id": 518,
+ "content": "This code snippet is part of a transformer model's forward pass implementation. It reshapes the positional embeddings to match the patch embedding dimension and performs interpolation if necessary, ensuring the correct size for the subsequent layers.",
+ "type": "comment"
+ },
+ "7180": {
+ "file_id": 518,
+ "content": " new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed),\n axis=1)\n x = paddle.add(x, new_pos_embed)\n else:\n x = paddle.add(x, self.pos_embed)\n x = self.pos_drop(x)\n # Time Embeddings\n if self.attention_type != 'space_only':\n cls_tokens = x[:B, 0, :].unsqueeze(1) if B > 0 else x.split(\n T)[0].index_select(paddle.to_tensor([0]), axis=1)\n x = x[:, 1:]\n _, _n, _m = x.shape\n _t = T\n x = x.reshape([-1, _t, _n, _m]).transpose(\n (0, 2, 1, 3)).reshape([-1, _t, _m])\n # Resizing time embeddings in case they don't match\n time_interp = (T != self.time_embed.shape[1])\n if time_interp: # T' != T\n time_embed = self.time_embed.transpose((0, 2, 1)).unsqueeze(0)\n new_time_embed = F.interpolate(time_embed,\n size=(T, x.shape[-1]),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:465-487"
+ },
+ "7181": {
+ "file_id": 518,
+ "content": "This code is part of a vision transformer model. It concatenates the class position embeddings with new position embeddings, adds them to the input tensor, and applies positional dropout. If attention type is not \"space_only,\" it extracts time embeddings from the input tensor, reshapes them, interpolates time embeddings if their size doesn't match, and performs some operations on them.",
+ "type": "comment"
+ },
+ "7182": {
+ "file_id": 518,
+ "content": " mode='nearest').squeeze(0)\n new_time_embed = new_time_embed.transpose((0, 2, 1))\n x = paddle.add(x, new_time_embed)\n else:\n x = paddle.add(x, self.time_embed)\n x = self.time_drop(x)\n _, _t, _m = x.shape\n x = x.reshape([-1, W * W * T, _m])\n x = paddle.concat((cls_tokens, x), axis=1)\n # Attention blocks\n for blk in self.blocks:\n x = blk(x, B, T, W)\n # Predictions for space-only baseline\n if self.attention_type == 'space_only':\n _, _n, _m = x.shape\n _t = T\n x = x.reshape([-1, _t, _n, _m])\n x = paddle.mean(x, 1) # averaging predictions for every frame\n x = self.norm(x)\n return x[:, 0] # [B, embed_dim]\n def forward(self, x):\n x = self.forward_features(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/vit_tweaks.py:488-515"
+ },
+ "7183": {
+ "file_id": 518,
+ "content": "This code performs time-based feature modification and passes the data through attention blocks for a Vision Transformer model. It also provides an option to compute space-only predictions by averaging predictions for every frame. The forward function applies the forward_features transformation before passing data through attention blocks and normalization.",
+ "type": "comment"
+ },
+ "7184": {
+ "file_id": 519,
+ "content": "/paddlevideo/modeling/backbones/yowo.py",
+ "type": "filepath"
+ },
+ "7185": {
+ "file_id": 519,
+ "content": "The CAM_Module and YOWO backbone model are for image processing and video classification respectively, using attention mechanism and convolutional layers. The code loads pretrain weights correctly and returns a Paddle Video YOWO model after processing input clips through backbones, CFAM, and convolutional layers.",
+ "type": "summary"
+ },
+ "7186": {
+ "file_id": 519,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..registry import BACKBONES\nfrom .darknet import Darknet\nfrom .resnext101 import ResNext101\nimport paddle.nn as nn\nimport paddle\nclass CAM_Module(nn.Layer):\n def __init__(self, in_dim):\n super(CAM_Module, self).__init__()\n self.chanel_in = in_dim\n temp = paddle.zeros([1], dtype='float32')\n self.gamma = paddle.create_parameter(shape=temp.shape, dtype=str(temp.numpy().dtype),\n default_initializer=paddle.nn.initializer.Assign(temp))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/yowo.py:1-28"
+ },
+ "7187": {
+ "file_id": 519,
+ "content": "This code is a part of the PaddleVideo library and defines a custom layer called CAM_Module. It takes an input dimension as a parameter, initializes a gamma parameter, and inherits from nn.Layer. The class constructor creates a zero-dimensional tensor as the initial value for gamma using paddle.create_parameter function. This module is used in backbone architectures to enable Channel Attention Mechanism for image processing tasks.",
+ "type": "comment"
+ },
+ "7188": {
+ "file_id": 519,
+ "content": " self.softmax = nn.Softmax(axis=-1)\n def forward(self, x):\n m_batchsize, C, height, width = x.shape\n proj_query = paddle.reshape(x, [m_batchsize, C, -1])\n proj_key = paddle.transpose(paddle.reshape(\n x, [m_batchsize, C, -1]), perm=[0, 2, 1])\n energy = paddle.bmm(proj_query, proj_key)\n energy_new = paddle.expand_as(paddle.max(\n energy, axis=-1, keepdim=True), energy) - energy\n attention = self.softmax(energy_new)\n proj_value = paddle.reshape(x, [m_batchsize, C, -1])\n out = paddle.bmm(attention, proj_value)\n out = out.reshape([m_batchsize, C, height, width])\n out = self.gamma * out + x\n return out\nclass CFAMBlock(nn.Layer):\n def __init__(self, in_channels, out_channels):\n super(CFAMBlock, self).__init__()\n inter_channels = 1024\n self.conv_bn_relu1 = nn.Sequential(nn.Conv2D(in_channels, inter_channels, kernel_size=1, bias_attr=False),\n nn.BatchNorm2D(inter_channels),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/yowo.py:29-53"
+ },
+ "7189": {
+ "file_id": 519,
+ "content": "The code defines a CFAMPBlock layer with a Channel-wise Attention Mechanism. It contains a convolution, batch normalization, and ReLU layers for the attention mechanism, followed by a gamma scaling and channel-wise attention calculation. The forward function performs the attention operation and scales the input using the attention map.",
+ "type": "comment"
+ },
+ "7190": {
+ "file_id": 519,
+ "content": " nn.ReLU())\n self.conv_bn_relu2 = nn.Sequential(nn.Conv2D(inter_channels, inter_channels, 3, padding=1, bias_attr=False),\n nn.BatchNorm2D(inter_channels),\n nn.ReLU())\n self.sc = CAM_Module(inter_channels)\n self.conv_bn_relu3 = nn.Sequential(nn.Conv2D(inter_channels, inter_channels, 3, padding=1, bias_attr=False),\n nn.BatchNorm2D(inter_channels),\n nn.ReLU())\n self.conv_out = nn.Sequential(nn.Dropout2D(0.1), nn.Conv2D(\n inter_channels, out_channels, 1, bias_attr=True))\n def forward(self, x):\n x = self.conv_bn_relu1(x)\n x = self.conv_bn_relu2(x)\n x = self.sc(x)\n x = self.conv_bn_relu3(x)\n output = self.conv_out(x)\n return output\n@BACKBONES.register()\nclass YOWO(nn.Layer):\n def __init__(self, num_class, pretrained_2d=None, pretrained_3d=None):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/yowo.py:54-79"
+ },
+ "7191": {
+ "file_id": 519,
+ "content": "This code defines a YOWO backbone model, which is a neural network architecture for video classification tasks. It consists of several convolutional layers followed by batch normalization and ReLU activations. The CAM_Module is also included, which might be a custom attention mechanism. The output channels are adjusted based on the input size. Dropout regularization is applied to prevent overfitting.",
+ "type": "comment"
+ },
+ "7192": {
+ "file_id": 519,
+ "content": " super(YOWO, self).__init__()\n self.pretrained_2d = pretrained_2d\n self.pretrained_3d = pretrained_3d\n self.backbone_2d = Darknet()\n self.backbone_3d = ResNext101()\n self.num_ch_2d = 425\n self.num_ch_3d = 2048\n self.num_class = num_class\n self.cfam = CFAMBlock(self.num_ch_2d + self.num_ch_3d, 1024)\n self.conv_final = nn.Conv2D(\n 1024, 5 * (self.num_class + 4 + 1), kernel_size=1, bias_attr=False)\n self.seen = 0\n def init_weights(self):\n if self.pretrained_2d is not None:\n self.backbone_2d = self.load_pretrain_weight(\n self.backbone_2d, self.pretrained_2d)\n if self.pretrained_3d is not None:\n self.backbone_3d = self.load_pretrain_weight(\n self.backbone_3d, self.pretrained_3d)\n def load_pretrain_weight(self, model, weights_path):\n model_dict = model.state_dict()\n param_state_dict = paddle.load(weights_path)\n ignore_weights = set()\n # hack: fit for faster rcnn. Pretrain weights contain prefix of 'backbone'",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/yowo.py:80-108"
+ },
+ "7193": {
+ "file_id": 519,
+ "content": "The code initializes a YOWO model with pre-trained 2D and 3D backbones, loads pre-trained weights if provided for both backbones, and has a method to initialize weights.",
+ "type": "comment"
+ },
+ "7194": {
+ "file_id": 519,
+ "content": " # while res5 module is located in bbox_head.head. Replace the prefix of\n # res5 with 'bbox_head.head' to load pretrain weights correctly.\n for k in list(param_state_dict.keys()):\n if 'backbone.res5' in k:\n new_k = k.replace('backbone', 'bbox_head.head')\n if new_k in model_dict.keys():\n value = param_state_dict.pop(k)\n param_state_dict[new_k] = value\n for name, weight in param_state_dict.items():\n if name in model_dict.keys():\n if list(weight.shape) != list(model_dict[name].shape):\n print(\n '{} not used, shape {} unmatched with {} in model.'.format(\n name, weight.shape, list(model_dict[name].shape)))\n ignore_weights.add(name)\n else:\n print('Redundant weight {} and ignore it.'.format(name))\n ignore_weights.add(name)\n for weight in ignore_weights:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/yowo.py:109-129"
+ },
+ "7195": {
+ "file_id": 519,
+ "content": "This code is replacing the prefix of 'res5' with 'bbox_head.head' in param_state_dict to load pretrain weights correctly. It then checks if the weight shapes match and adds redundant or unmatched weights to ignore_weights.",
+ "type": "comment"
+ },
+ "7196": {
+ "file_id": 519,
+ "content": " param_state_dict.pop(weight, None)\n model.set_dict(param_state_dict)\n print('Finish loading model weights: {}'.format(weights_path))\n return model\n def forward(self, input):\n x_3d = input # Input clip\n x_2d = input[:, :, -1, :, :] # Last frame of the clip that is read\n x_2d = self.backbone_2d(x_2d)\n x_3d = self.backbone_3d(x_3d)\n x_3d = paddle.squeeze(x_3d, axis=2)\n x = paddle.concat([x_3d, x_2d], axis=1)\n x = self.cfam(x)\n out = self.conv_final(x)\n return out",
+ "type": "code",
+ "location": "/paddlevideo/modeling/backbones/yowo.py:130-150"
+ },
+ "7197": {
+ "file_id": 519,
+ "content": "This function loads model weights from the specified path and returns a Paddle Video YOWO model. The model's `forward` method takes an input clip, separates it into 3D and 2D representations, passes them through their respective backbones, concatenates them together, and finally feeds it to CFAM and a convolutional layer for processing before returning the output.",
+ "type": "comment"
+ },
+ "7198": {
+ "file_id": 520,
+ "content": "/paddlevideo/modeling/bbox_utils.py",
+ "type": "filepath"
+ },
+ "7199": {
+ "file_id": 520,
+ "content": "This code calculates delta between bounding boxes, adjusts using weighted averages, provides functions for filtering, computing overlaps, generating anchor points, decoding YOLO boxes, and calculating IoU. It transforms coordinates, computes deltas, stacks results, calculates dimensions and center of rotated boxes, converts rectangles to polygons, and finds the best begin point for a coordinate.",
+ "type": "summary"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/72.json b/docs/data/72.json
new file mode 100644
index 000000000..52d0a0df0
--- /dev/null
+++ b/docs/data/72.json
@@ -0,0 +1,543 @@
+{
+ "7200": {
+ "file_id": 520,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn.functional as F\nimport math\nimport numpy as np\ndef bbox2delta(src_boxes, tgt_boxes, weights):\n src_w = src_boxes[:, 2] - src_boxes[:, 0]\n src_h = src_boxes[:, 3] - src_boxes[:, 1]\n src_ctr_x = src_boxes[:, 0] + 0.5 * src_w\n src_ctr_y = src_boxes[:, 1] + 0.5 * src_h\n tgt_w = tgt_boxes[:, 2] - tgt_boxes[:, 0]\n tgt_h = tgt_boxes[:, 3] - tgt_boxes[:, 1]\n tgt_ctr_x = tgt_boxes[:, 0] + 0.5 * tgt_w",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:1-30"
+ },
+ "7201": {
+ "file_id": 520,
+ "content": "This code calculates the delta between source and target bounding boxes. It first computes the width and height of both source and target boxes, then their center coordinates. The variables are initialized and calculated for further usage in other functions related to bounding box transformation.",
+ "type": "comment"
+ },
+ "7202": {
+ "file_id": 520,
+ "content": " tgt_ctr_y = tgt_boxes[:, 1] + 0.5 * tgt_h\n wx, wy, ww, wh = weights\n dx = wx * (tgt_ctr_x - src_ctr_x) / src_w\n dy = wy * (tgt_ctr_y - src_ctr_y) / src_h\n dw = ww * paddle.log(tgt_w / src_w)\n dh = wh * paddle.log(tgt_h / src_h)\n deltas = paddle.stack((dx, dy, dw, dh), axis=1)\n return deltas\ndef delta2bbox(deltas, boxes, weights):\n clip_scale = math.log(1000.0 / 16)\n widths = boxes[:, 2] - boxes[:, 0]\n heights = boxes[:, 3] - boxes[:, 1]\n ctr_x = boxes[:, 0] + 0.5 * widths\n ctr_y = boxes[:, 1] + 0.5 * heights\n wx, wy, ww, wh = weights\n dx = deltas[:, 0::4] / wx\n dy = deltas[:, 1::4] / wy\n dw = deltas[:, 2::4] / ww\n dh = deltas[:, 3::4] / wh\n # Prevent sending too large values into paddle.exp()\n dw = paddle.clip(dw, max=clip_scale)\n dh = paddle.clip(dh, max=clip_scale)\n pred_ctr_x = dx * widths.unsqueeze(1) + ctr_x.unsqueeze(1)\n pred_ctr_y = dy * heights.unsqueeze(1) + ctr_y.unsqueeze(1)\n pred_w = paddle.exp(dw) * widths.unsqueeze(1)\n pred_h = paddle.exp(dh) * heights.unsqueeze(1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:31-63"
+ },
+ "7203": {
+ "file_id": 520,
+ "content": "This code calculates the differentials (deltas) between target and source bounding boxes, then converts those deltas back into new bounding box coordinates. The conversion is done with weighted averages for x, y, width, and height adjustments, ensuring values are clipped to avoid large inputs for paddle.exp().",
+ "type": "comment"
+ },
+ "7204": {
+ "file_id": 520,
+ "content": " pred_boxes = []\n pred_boxes.append(pred_ctr_x - 0.5 * pred_w)\n pred_boxes.append(pred_ctr_y - 0.5 * pred_h)\n pred_boxes.append(pred_ctr_x + 0.5 * pred_w)\n pred_boxes.append(pred_ctr_y + 0.5 * pred_h)\n pred_boxes = paddle.stack(pred_boxes, axis=-1)\n return pred_boxes\ndef expand_bbox(bboxes, scale):\n w_half = (bboxes[:, 2] - bboxes[:, 0]) * .5\n h_half = (bboxes[:, 3] - bboxes[:, 1]) * .5\n x_c = (bboxes[:, 2] + bboxes[:, 0]) * .5\n y_c = (bboxes[:, 3] + bboxes[:, 1]) * .5\n w_half *= scale\n h_half *= scale\n bboxes_exp = np.zeros(bboxes.shape, dtype=np.float32)\n bboxes_exp[:, 0] = x_c - w_half\n bboxes_exp[:, 2] = x_c + w_half\n bboxes_exp[:, 1] = y_c - h_half\n bboxes_exp[:, 3] = y_c + h_half\n return bboxes_exp\ndef clip_bbox(boxes, im_shape):\n h, w = im_shape[0], im_shape[1]\n x1 = boxes[:, 0].clip(0, w)\n y1 = boxes[:, 1].clip(0, h)\n x2 = boxes[:, 2].clip(0, w)\n y2 = boxes[:, 3].clip(0, h)\n return paddle.stack([x1, y1, x2, y2], axis=1)\ndef nonempty_bbox(boxes, min_size=0, return_mask=False):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:65-102"
+ },
+ "7205": {
+ "file_id": 520,
+ "content": "The code contains three functions: `expand_bbox`, `clip_bbox`, and `nonempty_bbox`. `expand_bbox` takes bbox coordinates, scales them by a factor, and returns the expanded bboxes. `clip_bbox` clips the bbox coordinates to the image shape boundaries. `nonempty_bbox` filters out empty bounding boxes based on a minimum size threshold or returns a mask if return_mask is True.",
+ "type": "comment"
+ },
+ "7206": {
+ "file_id": 520,
+ "content": " w = boxes[:, 2] - boxes[:, 0]\n h = boxes[:, 3] - boxes[:, 1]\n mask = paddle.logical_and(w > min_size, w > min_size)\n if return_mask:\n return mask\n keep = paddle.nonzero(mask).flatten()\n return keep\ndef bbox_area(boxes):\n return (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])\ndef bbox_overlaps(boxes1, boxes2):\n \"\"\"\n Calculate overlaps between boxes1 and boxes2\n Args:\n boxes1 (Tensor): boxes with shape [M, 4]\n boxes2 (Tensor): boxes with shape [N, 4]\n Return:\n overlaps (Tensor): overlaps between boxes1 and boxes2 with shape [M, N]\n \"\"\"\n area1 = bbox_area(boxes1)\n area2 = bbox_area(boxes2)\n xy_max = paddle.minimum(\n paddle.unsqueeze(boxes1, 1)[:, :, 2:], boxes2[:, 2:])\n xy_min = paddle.maximum(\n paddle.unsqueeze(boxes1, 1)[:, :, :2], boxes2[:, :2])\n width_height = xy_max - xy_min\n width_height = width_height.clip(min=0)\n inter = width_height.prod(axis=2)\n overlaps = paddle.where(inter > 0, inter /\n (paddle.unsqueeze(area1, 1) + area2 - inter),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:103-139"
+ },
+ "7207": {
+ "file_id": 520,
+ "content": "The function `bbox_utils.py` at line 102-138 contains two functions: 'filter_boxes_by_size' and 'bbox_overlaps'.\n'filter_boxes_by_size' filters the bounding boxes by size, only keeping those whose width or height exceeds a specified minimum size. If a mask is also desired, it returns one of true values for the selected bounding boxes.\n'bbox_overlaps' calculates overlaps between two sets of bounding boxes and returns them as a tensor with shape [M, N]. This function uses the areas of the bounding boxes to compute the intersections and unions, applying clipping for valid computations and handling non-intersecting boxes.\n\nExplanation: The code contains functions that filter bounding boxes by size and calculate overlaps between them. The 'filter_boxes_by_size' function filters bounding boxes based on their width or height, while the 'bbox_overlaps' function calculates overlap between two sets of bounding boxes and returns a tensor with shape [M, N].",
+ "type": "comment"
+ },
+ "7208": {
+ "file_id": 520,
+ "content": " paddle.zeros_like(inter))\n return overlaps\ndef xywh2xyxy(box):\n x, y, w, h = box\n x1 = x - w * 0.5\n y1 = y - h * 0.5\n x2 = x + w * 0.5\n y2 = y + h * 0.5\n return [x1, y1, x2, y2]\ndef make_grid(h, w, dtype):\n yv, xv = paddle.meshgrid([paddle.arange(h), paddle.arange(w)])\n return paddle.stack((xv, yv), 2).cast(dtype=dtype)\ndef decode_yolo(box, anchor, downsample_ratio):\n \"\"\"decode yolo box\n Args:\n box (list): [x, y, w, h], all have the shape [b, na, h, w, 1]\n anchor (list): anchor with the shape [na, 2]\n downsample_ratio (int): downsample ratio, default 32\n scale (float): scale, default 1.\n Return:\n box (list): decoded box, [x, y, w, h], all have the shape [b, na, h, w, 1]\n \"\"\"\n x, y, w, h = box\n na, grid_h, grid_w = x.shape[1:4]\n grid = make_grid(grid_h, grid_w, x.dtype).reshape((1, 1, grid_h, grid_w, 2))\n x1 = (x + grid[:, :, :, :, 0:1]) / grid_w\n y1 = (y + grid[:, :, :, :, 1:2]) / grid_h\n anchor = paddle.to_tensor(anchor)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:140-176"
+ },
+ "7209": {
+ "file_id": 520,
+ "content": "The code defines functions for converting box coordinates, generating a grid of anchor points, and decoding YOLO bounding boxes. The \"xywh2xyxy\" function transforms (x, y, w, h) to (x1, y1, x2, y2). The \"make_grid\" function generates a grid of coordinates for downsampled images. The \"decode_yolo\" function decodes YOLO bounding boxes using anchor points and downsample ratios.",
+ "type": "comment"
+ },
+ "7210": {
+ "file_id": 520,
+ "content": " anchor = paddle.cast(anchor, x.dtype)\n anchor = anchor.reshape((1, na, 1, 1, 2))\n w1 = paddle.exp(w) * anchor[:, :, :, :, 0:1] / (downsample_ratio * grid_w)\n h1 = paddle.exp(h) * anchor[:, :, :, :, 1:2] / (downsample_ratio * grid_h)\n return [x1, y1, w1, h1]\ndef iou_similarity(box1, box2, eps=1e-9):\n \"\"\"Calculate iou of box1 and box2\n Args:\n box1 (Tensor): box with the shape [N, M1, 4]\n box2 (Tensor): box with the shape [N, M2, 4]\n Return:\n iou (Tensor): iou between box1 and box2 with the shape [N, M1, M2]\n \"\"\"\n box1 = box1.unsqueeze(2) # [N, M1, 4] -> [N, M1, 1, 4]\n box2 = box2.unsqueeze(1) # [N, M2, 4] -> [N, 1, M2, 4]\n px1y1, px2y2 = box1[:, :, :, 0:2], box1[:, :, :, 2:4]\n gx1y1, gx2y2 = box2[:, :, :, 0:2], box2[:, :, :, 2:4]\n x1y1 = paddle.maximum(px1y1, gx1y1)\n x2y2 = paddle.minimum(px2y2, gx2y2)\n overlap = (x2y2 - x1y1).clip(0).prod(-1)\n area1 = (px2y2 - px1y1).clip(0).prod(-1)\n area2 = (gx2y2 - gx1y1).clip(0).prod(-1)\n union = area1 + area2 - overlap + eps",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:177-204"
+ },
+ "7211": {
+ "file_id": 520,
+ "content": "Code defines anchor and calculates width (w1) and height (h1) for bounding boxes based on exponential values of w and h, downsample ratio, and grid dimensions. It also includes a function iou_similarity that calculates the intersection over union (IoU) between two sets of bounding boxes.",
+ "type": "comment"
+ },
+ "7212": {
+ "file_id": 520,
+ "content": " return overlap / union\ndef bbox_iou(box1, box2, giou=False, diou=False, ciou=False, eps=1e-9):\n \"\"\"calculate the iou of box1 and box2\n Args:\n box1 (list): [x, y, w, h], all have the shape [b, na, h, w, 1]\n box2 (list): [x, y, w, h], all have the shape [b, na, h, w, 1]\n giou (bool): whether use giou or not, default False\n diou (bool): whether use diou or not, default False\n ciou (bool): whether use ciou or not, default False\n eps (float): epsilon to avoid divide by zero\n Return:\n iou (Tensor): iou of box1 and box1, with the shape [b, na, h, w, 1]\n \"\"\"\n px1, py1, px2, py2 = box1\n gx1, gy1, gx2, gy2 = box2\n x1 = paddle.maximum(px1, gx1)\n y1 = paddle.maximum(py1, gy1)\n x2 = paddle.minimum(px2, gx2)\n y2 = paddle.minimum(py2, gy2)\n overlap = ((x2 - x1).clip(0)) * ((y2 - y1).clip(0))\n area1 = (px2 - px1) * (py2 - py1)\n area1 = area1.clip(0)\n area2 = (gx2 - gx1) * (gy2 - gy1)\n area2 = area2.clip(0)\n union = area1 + area2 - overlap + eps",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:205-237"
+ },
+ "7213": {
+ "file_id": 520,
+ "content": "This function calculates the intersection over union (IoU) between two bounding boxes, box1 and box2. It supports various IoU metrics such as Giou, Diou, or Ciou. The calculated IoU is returned as a tensor with the same shape as box1 and box2. This function is used in object detection tasks to measure the overlap between predicted and ground truth bounding boxes.",
+ "type": "comment"
+ },
+ "7214": {
+ "file_id": 520,
+ "content": " iou = overlap / union\n if giou or ciou or diou:\n # convex w, h\n cw = paddle.maximum(px2, gx2) - paddle.minimum(px1, gx1)\n ch = paddle.maximum(py2, gy2) - paddle.minimum(py1, gy1)\n if giou:\n c_area = cw * ch + eps\n return iou - (c_area - union) / c_area\n else:\n # convex diagonal squared\n c2 = cw**2 + ch**2 + eps\n # center distance\n rho2 = ((px1 + px2 - gx1 - gx2)**2 + (py1 + py2 - gy1 - gy2)**2) / 4\n if diou:\n return iou - rho2 / c2\n else:\n w1, h1 = px2 - px1, py2 - py1 + eps\n w2, h2 = gx2 - gx1, gy2 - gy1 + eps\n delta = paddle.atan(w1 / h1) - paddle.atan(w2 / h2)\n v = (4 / math.pi**2) * paddle.pow(delta, 2)\n alpha = v / (1 + eps - iou + v)\n alpha.stop_gradient = True\n return iou - (rho2 / c2 + v * alpha)\n else:\n return iou\ndef rect2rbox(bboxes):\n \"\"\"\n :param bboxes: shape (n, 4) (xmin, ymin, xmax, ymax)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:238-268"
+ },
+ "7215": {
+ "file_id": 520,
+ "content": "This code calculates the Intersection over Union (IoU) between two bounding boxes and applies various forms of IoU calculations depending on the input parameters. It first checks if giou, ciou, or diou is True and then proceeds with the corresponding calculation based on the convex area, diagonal distance, or aspect ratio difference between the bounding boxes. The rect2rbox function transforms a set of bounding boxes from (xmin, ymin, xmax, ymax) format to (cx, cy, w, h) format where cx and cy are center coordinates and w and h are width and height respectively.",
+ "type": "comment"
+ },
+ "7216": {
+ "file_id": 520,
+ "content": " :return: dbboxes: shape (n, 5) (x_ctr, y_ctr, w, h, angle)\n \"\"\"\n bboxes = bboxes.reshape(-1, 4)\n num_boxes = bboxes.shape[0]\n x_ctr = (bboxes[:, 2] + bboxes[:, 0]) / 2.0\n y_ctr = (bboxes[:, 3] + bboxes[:, 1]) / 2.0\n edges1 = np.abs(bboxes[:, 2] - bboxes[:, 0])\n edges2 = np.abs(bboxes[:, 3] - bboxes[:, 1])\n angles = np.zeros([num_boxes], dtype=bboxes.dtype)\n inds = edges1 < edges2\n rboxes = np.stack((x_ctr, y_ctr, edges1, edges2, angles), axis=1)\n rboxes[inds, 2] = edges2[inds]\n rboxes[inds, 3] = edges1[inds]\n rboxes[inds, 4] = np.pi / 2.0\n return rboxes\ndef delta2rbox(Rrois,\n deltas,\n means=[0, 0, 0, 0, 0],\n stds=[1, 1, 1, 1, 1],\n wh_ratio_clip=1e-6):\n \"\"\"\n :param Rrois: (cx, cy, w, h, theta)\n :param deltas: (dx, dy, dw, dh, dtheta)\n :param means:\n :param stds:\n :param wh_ratio_clip:\n :return:\n \"\"\"\n means = paddle.to_tensor(means)\n stds = paddle.to_tensor(stds)\n deltas = paddle.reshape(deltas, [-1, deltas.shape[-1]])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:269-304"
+ },
+ "7217": {
+ "file_id": 520,
+ "content": "This function converts bounding box coordinates and dimensions to rotation-aligned bounding boxes by calculating the center, edge lengths, and angle. It returns a new tensor with reshaped and rotated bounding boxes.",
+ "type": "comment"
+ },
+ "7218": {
+ "file_id": 520,
+ "content": " denorm_deltas = deltas * stds + means\n dx = denorm_deltas[:, 0]\n dy = denorm_deltas[:, 1]\n dw = denorm_deltas[:, 2]\n dh = denorm_deltas[:, 3]\n dangle = denorm_deltas[:, 4]\n max_ratio = np.abs(np.log(wh_ratio_clip))\n dw = paddle.clip(dw, min=-max_ratio, max=max_ratio)\n dh = paddle.clip(dh, min=-max_ratio, max=max_ratio)\n Rroi_x = Rrois[:, 0]\n Rroi_y = Rrois[:, 1]\n Rroi_w = Rrois[:, 2]\n Rroi_h = Rrois[:, 3]\n Rroi_angle = Rrois[:, 4]\n gx = dx * Rroi_w * paddle.cos(Rroi_angle) - dy * Rroi_h * paddle.sin(\n Rroi_angle) + Rroi_x\n gy = dx * Rroi_w * paddle.sin(Rroi_angle) + dy * Rroi_h * paddle.cos(\n Rroi_angle) + Rroi_y\n gw = Rroi_w * dw.exp()\n gh = Rroi_h * dh.exp()\n ga = np.pi * dangle + Rroi_angle\n ga = (ga + np.pi / 4) % np.pi - np.pi / 4\n ga = paddle.to_tensor(ga)\n gw = paddle.to_tensor(gw, dtype='float32')\n gh = paddle.to_tensor(gh, dtype='float32')\n bboxes = paddle.stack([gx, gy, gw, gh, ga], axis=-1)\n return bboxes\ndef rbox2delta(proposals, gt, means=[0, 0, 0, 0, 0], stds=[1, 1, 1, 1, 1]):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:305-339"
+ },
+ "7219": {
+ "file_id": 520,
+ "content": "This code computes the bounding box regression results for each proposed bounding box. It calculates the deltas and applies clipping to ensure they stay within reasonable bounds, then transforms these deltas into actual bounding box coordinates. The resulting bounding boxes are stacked in a tensor and returned as output.",
+ "type": "comment"
+ },
+ "7220": {
+ "file_id": 520,
+ "content": " \"\"\"\n Args:\n proposals:\n gt:\n means: 1x5\n stds: 1x5\n Returns:\n \"\"\"\n proposals = proposals.astype(np.float64)\n PI = np.pi\n gt_widths = gt[..., 2]\n gt_heights = gt[..., 3]\n gt_angle = gt[..., 4]\n proposals_widths = proposals[..., 2]\n proposals_heights = proposals[..., 3]\n proposals_angle = proposals[..., 4]\n coord = gt[..., 0:2] - proposals[..., 0:2]\n dx = (np.cos(proposals[..., 4]) * coord[..., 0] + np.sin(proposals[..., 4])\n * coord[..., 1]) / proposals_widths\n dy = (-np.sin(proposals[..., 4]) * coord[..., 0] + np.cos(proposals[..., 4])\n * coord[..., 1]) / proposals_heights\n dw = np.log(gt_widths / proposals_widths)\n dh = np.log(gt_heights / proposals_heights)\n da = (gt_angle - proposals_angle)\n da = (da + PI / 4) % PI - PI / 4\n da /= PI\n deltas = np.stack([dx, dy, dw, dh, da], axis=-1)\n means = np.array(means, dtype=deltas.dtype)\n stds = np.array(stds, dtype=deltas.dtype)\n deltas = (deltas - means) / stds",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:340-378"
+ },
+ "7221": {
+ "file_id": 520,
+ "content": "This code calculates the delta values between ground truth and proposals for bounding boxes, taking into account their widths, heights, angles, and applying normalization based on means and stds. It also ensures that angle differences are within 0 to 2π range before scaling by the inverse of PI.",
+ "type": "comment"
+ },
+ "7222": {
+ "file_id": 520,
+ "content": " deltas = deltas.astype(np.float32)\n return deltas\ndef bbox_decode(bbox_preds,\n anchors,\n means=[0, 0, 0, 0, 0],\n stds=[1, 1, 1, 1, 1]):\n \"\"\"decode bbox from deltas\n Args:\n bbox_preds: [N,H,W,5]\n anchors: [H*W,5]\n return:\n bboxes: [N,H,W,5]\n \"\"\"\n means = paddle.to_tensor(means)\n stds = paddle.to_tensor(stds)\n num_imgs, H, W, _ = bbox_preds.shape\n bboxes_list = []\n for img_id in range(num_imgs):\n bbox_pred = bbox_preds[img_id]\n # bbox_pred.shape=[5,H,W]\n bbox_delta = bbox_pred\n anchors = paddle.to_tensor(anchors)\n bboxes = delta2rbox(\n anchors, bbox_delta, means, stds, wh_ratio_clip=1e-6)\n bboxes = paddle.reshape(bboxes, [H, W, 5])\n bboxes_list.append(bboxes)\n return paddle.stack(bboxes_list, axis=0)\ndef poly_to_rbox(polys):\n \"\"\"\n poly:[x0,y0,x1,y1,x2,y2,x3,y3]\n to\n rotated_boxes:[x_ctr,y_ctr,w,h,angle]\n \"\"\"\n rotated_boxes = []\n for poly in polys:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:379-417"
+ },
+ "7223": {
+ "file_id": 520,
+ "content": "Function `bbox_decode` takes bbox predictions, anchors and means/stds as inputs. It returns decoded bounding boxes. It first converts the means and stds to tensors. Then for each image, it computes the bbox delta from the bbox predictions. It then transforms these deltas to actual bounding box coordinates using `delta2rbox` function. Finally, it reshapes the obtained bounding boxes and stores them in a list. The function returns a stack of all the bounding boxes for each image.",
+ "type": "comment"
+ },
+ "7224": {
+ "file_id": 520,
+ "content": " poly = np.array(poly[:8], dtype=np.float32)\n pt1 = (poly[0], poly[1])\n pt2 = (poly[2], poly[3])\n pt3 = (poly[4], poly[5])\n pt4 = (poly[6], poly[7])\n edge1 = np.sqrt((pt1[0] - pt2[0]) * (pt1[0] - pt2[0]) + (pt1[1] - pt2[\n 1]) * (pt1[1] - pt2[1]))\n edge2 = np.sqrt((pt2[0] - pt3[0]) * (pt2[0] - pt3[0]) + (pt2[1] - pt3[\n 1]) * (pt2[1] - pt3[1]))\n width = max(edge1, edge2)\n height = min(edge1, edge2)\n rbox_angle = 0\n if edge1 > edge2:\n rbox_angle = np.arctan2(\n np.float(pt2[1] - pt1[1]), np.float(pt2[0] - pt1[0]))\n elif edge2 >= edge1:\n rbox_angle = np.arctan2(\n np.float(pt4[1] - pt1[1]), np.float(pt4[0] - pt1[0]))\n def norm_angle(angle, range=[-np.pi / 4, np.pi]):\n return (angle - range[0]) % range[1] + range[0]\n rbox_angle = norm_angle(rbox_angle)\n x_ctr = np.float(pt1[0] + pt3[0]) / 2\n y_ctr = np.float(pt1[1] + pt3[1]) / 2",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:418-447"
+ },
+ "7225": {
+ "file_id": 520,
+ "content": "This code calculates the width, height, and angle of a rotated bounding box based on its eight points. It first converts the polyline to a numpy array, then calculates the lengths of two edges. The function then determines the maximum edge length as the width and the minimum edge length as the height. Based on these values, it computes the rotation angle using arctan2. Finally, it normalizes the rotation angle within a specified range using the norm_angle function. It also calculates the x and y coordinates of the box's center.",
+ "type": "comment"
+ },
+ "7226": {
+ "file_id": 520,
+ "content": " rotated_box = np.array([x_ctr, y_ctr, width, height, rbox_angle])\n rotated_boxes.append(rotated_box)\n ret_rotated_boxes = np.array(rotated_boxes)\n assert ret_rotated_boxes.shape[1] == 5\n return ret_rotated_boxes\ndef cal_line_length(point1, point2):\n import math\n return math.sqrt(\n math.pow(point1[0] - point2[0], 2) + math.pow(point1[1] - point2[1], 2))\ndef get_best_begin_point_single(coordinate):\n x1, y1, x2, y2, x3, y3, x4, y4 = coordinate\n xmin = min(x1, x2, x3, x4)\n ymin = min(y1, y2, y3, y4)\n xmax = max(x1, x2, x3, x4)\n ymax = max(y1, y2, y3, y4)\n combinate = [[[x1, y1], [x2, y2], [x3, y3], [x4, y4]],\n [[x4, y4], [x1, y1], [x2, y2], [x3, y3]],\n [[x3, y3], [x4, y4], [x1, y1], [x2, y2]],\n [[x2, y2], [x3, y3], [x4, y4], [x1, y1]]]\n dst_coordinate = [[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]]\n force = 100000000.0\n force_flag = 0\n for i in range(4):\n temp_force = cal_line_length(combinate[i][0], dst_coordinate[0]) \\",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:448-475"
+ },
+ "7227": {
+ "file_id": 520,
+ "content": "This function `get_best_begin_point_single` takes a coordinate as input and calculates the minimum x and y coordinates (xmin, ymin) and maximum x and y coordinates (xmax, ymax). It then defines four different combinations of the four points in the coordinate and compares these combinations to the destination coordinate (dst_coordinate) by calculating the distance using `cal_line_length` function. The combination with the smallest distance is returned as the best begin point. The code also includes a force variable to handle potential edge cases where no valid begin point can be found.",
+ "type": "comment"
+ },
+ "7228": {
+ "file_id": 520,
+ "content": " + cal_line_length(combinate[i][1], dst_coordinate[1]) \\\n + cal_line_length(combinate[i][2], dst_coordinate[2]) \\\n + cal_line_length(combinate[i][3], dst_coordinate[3])\n if temp_force < force:\n force = temp_force\n force_flag = i\n if force_flag != 0:\n pass\n return np.array(combinate[force_flag]).reshape(8)\ndef rbox2poly_single(rrect):\n \"\"\"\n rrect:[x_ctr,y_ctr,w,h,angle]\n to\n poly:[x0,y0,x1,y1,x2,y2,x3,y3]\n \"\"\"\n x_ctr, y_ctr, width, height, angle = rrect[:5]\n tl_x, tl_y, br_x, br_y = -width / 2, -height / 2, width / 2, height / 2\n # rect 2x4\n rect = np.array([[tl_x, br_x, br_x, tl_x], [tl_y, tl_y, br_y, br_y]])\n R = np.array([[np.cos(angle), -np.sin(angle)],\n [np.sin(angle), np.cos(angle)]])\n # poly\n poly = R.dot(rect)\n x0, x1, x2, x3 = poly[0, :4] + x_ctr\n y0, y1, y2, y3 = poly[1, :4] + y_ctr\n poly = np.array([x0, y0, x1, y1, x2, y2, x3, y3], dtype=np.float32)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:476-503"
+ },
+ "7229": {
+ "file_id": 520,
+ "content": "This function `rbox2poly_single` takes a rectangle represented by its center coordinates, width, height, and angle, and converts it to a polygon representation. It first calculates the top-left and bottom-right coordinates of the rectangle. Then, it creates a 2x4 matrix representing the four corners of the rectangle. The function applies a rotation matrix to transform the rectangle into a rotated coordinate system. Finally, it shifts the transformed coordinates by the center coordinates and returns the polygon representation as an array.",
+ "type": "comment"
+ },
+ "7230": {
+ "file_id": 520,
+ "content": " poly = get_best_begin_point_single(poly)\n return poly\ndef rbox2poly(rrects):\n \"\"\"\n rrect:[x_ctr,y_ctr,w,h,angle]\n to\n poly:[x0,y0,x1,y1,x2,y2,x3,y3]\n \"\"\"\n polys = []\n for rrect in rrects:\n x_ctr, y_ctr, width, height, angle = rrect[:5]\n tl_x, tl_y, br_x, br_y = -width / 2, -height / 2, width / 2, height / 2\n rect = np.array([[tl_x, br_x, br_x, tl_x], [tl_y, tl_y, br_y, br_y]])\n R = np.array([[np.cos(angle), -np.sin(angle)],\n [np.sin(angle), np.cos(angle)]])\n poly = R.dot(rect)\n x0, x1, x2, x3 = poly[0, :4] + x_ctr\n y0, y1, y2, y3 = poly[1, :4] + y_ctr\n poly = np.array([x0, y0, x1, y1, x2, y2, x3, y3], dtype=np.float32)\n poly = get_best_begin_point_single(poly)\n polys.append(poly)\n polys = np.array(polys)\n return polys",
+ "type": "code",
+ "location": "/paddlevideo/modeling/bbox_utils.py:504-528"
+ },
+ "7231": {
+ "file_id": 520,
+ "content": "This function `rbox2poly` converts a list of rotation rectangles (rrects) into polygons (polys). It first calculates the top-left and bottom-right coordinates of each rrect. Then, it rotates the rectangle using the given angle. The function adjusts the poly points by adding the x_ctr and y_ctr values to obtain the final poly. It applies a single best begin point adjustment (`get_best_begin_point_single`) and adds the poly to the list of polys. Finally, it returns the array of polygons.",
+ "type": "comment"
+ },
+ "7232": {
+ "file_id": 521,
+ "content": "/paddlevideo/modeling/builder.py",
+ "type": "filepath"
+ },
+ "7233": {
+ "file_id": 521,
+ "content": "This code imports modules and registers functions for building a video object detection model, as well as dynamically constructing components based on a configuration file.",
+ "type": "summary"
+ },
+ "7234": {
+ "file_id": 521,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import BACKBONES, HEADS, LOSSES, RECOGNIZERS, LOCALIZERS, ROI_EXTRACTORS, DETECTORS, BBOX_ASSIGNERS, BBOX_SAMPLERS, BBOX_CODERS, PARTITIONERS, MULTIMODAL, SEGMENT, SEGMENTERS\nfrom ..utils import build\nfrom .registry import (BACKBONES, BBOX_ASSIGNERS, BBOX_CODERS, BBOX_SAMPLERS,\n DETECTORS, ESTIMATORS, HEADS, LOCALIZERS, LOSSES,\n MULTIMODAL, PARTITIONERS, RECOGNIZERS, ROI_EXTRACTORS)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/builder.py:1-19"
+ },
+ "7235": {
+ "file_id": 521,
+ "content": "Imports necessary modules and registers various components for video object detection model building.",
+ "type": "comment"
+ },
+ "7236": {
+ "file_id": 521,
+ "content": "def build_backbone(cfg):\n \"\"\"Build backbone.\"\"\"\n return build(cfg, BACKBONES)\ndef build_roi_extractor(cfg):\n \"\"\"Build roi extractor.\"\"\"\n return build(cfg, ROI_EXTRACTORS)\ndef build_assigner(cfg, **default_args):\n \"\"\"Builder of box assigner.\"\"\"\n return build(cfg, BBOX_ASSIGNERS)\ndef build_sampler(cfg, **default_args):\n \"\"\"Builder of box sampler.\"\"\"\n return build(cfg, BBOX_SAMPLERS)\ndef build_roi_extractor(cfg):\n \"\"\"Build roi extractor.\"\"\"\n return build(cfg, ROI_EXTRACTORS)\ndef build_assigner(cfg, **default_args):\n \"\"\"Builder of box assigner.\"\"\"\n return build(cfg, BBOX_ASSIGNERS)\ndef build_sampler(cfg, **default_args):\n \"\"\"Builder of box sampler.\"\"\"\n return build(cfg, BBOX_SAMPLERS)\ndef build_head(cfg):\n \"\"\"Build head.\"\"\"\n return build(cfg, HEADS)\ndef build_loss(cfg):\n \"\"\"Build loss.\"\"\"\n return build(cfg, LOSSES)\ndef build_recognizer(cfg):\n \"\"\"Build recognizer.\"\"\"\n return build(cfg, RECOGNIZERS, key='framework')\ndef build_segmenter(cfg):\n \"\"\"Build segmenter.\"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/builder.py:22-73"
+ },
+ "7237": {
+ "file_id": 521,
+ "content": "This code defines various building functions for different parts of a model. The \"build_backbone\" function builds the backbone of the model, while \"build_roi_extractor\", \"build_assigner\", and \"build_sampler\" build the region of interest extractor, box assigner, and box sampler respectively. \"build_head\" builds the head of the model, \"build_loss\" builds the loss function, and \"build_recognizer\" and \"build_segmenter\" build recognizers and segmenters with different frameworks or keys. The functions use the \"build\" method from an unspecified source (\"*\") to perform the actual building process.",
+ "type": "comment"
+ },
+ "7238": {
+ "file_id": 521,
+ "content": " return build(cfg, SEGMENTERS, key='framework')\ndef build_localizer(cfg):\n \"\"\"Build localizer.\"\"\"\n return build(cfg, LOCALIZERS, key='framework')\ndef build_detector(cfg, train_cfg=None, test_cfg=None):\n \"\"\"Build detector.\"\"\"\n return build(cfg, DETECTORS, key='framework')\ndef build_partitioner(cfg):\n \"\"\"Build partitioner.\"\"\"\n return build(cfg, PARTITIONERS, key='framework')\ndef build_estimator(cfg):\n \"\"\"Build estimator.\"\"\"\n return build(cfg, ESTIMATORS, key='framework')\ndef build_multimodal(cfg):\n \"\"\"Build multimodal.\"\"\"\n return build(cfg, MULTIMODAL, key='framework')\ndef build_segment(cfg):\n \"\"\"Build segment.\"\"\"\n return build(cfg, SEGMENT, key='framework')\ndef build_model(cfg):\n cfg_copy = cfg.copy()\n framework_type = cfg_copy.get('framework')\n if framework_type in RECOGNIZERS:\n return build_recognizer(cfg)\n elif framework_type in LOCALIZERS:\n return build_localizer(cfg)\n elif framework_type in PARTITIONERS:\n return build_partitioner(cfg)\n elif framework_type in DETECTORS:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/builder.py:74-116"
+ },
+ "7239": {
+ "file_id": 521,
+ "content": "The code is a builder function that dynamically builds various components (recognizers, localizers, detectors, partitioners, estimators, and multimodal) based on the specified framework type in the configuration file. It utilizes the 'build' function to return the appropriate component for further processing.",
+ "type": "comment"
+ },
+ "7240": {
+ "file_id": 521,
+ "content": " return build_detector(cfg)\n elif framework_type in ESTIMATORS:\n return build_estimator(cfg)\n elif framework_type in MULTIMODAL:\n return build_multimodal(cfg)\n elif framework_type in SEGMENTERS:\n return build_segmenter(cfg)\n elif framework_type in SEGMENT:\n return build_segment(cfg)\n else:\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/builder.py:117-127"
+ },
+ "7241": {
+ "file_id": 521,
+ "content": "This code selects a specific function to build based on the framework type. It checks if the framework type is in defined lists of detectors, estimators, multimodal models, segmenters, or segments. If none match, it raises NotImplementedError.",
+ "type": "comment"
+ },
+ "7242": {
+ "file_id": 522,
+ "content": "/paddlevideo/modeling/framework/__init__.py",
+ "type": "filepath"
+ },
+ "7243": {
+ "file_id": 522,
+ "content": "This code is part of the PaddleVideo framework, which provides base classes for various model classes like 'BasePartitioner', 'TransNetV2Partitioner', 'BaseEstimator', 'DepthEstimator', and more. The list contains names of these available base classes within the module.",
+ "type": "summary"
+ },
+ "7244": {
+ "file_id": 522,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .estimators import BaseEstimator, DepthEstimator\nfrom .localizers import BaseLocalizer, BMNLocalizer\nfrom .partitioners import BasePartitioner, TransNetV2Partitioner\nfrom .recognizers import BaseRecognizer, Recognizer2D\nfrom .multimodal import ActBert, BaseMultimodal\nfrom .segment import BaseSegment, CFBI\nfrom .segmenters import MSTCN\n__all__ = [\n 'BaseRecognizer', 'Recognizer2D', 'BaseLocalizer', 'BMNLocalizer',",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/__init__.py:1-24"
+ },
+ "7245": {
+ "file_id": 522,
+ "content": "This code is part of the PaddleVideo framework, which provides several base classes for estimators, localizers, partitioners, recognizers, multimodal models, segments, and segmenters. The commented lines describe the license information and the imported classes from different modules within the framework. The '__all__' list contains the names of the base classes available in this module.",
+ "type": "comment"
+ },
+ "7246": {
+ "file_id": 522,
+ "content": " 'BasePartitioner', 'TransNetV2Partitioner', 'BaseEstimator',\n 'DepthEstimator', 'BaseMultimodal', 'ActBert', 'BaseSegment', 'CFBI',\n 'MSTCN'\n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/__init__.py:25-28"
+ },
+ "7247": {
+ "file_id": 522,
+ "content": "This code snippet contains a list of various model classes, including 'BasePartitioner', 'TransNetV2Partitioner', 'BaseEstimator', 'DepthEstimator', 'BaseMultimodal', 'ActBert', 'BaseSegment', 'CFBI', and 'MSTCN'. These models are likely used for different tasks within the PaddleVideo framework.",
+ "type": "comment"
+ },
+ "7248": {
+ "file_id": 523,
+ "content": "/paddlevideo/modeling/framework/detectors/__init__.py",
+ "type": "filepath"
+ },
+ "7249": {
+ "file_id": 523,
+ "content": "This code is part of the PaddleVideo library, specifically defining detectors. It imports three detector classes (BaseDetector, FastRCNN, and TwoStageDetector) from its local directory and lists them in __all__. The comment at the beginning establishes copyright information and licensing.",
+ "type": "summary"
+ },
+ "7250": {
+ "file_id": 523,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom .base import BaseDetector\nfrom .fast_rcnn import FastRCNN\nfrom .two_stage import TwoStageDetector\n__all__ = ['BaseDetector', 'TwoStageDetector', 'FastRCNN']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/__init__.py:1-17"
+ },
+ "7251": {
+ "file_id": 523,
+ "content": "This code is part of the PaddleVideo library, specifically defining detectors. It imports three detector classes (BaseDetector, FastRCNN, and TwoStageDetector) from its local directory and lists them in __all__. The comment at the beginning establishes copyright information and licensing.",
+ "type": "comment"
+ },
+ "7252": {
+ "file_id": 524,
+ "content": "/paddlevideo/modeling/framework/detectors/base.py",
+ "type": "filepath"
+ },
+ "7253": {
+ "file_id": 524,
+ "content": "BaseDetector class serves as a parent for detectors, providing common features and abstract train_step method implementation. Abstract base classes are defined for training, validating, and testing steps in machine learning models.",
+ "type": "summary"
+ },
+ "7254": {
+ "file_id": 524,
+ "content": "from abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nfrom ...registry import DETECTORS\n@DETECTORS.register()\nclass BaseDetector(nn.Layer):\n \"\"\"Base class for detectors. \"\"\"\n def __init__(self, backbone=None, head=None):\n super().__init__()\n def init_weights(self):\n \"\"\"Initialize the model network weights. \"\"\"\n self.backbone.init_weights() \n self.head.init_weights()\n def extract_feature(self, imgs, iter_num):\n \"\"\"Extract features through a backbone. \"\"\"\n feature = self.backbone(imgs)\n return feature\n def forward(self, data_batch, mode='infer'):\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/base.py:1-36"
+ },
+ "7255": {
+ "file_id": 524,
+ "content": "BaseDetector class is the parent class for detectors, providing common functionality like extracting features and initializing weights. It defines an abstract train_step method that needs to be implemented by subclasses for training. The class also contains methods for feature extraction, model forward pass, and handling different modes (train, valid, test, infer).",
+ "type": "comment"
+ },
+ "7256": {
+ "file_id": 524,
+ "content": " \"\"\"Training step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/base.py:37-51"
+ },
+ "7257": {
+ "file_id": 524,
+ "content": "This code defines abstract base classes for training, validating, and testing steps in a machine learning model. These methods must be implemented by subclasses to perform the specific tasks accordingly.",
+ "type": "comment"
+ },
+ "7258": {
+ "file_id": 525,
+ "content": "/paddlevideo/modeling/framework/detectors/fast_rcnn.py",
+ "type": "filepath"
+ },
+ "7259": {
+ "file_id": 525,
+ "content": "FastRCNN is a two-stage object detection class inheriting from TwoStageDetector, created with specified head, train and test configurations, and optional pretrained weights.",
+ "type": "summary"
+ },
+ "7260": {
+ "file_id": 525,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .two_stage import TwoStageDetector\nfrom ...registry import DETECTORS\n@DETECTORS.register()\nclass FastRCNN(TwoStageDetector):\n def __init__(self,\n backbone,\n head=None,\n train_cfg=None,\n test_cfg=None,\n neck=None,\n pretrained=None):\n super(FastRCNN, self).__init__(\n backbone=backbone,\n neck=neck,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/fast_rcnn.py:1-30"
+ },
+ "7261": {
+ "file_id": 525,
+ "content": "Defines the FastRCNN class, a two-stage detector that inherits from TwoStageDetector. It takes backbone, head, train_cfg, test_cfg, neck, and pretrained as parameters for object detection.",
+ "type": "comment"
+ },
+ "7262": {
+ "file_id": 525,
+ "content": " roi_head=head,\n train_cfg=train_cfg,\n test_cfg=test_cfg,\n pretrained=pretrained)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/fast_rcnn.py:31-34"
+ },
+ "7263": {
+ "file_id": 525,
+ "content": "Creates a Fast R-CNN detector with specified head, train and test configurations, and optionally pretrained weights.",
+ "type": "comment"
+ },
+ "7264": {
+ "file_id": 526,
+ "content": "/paddlevideo/modeling/framework/detectors/two_stage.py",
+ "type": "filepath"
+ },
+ "7265": {
+ "file_id": 526,
+ "content": "This code uses the slowfast model for object detection, initializes components and parameters, supports pretrained weights, and provides methods for training, testing, and inference. It also retrieves data from PaddleVideo's two-stage detector with various inputs and entity ID selection using index_select.",
+ "type": "summary"
+ },
+ "7266": {
+ "file_id": 526,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nfrom ... import builder\nimport paddle.distributed as dist\nfrom ...registry import DETECTORS\nfrom .base import BaseDetector\n@DETECTORS.register()\nclass TwoStageDetector(BaseDetector):\n \"\"\"Base class for two-stage detectors. \"\"\"\n def __init__(self,\n backbone,\n neck=None,\n rpn_head=None,\n roi_head=None,\n train_cfg=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:1-32"
+ },
+ "7267": {
+ "file_id": 526,
+ "content": "This code is importing necessary libraries, registering a two-stage detector class (TwoStageDetector) within the DETECTORS registry, and initializing its components. The class serves as a base for implementing two-stage object detection algorithms.",
+ "type": "comment"
+ },
+ "7268": {
+ "file_id": 526,
+ "content": " test_cfg=None,\n pretrained=None):\n super(TwoStageDetector, self).__init__()\n self.backbone = builder.build_backbone(backbone)\n if neck is not None:\n self.neck = neck # useless\n if rpn_head is not None:\n rpn_train_cfg = train_cfg.rpn if train_cfg is not None else None\n rpn_head_ = rpn_head.copy()\n rpn_head_.update(train_cfg=rpn_train_cfg, test_cfg=test_cfg.rpn)\n self.rpn_head = builder.build_head(rpn_head_)\n if roi_head is not None:\n self.roi_head = builder.build_head(roi_head)\n self.train_cfg = train_cfg\n self.test_cfg = test_cfg\n if pretrained is not None:\n self.init_weights(pretrained=pretrained)\n @property\n def with_rpn(self):\n \"\"\"whether the detector has RPN\"\"\"\n return hasattr(self, 'rpn_head') and self.rpn_head is not None\n @property\n def with_roi_head(self):\n \"\"\"whether the detector has a RoI head\"\"\"\n return hasattr(self, 'roi_head') and self.roi_head is not None",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:33-64"
+ },
+ "7269": {
+ "file_id": 526,
+ "content": "This code defines a class for a two-stage object detection model. It initializes the backbone, neck (if provided), and heads for RPN and ROI. The constructor also takes optional train_cfg and test_cfg parameters for each head. Additional pretrained weights can be loaded later if provided. The @property methods check whether the detector has RPN or ROI head.",
+ "type": "comment"
+ },
+ "7270": {
+ "file_id": 526,
+ "content": " def init_weights(self, pretrained=None):\n \"\"\"Initialize the weights in detector. \"\"\"\n super(TwoStageDetector, self).init_weights(pretrained)\n self.backbone.init_weights(pretrained=pretrained)\n if self.with_rpn:\n self.rpn_head.init_weights()\n if self.with_roi_head:\n self.roi_head.init_weights(pretrained)\n def extract_feat(self, img):\n \"\"\"Directly extract features from the backbone.\"\"\"\n x = self.backbone(img)\n return x\n def train_step(self, data, **kwargs):\n img_slow = data[0]\n img_fast = data[1]\n proposals, gt_bboxes, gt_labels, scores, entity_ids = self.get_unpad_datas(\n data)\n img_shape = data[7]\n img_idx = data[8]\n img_metas = scores, entity_ids\n x = self.extract_feat(img=[img_slow, img_fast])\n roi_losses = self.roi_head.train_step(x, img_metas, proposals,\n gt_bboxes, gt_labels, **kwargs)\n losses = dict()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:66-91"
+ },
+ "7271": {
+ "file_id": 526,
+ "content": "The code initializes the weights of a two-stage detector and extracts features from its backbone. The train_step function takes input data, extracts features using the extract_feat method, and computes roi_losses using the roi_head's train_step method. These losses are then stored in the losses dictionary.",
+ "type": "comment"
+ },
+ "7272": {
+ "file_id": 526,
+ "content": " losses.update(roi_losses)\n return losses\n def val_step(self, data, rescale=False):\n img_slow = data[0]\n img_fast = data[1]\n proposals, gt_bboxes, gt_labels, scores, entity_ids = self.get_unpad_datas(\n data)\n img_shape = data[7]\n img_metas = scores, entity_ids\n x = self.extract_feat(img=[img_slow, img_fast])\n return self.roi_head.simple_test(x,\n proposals[0],\n img_shape,\n rescale=rescale)\n def test_step(self, data, rescale=False):\n return self.val_step(data, rescale)\n def infer_step(self, data, rescale=False):\n ''' model inference'''\n img_slow = data[0]\n img_fast = data[1]\n proposals = data[2]\n img_shape = data[3]\n # using slowfast model to extract spatio-temporal features\n x = self.extract_feat(img=[img_slow, img_fast])\n ret = self.roi_head.simple_test(x,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:92-124"
+ },
+ "7273": {
+ "file_id": 526,
+ "content": "This code defines three methods, val_step, test_step, and infer_step. All these methods extract features using the slowfast model and then pass them to roi_head for further processing. Val_step is used for validation while test_step is used for testing. Infer_step performs inference using previously obtained data.",
+ "type": "comment"
+ },
+ "7274": {
+ "file_id": 526,
+ "content": " proposals[0],\n img_shape,\n rescale=rescale)\n return ret\n def get_unpad_datas(self, data):\n ''' get original datas padded in dataset '''\n pad_proposals = data[2]\n pad_gt_bboxes = data[3]\n pad_gt_labels = data[4]\n pad_scores, pad_entity_ids = data[5], data[6]\n len_proposals = data[9]\n len_gt_bboxes = data[10]\n len_gt_labels = data[11]\n len_scores = data[12]\n len_entity_ids = data[13]\n N = pad_proposals.shape[0]\n proposals = []\n gt_bboxes = []\n gt_labels = []\n scores = []\n entity_ids = []\n for bi in range(N):\n pad_proposal = pad_proposals[bi]\n len_proposal = len_proposals[bi]\n index_proposal = paddle.arange(len_proposal)\n proposal = paddle.index_select(x=pad_proposal,\n index=index_proposal,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:125-152"
+ },
+ "7275": {
+ "file_id": 526,
+ "content": "This code snippet is part of the PaddleVideo library's two-stage detector implementation. It defines a function that retrieves original data from padded dataset, and another function for getting unpadded datas. The first function takes in a set of proposals, ground truth bboxes, labels, scores, and entity ids, and returns them as unpadded data based on the number of proposals at each index. The second function retrieves original datas padded in dataset for two-stage detector implementation.",
+ "type": "comment"
+ },
+ "7276": {
+ "file_id": 526,
+ "content": " axis=0)\n proposals.append(proposal)\n pad_gt_bbox = pad_gt_bboxes[bi]\n len_gt_bbox = len_gt_bboxes[bi]\n index_gt_bbox = paddle.arange(len_gt_bbox)\n gt_bbox = paddle.index_select(x=pad_gt_bbox,\n index=index_gt_bbox,\n axis=0)\n gt_bboxes.append(gt_bbox)\n pad_gt_label = pad_gt_labels[bi]\n len_gt_label = len_gt_labels[bi]\n index_gt_label = paddle.arange(len_gt_label)\n gt_label = paddle.index_select(x=pad_gt_label,\n index=index_gt_label,\n axis=0)\n gt_labels.append(gt_label)\n pad_score = pad_scores[bi]\n len_score = len_scores[bi]\n index_score = paddle.arange(len_score)\n score = paddle.index_select(x=pad_score, index=index_score, axis=0)\n scores.append(score)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:153-176"
+ },
+ "7277": {
+ "file_id": 526,
+ "content": "This code creates a list of proposals, and corresponding ground truth bounding boxes (gt_bboxes), labels (gt_labels), and scores. It handles batches by iterating over each batch index (bi) and for each batch, it performs index selection on the padded data based on the indices of the current length of the batch to extract the relevant gt_bbox, gt_label, and score information. These are then appended to their respective lists.",
+ "type": "comment"
+ },
+ "7278": {
+ "file_id": 526,
+ "content": " pad_entity_id = pad_entity_ids[bi]\n len_entity_id = len_entity_ids[bi]\n index_entity_id = paddle.arange(len_entity_id)\n entity_id = paddle.index_select(x=pad_entity_id,\n index=index_entity_id,\n axis=0)\n entity_ids.append(entity_id)\n return proposals, gt_bboxes, gt_labels, scores, entity_ids",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/detectors/two_stage.py:178-186"
+ },
+ "7279": {
+ "file_id": 526,
+ "content": "This code segment is selecting specific entity IDs from a list and appending them to the 'entity_ids' list. It uses Paddle's index_select function to achieve this.",
+ "type": "comment"
+ },
+ "7280": {
+ "file_id": 527,
+ "content": "/paddlevideo/modeling/framework/estimators/__init__.py",
+ "type": "filepath"
+ },
+ "7281": {
+ "file_id": 527,
+ "content": "This code imports necessary classes and defines the publically accessible '__all__' list, containing the DepthEstimator and BaseEstimator classes.",
+ "type": "summary"
+ },
+ "7282": {
+ "file_id": 527,
+ "content": "from .base import BaseEstimator\nfrom .depth_estimator import DepthEstimator\n__all__ = ['DepthEstimator', 'BaseEstimator']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/__init__.py:1-4"
+ },
+ "7283": {
+ "file_id": 527,
+ "content": "This code imports necessary classes and defines the publically accessible '__all__' list, containing the DepthEstimator and BaseEstimator classes.",
+ "type": "comment"
+ },
+ "7284": {
+ "file_id": 528,
+ "content": "/paddlevideo/modeling/framework/estimators/base.py",
+ "type": "filepath"
+ },
+ "7285": {
+ "file_id": 528,
+ "content": "The code creates a PaddleVideo BaseEstimator class, inheriting from nn.Layer and utilizing builder for backbone construction. It initializes weights, registers the class, and sets forward modes for validation, testing, and inference, with abstract methods that must be implemented by subclasses.",
+ "type": "summary"
+ },
+ "7286": {
+ "file_id": 528,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\nfrom paddlevideo.modeling.registry import ESTIMATORS\nfrom paddlevideo.utils import get_logger\nfrom ... import builder\nlogger = get_logger(\"paddlevideo\")\n@ESTIMATORS.register()\nclass BaseEstimator(nn.Layer):\n \"\"\"BaseEstimator\n \"\"\"\n def __init__(self, backbone=None, head=None):\n super().__init__()\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/base.py:1-34"
+ },
+ "7287": {
+ "file_id": 528,
+ "content": "This code is defining a base class for an estimator in PaddleVideo. It inherits from nn.Layer, uses builder to construct the backbone if specified, and initializes the weights of the backbone if it has an init_weights method. The ESTIMATORS registry is used to register this BaseEstimator class.",
+ "type": "comment"
+ },
+ "7288": {
+ "file_id": 528,
+ "content": " self.backbone.init_weights()\n else:\n self.backbone = None\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/base.py:35-66"
+ },
+ "7289": {
+ "file_id": 528,
+ "content": "The code initializes the backbone and head components of a model depending on their availability. It then defines four forward modes (train, valid, test, infer) to execute the model accordingly. The train_step abstract method must be implemented separately.",
+ "type": "comment"
+ },
+ "7290": {
+ "file_id": 528,
+ "content": " raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch):\n \"\"\"Define how the model is going to valid, from input to output.\"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/base.py:67-82"
+ },
+ "7291": {
+ "file_id": 528,
+ "content": "This code defines abstract methods for model validation, testing, and inference steps. It raises a NotImplementedError to ensure subclasses must implement these methods.",
+ "type": "comment"
+ },
+ "7292": {
+ "file_id": 529,
+ "content": "/paddlevideo/modeling/framework/estimators/depth_estimator.py",
+ "type": "filepath"
+ },
+ "7293": {
+ "file_id": 529,
+ "content": "The DepthEstimator class inherits from BaseEstimator and contains a forward_net method for feature extraction. It has training, validation, testing, and inference methods with loss metrics calculated using the forward_net and head.loss.",
+ "type": "summary"
+ },
+ "7294": {
+ "file_id": 529,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nfrom paddlevideo.modeling.framework.estimators.base import BaseEstimator\nfrom paddlevideo.modeling.registry import ESTIMATORS\nfrom paddlevideo.utils import get_logger\nfrom ... import builder\nlogger = get_logger(\"paddlevideo\")\n@ESTIMATORS.register()\nclass DepthEstimator(BaseEstimator):\n \"\"\"DepthEstimator\n \"\"\"\n def forward_net(self, inputs, day_or_night='day_and_night'):\n if self.backbone is not None:\n outputs = self.backbone(inputs, day_or_night)\n else:\n outputs = inputs",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/depth_estimator.py:1-31"
+ },
+ "7295": {
+ "file_id": 529,
+ "content": "The code defines a DepthEstimator class that inherits from BaseEstimator. It has a forward_net method that takes inputs and optionally applies a backbone network for feature extraction. The results are stored in outputs.",
+ "type": "comment"
+ },
+ "7296": {
+ "file_id": 529,
+ "content": " return outputs\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n inputs, _ = data_batch\n outputs = self.forward_net(inputs, day_or_night='day_and_night')\n loss_metrics = self.head.loss(inputs, outputs)\n return loss_metrics\n def val_step(self, data_batch):\n inputs, day_or_night = data_batch\n outputs = self.forward_net(inputs, day_or_night=day_or_night)\n loss_metrics = self.head.loss(inputs, outputs)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n inputs, day_or_night = data_batch\n outputs = self.forward_net(inputs, day_or_night=day_or_night)\n loss_metrics = self.head.loss(inputs, outputs)\n return loss_metrics\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n inputs = data_batch[0]\n outputs = self.forward_net(inputs, day_or_night='day')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/depth_estimator.py:32-58"
+ },
+ "7297": {
+ "file_id": 529,
+ "content": "The code defines four methods: train_step, val_step, test_step, and infer_step. The main purpose of each step is to calculate the loss metrics from the input data. The 'forward_net' method is used in all steps to process the inputs and generate outputs, which are then passed to the 'head.loss' method to compute the loss metrics for each step.",
+ "type": "comment"
+ },
+ "7298": {
+ "file_id": 529,
+ "content": " return outputs",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/estimators/depth_estimator.py:59-59"
+ },
+ "7299": {
+ "file_id": 529,
+ "content": "This code snippet returns the output results from a depth estimator model.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/73.json b/docs/data/73.json
new file mode 100644
index 000000000..63b7220c9
--- /dev/null
+++ b/docs/data/73.json
@@ -0,0 +1,541 @@
+{
+ "7300": {
+ "file_id": 530,
+ "content": "/paddlevideo/modeling/framework/localizers/__init__.py",
+ "type": "filepath"
+ },
+ "7301": {
+ "file_id": 530,
+ "content": "This code is a part of the PaddleVideo library and contains localizers, which are responsible for handling different types of video localization tasks. The base class \"BaseLocalizer\" serves as a parent class, while specific classes like \"BMNLocalizer\" and \"YOWOLocalizer\" extend it to handle various localization techniques. These localizers may be used in a wide range of applications depending on the required localization approach.",
+ "type": "summary"
+ },
+ "7302": {
+ "file_id": 530,
+ "content": "# copyright (c) 2020 paddlepaddle authors. all rights reserved.\n#\n# licensed under the apache license, version 2.0 (the \"license\"\n# you may not use this file except in compliance with the license.\n# you may obtain a copy of the license at\n#\n# http://www.apache.org/licenses/license-2.0\n#\n# unless required by applicable law or agreed to in writing, software\n# distributed under the license is distributed on an \"as is\" basis,\n# without warranties or conditions of any kind, either express or implied.\n# see the license for the specific language governing permissions and\n# limitations under the license.\nfrom .base import BaseLocalizer\nfrom .bmn_localizer import BMNLocalizer\nfrom .yowo_localizer import YOWOLocalizer\n__all__ = ['BaseLocalizer', 'BMNLocalizer', 'YOWOLocalizer']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/__init__.py:1-19"
+ },
+ "7303": {
+ "file_id": 530,
+ "content": "This code is a part of the PaddleVideo library and contains localizers, which are responsible for handling different types of video localization tasks. The base class \"BaseLocalizer\" serves as a parent class, while specific classes like \"BMNLocalizer\" and \"YOWOLocalizer\" extend it to handle various localization techniques. These localizers may be used in a wide range of applications depending on the required localization approach.",
+ "type": "comment"
+ },
+ "7304": {
+ "file_id": 531,
+ "content": "/paddlevideo/modeling/framework/localizers/base.py",
+ "type": "filepath"
+ },
+ "7305": {
+ "file_id": 531,
+ "content": "This code defines a base class for localization models using PaddlePaddle framework, with train, valid, and test steps implemented in subclasses. It supports different operation modes and allows weight initialization.",
+ "type": "summary"
+ },
+ "7306": {
+ "file_id": 531,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom abc import abstractmethod\nimport paddle.nn as nn\nfrom ... import builder\nclass BaseLocalizer(nn.Layer):\n \"\"\"Base class for Localization.\n All localizer should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, define your train step.\n - Methods:``valid_step``, define your valid step, always the same as train_step.\n - Methods:``test_step``, define your test step.\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/base.py:1-27"
+ },
+ "7307": {
+ "file_id": 531,
+ "content": "This code snippet defines a base class for localization models. All subclasses of this base class should implement train_step, valid_step, and test_step methods to define their respective steps in the model's training process. It uses PaddlePaddle's framework and is licensed under the Apache License, Version 2.0.",
+ "type": "comment"
+ },
+ "7308": {
+ "file_id": 531,
+ "content": " def __init__(self, backbone, loss):\n super().__init__()\n self.backbone = builder.build_backbone(backbone)\n self.loss = builder.build_loss(loss)\n self.init_weights()\n def init_weights(self):\n \"\"\"Initialize the model network weights. \"\"\"\n if getattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n pass\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/base.py:28-56"
+ },
+ "7309": {
+ "file_id": 531,
+ "content": "This code initializes a localizer model, handling backbone and loss functions, and allows for different operation modes (train, valid, test, infer). It also includes a function to initialize the model's network weights.",
+ "type": "comment"
+ },
+ "7310": {
+ "file_id": 531,
+ "content": " @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step. input_data_batch -> loss_metric\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating setp. input_data_batch -> loss_metric\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Tets setp. to get acc in test data. input_data_batch -> output\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/base.py:58-74"
+ },
+ "7311": {
+ "file_id": 531,
+ "content": "This code defines abstract classes for training, validation, and testing steps in a model. The train_step, val_step, and test_step methods require implementation by subclasses to perform the necessary computations.",
+ "type": "comment"
+ },
+ "7312": {
+ "file_id": 532,
+ "content": "/paddlevideo/modeling/framework/localizers/bmn_localizer.py",
+ "type": "filepath"
+ },
+ "7313": {
+ "file_id": 532,
+ "content": "This code defines a localizer model for PaddleVideo with forward network and methods for training, validating, testing, and inferring. It uses input data to predict bounding boxes, start position, and end position while calculating loss using ground truth values.",
+ "type": "summary"
+ },
+ "7314": {
+ "file_id": 532,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import LOCALIZERS\nfrom .base import BaseLocalizer\nimport paddle\n@LOCALIZERS.register()\nclass BMNLocalizer(BaseLocalizer):\n \"\"\"BMN Localization framework\n \"\"\"\n def forward_net(self, imgs):\n \"\"\"Call backbone forward.\n \"\"\"\n preds = self.backbone(imgs)\n return preds\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n x_data = data_batch[0]\n gt_iou_map = data_batch[1]\n gt_start = data_batch[2]\n gt_end = data_batch[3]\n gt_iou_map.stop_gradient = True",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/bmn_localizer.py:1-36"
+ },
+ "7315": {
+ "file_id": 532,
+ "content": "This code is part of the PaddleVideo library and defines a BMNLocalizer class, which is a localization framework. It includes a forward_net method for calling the backbone's forward function and a train_step method for handling training steps with input data. The gt_iou_map, gt_start, and gt_end are provided as part of the data batch to be used in the training step.",
+ "type": "comment"
+ },
+ "7316": {
+ "file_id": 532,
+ "content": " gt_start.stop_gradient = True\n gt_end.stop_gradient = True\n # call Model forward\n pred_bm, pred_start, pred_end = self.forward_net(x_data)\n # call Loss forward\n loss = self.loss(pred_bm, pred_start, pred_end, gt_iou_map, gt_start,\n gt_end)\n avg_loss = paddle.mean(loss)\n loss_metrics = dict()\n loss_metrics['loss'] = avg_loss\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n return self.train_step(data_batch)\n def test_step(self, data_batch):\n \"\"\"Test step.\n \"\"\"\n x_data = data_batch[0]\n pred_bm, pred_start, pred_end = self.forward_net(x_data)\n return pred_bm, pred_start, pred_end\n def infer_step(self, data_batch):\n \"\"\"Infer step\n \"\"\"\n x_data = data_batch[0]\n # call Model forward\n pred_bm, pred_start, pred_end = self.forward_net(x_data)\n return pred_bm, pred_start, pred_end",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/bmn_localizer.py:37-69"
+ },
+ "7317": {
+ "file_id": 532,
+ "content": "This code defines a localizer model for PaddleVideo. It includes functions for training, validating, testing, and inferring steps. The localizer has a forward network which takes input data and returns predictions for bounding boxes (pred_bm), start position (pred_start), and end position (pred_end). Loss is calculated using the provided ground truth values (gt_iou_map, gt_start, gt_end) and averaged over the batch.",
+ "type": "comment"
+ },
+ "7318": {
+ "file_id": 533,
+ "content": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py",
+ "type": "filepath"
+ },
+ "7319": {
+ "file_id": 533,
+ "content": "The YOWOLocalizer extends BaseLocalizer, performs NMS on detected boxes, matches ground truth with predicted boxes based on IoU threshold, and calculates precision, recall, and F-score for YOWO localizer using test step function.",
+ "type": "summary"
+ },
+ "7320": {
+ "file_id": 533,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import LOCALIZERS\nfrom .base import BaseLocalizer\nfrom .yowo_utils import truths_length, nms, get_region_boxes, bbox_iou\n@LOCALIZERS.register()\nclass YOWOLocalizer(BaseLocalizer):\n \"\"\"YOWO Localization framework\n \"\"\"\n def forward_net(self, imgs):\n \"\"\"Call backbone forward.\n \"\"\"\n # imgs.shape=[N,C,T,H,W], for YOWO\n preds = self.backbone(imgs)\n return preds\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n x_data = data_batch[0]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py:1-33"
+ },
+ "7321": {
+ "file_id": 533,
+ "content": "Code from PaddleVideo's yowo_localizer.py file defines a YOWOLocalizer class which extends BaseLocalizer and utilizes the backbone function for forwarding image data. It also includes methods forward_net and train_step for processing images in training context.",
+ "type": "comment"
+ },
+ "7322": {
+ "file_id": 533,
+ "content": " target = data_batch[1].squeeze(1) # indeed do squeeze to adapt to paddle tensor\n target.stop_gradient = True\n # call Model forward\n out = self.forward_net(x_data)\n # call Loss forward\n loss, nCorrect = self.loss(out, target)\n loss_metrics = dict()\n loss_metrics['loss'] = loss\n loss_metrics['nCorrect'] = nCorrect\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n total = 0.0\n proposals = 0.0\n correct = 0.0\n fscore = 0.0\n eps = 1e-5\n nms_thresh = 0.4\n iou_thresh = 0.5\n x_data = data_batch[0]\n target = data_batch[1].squeeze(1) # indeed do squeeze to adapt to paddle tensor\n frame_idx = data_batch[2]\n target.stop_gradient = True\n # call Model forward\n out = self.forward_net(x_data)\n all_boxes = get_region_boxes(out)\n out_boxes = []\n for i in range(out.shape[0]):\n boxes = all_boxes[i]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py:34-67"
+ },
+ "7323": {
+ "file_id": 533,
+ "content": "Function \"forward_net\" is called to perform model's forward pass, and the output of this function call is stored in 'out'. The code then calls another function \"loss\" passing the output from the model (out) and the target data (target). The output from the loss function call is stored in 'loss', along with number of correct predictions ('nCorrect'). A dictionary named 'loss_metrics' is created storing 'loss' and 'nCorrect'. This process is part of training set step. \nThe 'val_step' function performs validation steps like calculating total, proposals, correct and fscore variables using specific values (total = 0.0, proposals = 0.0, correct = 0.0, fscore = 0.0). It also uses an epsilon value of 1e-5 and a nms_thresh and iou_thresh of 0.4 for certain calculations. It calls the model's forward pass (forward_net) to get 'out', then gets all region boxes using get_region_boxes function, then iterates over each box in out, storing them in a list named 'out_boxes'. This process is part of validating step.",
+ "type": "comment"
+ },
+ "7324": {
+ "file_id": 533,
+ "content": " boxes = nms(boxes, nms_thresh)\n out_boxes.append(boxes)\n truths = target[i].reshape([-1, 5])\n num_gts = truths_length(truths)\n total = total + num_gts\n pred_list = []\n for i in range(len(boxes)):\n if boxes[i][4] > 0.25:\n proposals = proposals + 1\n pred_list.append(i)\n for i in range(num_gts):\n box_gt = [truths[i][1], truths[i][2], truths[i][3], truths[i][4], 1.0, 1.0, truths[i][0]]\n best_iou = 0\n best_j = -1\n for j in pred_list: # ITERATE THROUGH ONLY CONFIDENT BOXES\n iou = bbox_iou(box_gt, boxes[j], x1y1x2y2=False)\n if iou > best_iou:\n best_j = j\n best_iou = iou\n if best_iou > iou_thresh and int(boxes[best_j][6]) == box_gt[6]:\n correct = correct + 1\n precision = 1.0 * correct / (proposals + eps)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py:68-90"
+ },
+ "7325": {
+ "file_id": 533,
+ "content": "This code performs Non-Maximum Suppression (NMS) on detected boxes, selects confident boxes for further processing, and associates ground truth boxes with predicted boxes based on Intersection over Union (IoU) threshold. It counts correct matches, proposals, total ground truth boxes, and calculates precision. The precision is calculated by dividing the number of correct matches by the sum of proposals and a small epsilon value to avoid division by zero.",
+ "type": "comment"
+ },
+ "7326": {
+ "file_id": 533,
+ "content": " recall = 1.0 * correct / (total + eps)\n fscore = 2.0 * precision * recall / (precision + recall + eps)\n outs = dict()\n outs['precision'] = precision\n outs['recall'] = recall\n outs['fscore'] = fscore\n outs['frame_idx'] = frame_idx\n return outs\n def test_step(self, data_batch):\n \"\"\"Test step.\n \"\"\"\n total = 0.0\n proposals = 0.0\n correct = 0.0\n fscore = 0.0\n eps = 1e-5\n nms_thresh = 0.4\n iou_thresh = 0.5\n x_data = data_batch[0]\n target = data_batch[1].squeeze(1) # indeed do squeeze to adapt to paddle tensor\n frame_idx = data_batch[2]\n target.stop_gradient = True\n # call Model forward\n out = self.forward_net(x_data)\n all_boxes = get_region_boxes(out)\n out_boxes = []\n for i in range(out.shape[0]):\n boxes = all_boxes[i]\n boxes = nms(boxes, nms_thresh)\n out_boxes.append(boxes)\n truths = target[i].reshape([-1, 5])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py:91-125"
+ },
+ "7327": {
+ "file_id": 533,
+ "content": "This code defines a localizer for the YOLOv3 model and calculates precision, recall, and F-score using test step function. It initializes variables, processes input data batch, applies non-maximum suppression (NMS) to regions of interest, and returns output metrics including precision, recall, F-score, and frame index.",
+ "type": "comment"
+ },
+ "7328": {
+ "file_id": 533,
+ "content": " num_gts = truths_length(truths)\n total = total + num_gts\n pred_list = []\n for i in range(len(boxes)):\n if boxes[i][4] > 0.25:\n proposals = proposals + 1\n pred_list.append(i)\n for i in range(num_gts):\n box_gt = [truths[i][1], truths[i][2], truths[i][3], truths[i][4], 1.0, 1.0, truths[i][0]]\n best_iou = 0\n best_j = -1\n for j in pred_list: # ITERATE THROUGH ONLY CONFIDENT BOXES\n iou = bbox_iou(box_gt, boxes[j], x1y1x2y2=False)\n if iou > best_iou:\n best_j = j\n best_iou = iou\n if best_iou > iou_thresh and int(boxes[best_j][6]) == box_gt[6]:\n correct = correct + 1\n precision = 1.0 * correct / (proposals + eps)\n recall = 1.0 * correct / (total + eps)\n fscore = 2.0 * precision * recall / (precision + recall + eps)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py:126-147"
+ },
+ "7329": {
+ "file_id": 533,
+ "content": "The code computes precision, recall, and F-score for each class of the YOWO localizer. It iterates through ground truth boxes and confident proposal boxes to match them based on Intersection over Union (IoU) threshold. It counts correctly matched boxes and total boxes, then calculates precision, recall, and F-score using these values.",
+ "type": "comment"
+ },
+ "7330": {
+ "file_id": 533,
+ "content": " outs = dict()\n outs['boxes'] = out_boxes\n outs['precision'] = precision\n outs['recall'] = recall\n outs['fscore'] = fscore\n outs['frame_idx'] = frame_idx\n return outs\n def infer_step(self, data_batch):\n \"\"\"Infer step.\n \"\"\"\n out = self.forward_net(data_batch[0])\n return out",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_localizer.py:149-161"
+ },
+ "7331": {
+ "file_id": 533,
+ "content": "This code defines two functions within the yowo_localizer class. The first function, \"infer_step\", takes in a data batch and feeds it into the forward_net to get an output. The second function returns dictionaries for boxes, precision, recall, fscore, and frame_idx as outputs.",
+ "type": "comment"
+ },
+ "7332": {
+ "file_id": 534,
+ "content": "/paddlevideo/modeling/framework/localizers/yowo_utils.py",
+ "type": "filepath"
+ },
+ "7333": {
+ "file_id": 534,
+ "content": "The code contains functions for non-maximum suppression, tensor movement, and applying NMS to anchor boxes in images using PaddlePaddle. It transforms YOLOv2 output, generates ground truth targets, calculates IoU, counts instances, updates predictions, and returns masks and transformation parameters for translation, width, and height.",
+ "type": "summary"
+ },
+ "7334": {
+ "file_id": 534,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn as nn\nimport numpy as np\nfrom builtins import range as xrange\ndef truths_length(truths):\n for i in range(50):\n if truths[i][1] == 0:\n return i\ndef nms(boxes, nms_thresh):\n if len(boxes) == 0:\n return boxes\n det_confs = paddle.zeros([len(boxes)])\n for i in range(len(boxes)):\n det_confs[i] = 1 - boxes[i][4]\n sortIds = paddle.argsort(det_confs)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:1-36"
+ },
+ "7335": {
+ "file_id": 534,
+ "content": "This code snippet defines a function `truths_length()` that returns the index of the first occurrence where the second element in 'truths' array is 0. It also defines a function `nms()` that applies Non-Maximum Suppression to filter out bounding boxes based on a given NMS threshold. The code checks if there are any bounding boxes, assigns confidence scores, sorts them in descending order of confidence, and removes overlapping bounding boxes with IoU greater than the NMS threshold.",
+ "type": "comment"
+ },
+ "7336": {
+ "file_id": 534,
+ "content": " out_boxes = []\n for i in range(len(boxes)):\n box_i = boxes[sortIds[i]]\n if box_i[4] > 0:\n out_boxes.append(box_i)\n for j in range(i + 1, len(boxes)):\n box_j = boxes[sortIds[j]]\n if bbox_iou(box_i, box_j, x1y1x2y2=False) > nms_thresh:\n box_j[4] = 0\n return out_boxes\ndef convert2cpu(gpu_matrix):\n float_32_g = gpu_matrix.astype('float32')\n return float_32_g.cpu()\ndef convert2cpu_long(gpu_matrix):\n int_64_g = gpu_matrix.astype('int64')\n return int_64_g.cpu()\ndef get_region_boxes(output, conf_thresh=0.005, num_classes=24,\n anchors=[0.70458, 1.18803, 1.26654, 2.55121, 1.59382,\n 4.08321, 2.30548, 4.94180, 3.52332, 5.91979],\n num_anchors=5, only_objectness=1, validation=False):\n anchor_step = len(anchors) // num_anchors\n if output.dim() == 3:\n output = output.unsqueeze(0)\n batch = output.shape[0]\n assert (output.shape[1] == (5 + num_classes) * num_anchors)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:37-67"
+ },
+ "7337": {
+ "file_id": 534,
+ "content": "This code is defining three functions: \"out_boxes\" appears to perform non-maximum suppression on bounding boxes, \"convert2cpu\" converts a tensor from GPU memory to CPU memory as a float32 type, and \"convert2cpu_long\" performs the same operation but as an int64 type. The \"get_region_boxes\" function takes in output from a model and applies non-maximum suppression for each anchor box in the image, using provided anchors and thresholds. This function also reshapes the input to have shape (batch, num_anchors, 5 + num_classes). The code includes assertions to ensure proper input shapes are being used.",
+ "type": "comment"
+ },
+ "7338": {
+ "file_id": 534,
+ "content": " h = output.shape[2]\n w = output.shape[3]\n all_boxes = []\n output = paddle.reshape(\n output, [batch * num_anchors, 5 + num_classes, h * w])\n output = paddle.transpose(output, (1, 0, 2))\n output = paddle.reshape(\n output, [5 + num_classes, batch * num_anchors * h * w])\n grid_x = paddle.linspace(0, w - 1, w)\n grid_x = paddle.tile(grid_x, [h, 1])\n grid_x = paddle.tile(grid_x, [batch * num_anchors, 1, 1])\n grid_x = paddle.reshape(grid_x, [batch * num_anchors * h * w]).cuda()\n grid_y = paddle.linspace(0, h - 1, h)\n grid_y = paddle.tile(grid_y, [w, 1]).t()\n grid_y = paddle.tile(grid_y, [batch * num_anchors, 1, 1])\n grid_y = paddle.reshape(grid_y, [batch * num_anchors * h * w]).cuda()\n sigmoid = nn.Sigmoid()\n xs = sigmoid(output[0]) + grid_x\n ys = sigmoid(output[1]) + grid_y\n anchor_w = paddle.to_tensor(anchors)\n anchor_w = paddle.reshape(anchor_w, [num_anchors, anchor_step])\n anchor_w = paddle.index_select(anchor_w, index=paddle.to_tensor(\n np.array([0]).astype('int32')), axis=1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:68-94"
+ },
+ "7339": {
+ "file_id": 534,
+ "content": "This code performs box regression by reshaping the output tensor, creating grids for x and y coordinates, applying sigmoid function to the output, adding grid coordinates to get refined box coordinates. It also converts anchor widths into a tensor for further processing.",
+ "type": "comment"
+ },
+ "7340": {
+ "file_id": 534,
+ "content": " anchor_h = paddle.to_tensor(anchors)\n anchor_h = paddle.reshape(anchor_h, [num_anchors, anchor_step])\n anchor_h = paddle.index_select(anchor_h, index=paddle.to_tensor(\n np.array([1]).astype('int32')), axis=1)\n anchor_w = paddle.tile(anchor_w, [batch, 1])\n anchor_w = paddle.tile(anchor_w, [1, 1, h * w])\n anchor_w = paddle.reshape(anchor_w, [batch * num_anchors * h * w]).cuda()\n anchor_h = paddle.tile(anchor_h, [batch, 1])\n anchor_h = paddle.tile(anchor_h, [1, 1, h * w])\n anchor_h = paddle.reshape(anchor_h, [batch * num_anchors * h * w]).cuda()\n ws = paddle.exp(output[2]) * anchor_w\n hs = paddle.exp(output[3]) * anchor_h\n det_confs = sigmoid(output[4])\n cls_confs = paddle.to_tensor(output[5:5 + num_classes], stop_gradient=True)\n cls_confs = paddle.transpose(cls_confs, [1, 0])\n s = nn.Softmax()\n cls_confs = paddle.to_tensor(s(cls_confs))\n cls_max_confs = paddle.max(cls_confs, axis=1)\n cls_max_ids = paddle.argmax(cls_confs, axis=1)\n cls_max_confs = paddle.reshape(cls_max_confs, [-1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:96-122"
+ },
+ "7341": {
+ "file_id": 534,
+ "content": "Code prepares output from a YOLOv2 object detection model, performing necessary reshaping and transformations to obtain the final detections and classifications. It computes the widths (ws) and heights (hs) of the bounding boxes based on the input feature maps, applies sigmoid activation to the fifth output channel for detection confidences, and converts the rest of the outputs to stop_gradient=True tensors for class predictions. The code then performs softmax normalization over class predictions and retrieves the maximum confidence and corresponding class IDs for each bounding box. Finally, it reshapes cls_max_confs to a 1D tensor.",
+ "type": "comment"
+ },
+ "7342": {
+ "file_id": 534,
+ "content": " cls_max_ids = paddle.reshape(cls_max_ids, [-1])\n sz_hw = h * w\n sz_hwa = sz_hw * num_anchors\n det_confs = convert2cpu(det_confs)\n cls_max_confs = convert2cpu(cls_max_confs)\n cls_max_ids = convert2cpu_long(cls_max_ids)\n xs = convert2cpu(xs)\n ys = convert2cpu(ys)\n ws = convert2cpu(ws)\n hs = convert2cpu(hs)\n if validation:\n cls_confs = convert2cpu(cls_confs.reshape([-1, num_classes]))\n for b in range(batch):\n boxes = []\n for cy in range(h):\n for cx in range(w):\n for i in range(num_anchors):\n ind = b * sz_hwa + i * sz_hw + cy * w + cx\n det_conf = det_confs[ind]\n if only_objectness:\n conf = det_confs[ind]\n else:\n conf = det_confs[ind] * cls_max_confs[ind]\n if conf > conf_thresh:\n bcx = xs[ind]\n bcy = ys[ind]\n bw = ws[ind]\n bh = hs[ind]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:123-153"
+ },
+ "7343": {
+ "file_id": 534,
+ "content": "The code extracts data from a PaddlePaddle tensor and converts it to CPU memory. It then reshapes the data, applies conditions, and stores box coordinates and confidences in lists for each batch. The extracted data is used to create bounding boxes for objects detected within the input image.",
+ "type": "comment"
+ },
+ "7344": {
+ "file_id": 534,
+ "content": " cls_max_conf = cls_max_confs[ind]\n cls_max_id = cls_max_ids[ind]\n box = [bcx / w, bcy / h, bw / w, bh / h,\n det_conf, cls_max_conf, cls_max_id]\n if (not only_objectness) and validation:\n for c in range(num_classes):\n tmp_conf = cls_confs[ind][c]\n if c != cls_max_id and det_confs[ind] * tmp_conf > conf_thresh:\n box.append(tmp_conf)\n box.append(c)\n boxes.append(box)\n all_boxes.append(boxes)\n return all_boxes\ndef bbox_iou(box1, box2, x1y1x2y2=True):\n if x1y1x2y2:\n mx = min(box1[0], box2[0])\n Mx = max(box1[2], box2[2])\n my = min(box1[1], box2[1])\n My = max(box1[3], box2[3])\n w1 = box1[2] - box1[0]\n h1 = box1[3] - box1[1]\n w2 = box2[2] - box2[0]\n h2 = box2[3] - box2[1]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:154-178"
+ },
+ "7345": {
+ "file_id": 534,
+ "content": "The function `yowo_utils.py` returns a list of boxes with their respective confidences and class ids for each box. It includes only objectness if only_objectness is True, otherwise it also includes the per-class confidences. The function `bbox_iou` calculates the intersection over union between two bounding boxes, considering x1y1x2y2 format where (x1, y1) is the top left corner and (x2, y2) is the bottom right corner.",
+ "type": "comment"
+ },
+ "7346": {
+ "file_id": 534,
+ "content": " else:\n mx = min(float(box1[0] - box1[2] / 2.0),\n float(box2[0] - box2[2] / 2.0))\n Mx = max(float(box1[0] + box1[2] / 2.0),\n float(box2[0] + box2[2] / 2.0))\n my = min(float(box1[1] - box1[3] / 2.0),\n float(box2[1] - box2[3] / 2.0))\n My = max(float(box1[1] + box1[3] / 2.0),\n float(box2[1] + box2[3] / 2.0))\n w1 = box1[2]\n h1 = box1[3]\n w2 = box2[2]\n h2 = box2[3]\n uw = Mx - mx\n uh = My - my\n cw = w1 + w2 - uw\n ch = h1 + h2 - uh\n carea = 0\n if cw <= 0 or ch <= 0:\n return paddle.to_tensor(0.0)\n area1 = w1 * h1\n area2 = w2 * h2\n carea = cw * ch\n uarea = area1 + area2 - carea\n return carea / uarea\ndef bbox_ious(boxes1, boxes2, x1y1x2y2=True):\n if x1y1x2y2:\n mx = paddle.min(boxes1[0], boxes2[0])\n Mx = paddle.max(boxes1[2], boxes2[2])\n my = paddle.min(boxes1[1], boxes2[1])\n My = paddle.max(boxes1[3], boxes2[3])\n w1 = boxes1[2] - boxes1[0]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:179-213"
+ },
+ "7347": {
+ "file_id": 534,
+ "content": "The code calculates the intersection-over-union (IOU) between two bounding boxes, which is commonly used in object detection tasks. It first finds the overlapping area by computing the minimum and maximum coordinates of the bounding boxes, then calculates the union of these boxes, and finally returns the intersection over union ratio. This helps in determining if the two bounding boxes represent the same object or not.",
+ "type": "comment"
+ },
+ "7348": {
+ "file_id": 534,
+ "content": " h1 = boxes1[3] - boxes1[1]\n w2 = boxes2[2] - boxes2[0]\n h2 = boxes2[3] - boxes2[1]\n else:\n mx = paddle.min(paddle.stack(\n [boxes1[0] - boxes1[2] / 2.0, boxes2[0] - boxes2[2] / 2.0], axis=0), axis=0)\n Mx = paddle.max(paddle.stack(\n [boxes1[0] + boxes1[2] / 2.0, boxes2[0] + boxes2[2] / 2.0], axis=0), axis=0)\n my = paddle.min(paddle.stack(\n [boxes1[1] - boxes1[3] / 2.0, boxes2[1] - boxes2[3] / 2.0], axis=0), axis=0)\n My = paddle.max(paddle.stack(\n [boxes1[1] + boxes1[3] / 2.0, boxes2[1] + boxes2[3] / 2.0], axis=0), axis=0)\n w1 = boxes1[2]\n h1 = boxes1[3]\n w2 = boxes2[2]\n h2 = boxes2[3]\n uw = Mx - mx\n uh = My - my\n cw = w1 + w2 - uw\n ch = h1 + h2 - uh\n mask = paddle.cast(cw <= 0, dtype=\"int32\") + \\\n paddle.cast(ch <= 0, dtype=\"int32\") > 0\n area1 = w1 * h1\n area2 = w2 * h2\n carea = cw * ch\n carea[mask] = 0\n uarea = area1 + area2 - carea\n return carea / uarea",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:214-241"
+ },
+ "7349": {
+ "file_id": 534,
+ "content": "This code calculates the intersection over union (IoU) between two bounding boxes. It first checks if both boxes have valid dimensions, then computes the coordinates of each box and their widths and heights. If the boxes overlap, it calculates the intersection area and union area of the bounding boxes, taking into account non-overlapping areas by setting them to 0 in the case of non-intersection. Finally, it returns the IoU as the intersection area divided by the union area.",
+ "type": "comment"
+ },
+ "7350": {
+ "file_id": 534,
+ "content": "# this function works for building the groud truth\ndef build_targets(pred_boxes, target, anchors, num_anchors, num_classes, nH, nW, noobject_scale, object_scale,\n sil_thresh):\n # nH, nW here are number of grids in y and x directions (7, 7 here)\n nB = target.shape[0] # batch size\n nA = num_anchors # 5 for our case\n nC = num_classes\n anchor_step = len(anchors) // num_anchors\n conf_mask = paddle.ones([nB, nA, nH, nW]) * noobject_scale\n coord_mask = paddle.zeros([nB, nA, nH, nW])\n cls_mask = paddle.zeros([nB, nA, nH, nW])\n tx = paddle.zeros([nB, nA, nH, nW])\n ty = paddle.zeros([nB, nA, nH, nW])\n tw = paddle.zeros([nB, nA, nH, nW])\n th = paddle.zeros([nB, nA, nH, nW])\n tconf = paddle.zeros([nB, nA, nH, nW])\n tcls = paddle.zeros([nB, nA, nH, nW])\n # for each grid there are nA anchors\n # nAnchors is the number of anchor for one image\n nAnchors = nA * nH * nW\n nPixels = nH * nW\n # for each image\n for b in xrange(nB):\n # get all anchor boxes in one image",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:244-268"
+ },
+ "7351": {
+ "file_id": 534,
+ "content": "This function builds ground truth targets for each grid in the image. It iterates over each batch, anchor, height, and width to create confidence, coordinate, and class masks, as well as target coordinates and classes for each anchor box. The targets are then concatenated into a single tensor.",
+ "type": "comment"
+ },
+ "7352": {
+ "file_id": 534,
+ "content": " # (4 * nAnchors)\n cur_pred_boxes = pred_boxes[b * nAnchors:(b + 1) * nAnchors].t()\n # initialize iou score for each anchor\n cur_ious = paddle.zeros([nAnchors])\n for t in xrange(50):\n # for each anchor 4 coordinate parameters, already in the coordinate system for the whole image\n # this loop is for anchors in each image\n # for each anchor 5 parameters are available (class, x, y, w, h)\n if target[b][t * 5 + 1] == 0:\n break\n gx = target[b][t * 5 + 1] * nW\n gy = target[b][t * 5 + 2] * nH\n gw = target[b][t * 5 + 3] * nW\n gh = target[b][t * 5 + 4] * nH\n # groud truth boxes\n cur_gt_boxes = paddle.tile(paddle.to_tensor(\n [gx, gy, gw, gh], dtype='float32').t(), [nAnchors, 1]).t()\n # bbox_ious is the iou value between orediction and groud truth\n cur_ious = paddle.max(\n paddle.stack([cur_ious, bbox_ious(cur_pred_boxes, cur_gt_boxes, x1y1x2y2=False)], axis=0), axis=0)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:269-288"
+ },
+ "7353": {
+ "file_id": 534,
+ "content": "This code calculates the IoU (Intersection over Union) between predicted and ground truth boxes for each anchor. It uses a loop to iterate through 50 time steps, breaks if no target is available at the current time step, and calculates the bbox_ious function using cur_pred_boxes and cur_gt_boxes for IoU calculation. The highest IoU value is stored in cur_ious for each anchor.",
+ "type": "comment"
+ },
+ "7354": {
+ "file_id": 534,
+ "content": " # if iou > a given threshold, it is seen as it includes an object\n # conf_mask[b][cur_ious>sil_thresh] = 0\n conf_mask_t = paddle.reshape(conf_mask, [nB, -1])\n conf_mask_t[b, cur_ious > sil_thresh] = 0\n conf_mask_tt = paddle.reshape(conf_mask_t[b], [nA, nH, nW])\n conf_mask[b] = conf_mask_tt\n # number of ground truth\n nGT = 0\n nCorrect = 0\n for b in xrange(nB):\n # anchors for one batch (at least batch size, and for some specific classes, there might exist more than one anchor)\n for t in xrange(50):\n if target[b][t * 5 + 1] == 0:\n break\n nGT = nGT + 1\n best_iou = 0.0\n best_n = -1\n min_dist = 10000\n # the values saved in target is ratios\n # times by the width and height of the output feature maps nW and nH\n gx = target[b][t * 5 + 1] * nW\n gy = target[b][t * 5 + 2] * nH\n gi = int(gx)\n gj = int(gy)\n gw = target[b][t * 5 + 3] * nW",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:289-315"
+ },
+ "7355": {
+ "file_id": 534,
+ "content": "This code calculates the IoU (Intersection over Union) between predicted bounding boxes and ground truth bounding boxes, and applies a mask to the confidences based on this IoU. It also counts the number of ground truth instances (nGT) and correct detections (nCorrect). The target values are ratios multiplied by the width and height of the output feature maps.",
+ "type": "comment"
+ },
+ "7356": {
+ "file_id": 534,
+ "content": " gh = target[b][t * 5 + 4] * nH\n gt_box = [0, 0, gw, gh]\n for n in xrange(nA):\n # get anchor parameters (2 values)\n aw = anchors[anchor_step * n]\n ah = anchors[anchor_step * n + 1]\n anchor_box = [0, 0, aw, ah]\n # only consider the size (width and height) of the anchor box\n iou = bbox_iou(anchor_box, gt_box, x1y1x2y2=False)\n # get the best anchor form with the highest iou\n if iou > best_iou:\n best_iou = iou\n best_n = n\n # then we determine the parameters for an anchor (4 values together)\n gt_box = [gx, gy, gw, gh]\n # find corresponding prediction box\n pred_box = pred_boxes[b * nAnchors +\n best_n * nPixels + gj * nW + gi]\n # only consider the best anchor box, for each image\n coord_mask[b, best_n, gj, gi] = 1\n cls_mask[b, best_n, gj, gi] = 1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:316-338"
+ },
+ "7357": {
+ "file_id": 534,
+ "content": "This code iterates over anchor boxes, calculates IoU with ground truth boxes and selects the best matching one. It then updates the corresponding prediction box for that image and marks it as valid in coord_mask and cls_mask matrices.",
+ "type": "comment"
+ },
+ "7358": {
+ "file_id": 534,
+ "content": " # in this cell of the output feature map, there exists an object\n conf_mask[b, best_n, gj, gi] = object_scale\n tx[b, best_n, gj, gi] = paddle.cast(\n target[b][t * 5 + 1] * nW - gi, dtype='float32')\n ty[b, best_n, gj, gi] = paddle.cast(\n target[b][t * 5 + 2] * nH - gj, dtype='float32')\n tw[b, best_n, gj, gi] = math.log(\n gw / anchors[anchor_step * best_n])\n th[b, best_n, gj, gi] = math.log(\n gh / anchors[anchor_step * best_n + 1])\n iou = bbox_iou(gt_box, pred_box, x1y1x2y2=False) # best_iou\n # confidence equals to iou of the corresponding anchor\n tconf[b, best_n, gj, gi] = paddle.cast(iou, dtype='float32')\n tcls[b, best_n, gj, gi] = paddle.cast(\n target[b][t * 5], dtype='float32')\n # if ious larger than 0.5, we justify it as a correct prediction\n if iou > 0.5:\n nCorrect = nCorrect + 1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:340-357"
+ },
+ "7359": {
+ "file_id": 534,
+ "content": "The code calculates object position, size, and confidence for each detected object. It then counts the number of correct detections by checking if the IOU is greater than 0.5.",
+ "type": "comment"
+ },
+ "7360": {
+ "file_id": 534,
+ "content": " # true values are returned\n return nGT, nCorrect, coord_mask, conf_mask, cls_mask, tx, ty, tw, th, tconf, tcls",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/localizers/yowo_utils.py:358-359"
+ },
+ "7361": {
+ "file_id": 534,
+ "content": "The function returns the ground truth values (nGT), correct predictions (nCorrect), and corresponding masks for coordinates, confidence, and class labels, as well as the transformation parameters for translation, width, and height.",
+ "type": "comment"
+ },
+ "7362": {
+ "file_id": 535,
+ "content": "/paddlevideo/modeling/framework/multimodal/__init__.py",
+ "type": "filepath"
+ },
+ "7363": {
+ "file_id": 535,
+ "content": "This code file is part of the PaddleVideo library and contains the initialization, base class (BaseMultimodal), and a specific multimodal model (ActBert). It also mentions licensing information and a link to access it. The __all__ variable lists the available modules for importing from this file.",
+ "type": "summary"
+ },
+ "7364": {
+ "file_id": 535,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom .base import BaseMultimodal\nfrom .actbert import ActBert\n__all__ = ['BaseMultimodal', 'ActBert']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/__init__.py:1-16"
+ },
+ "7365": {
+ "file_id": 535,
+ "content": "This code file is part of the PaddleVideo library and contains the initialization, base class (BaseMultimodal), and a specific multimodal model (ActBert). It also mentions licensing information and a link to access it. The __all__ variable lists the available modules for importing from this file.",
+ "type": "comment"
+ },
+ "7366": {
+ "file_id": 536,
+ "content": "/paddlevideo/modeling/framework/multimodal/actbert.py",
+ "type": "filepath"
+ },
+ "7367": {
+ "file_id": 536,
+ "content": "The code introduces the ActBert model for multimodal tasks, including training and validation steps. It utilizes a backbone function for predictions with text, video, and action scores along with sequence relationship scores. The infer_step is yet to be implemented.",
+ "type": "summary"
+ },
+ "7368": {
+ "file_id": 536,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import MULTIMODAL\nfrom .base import BaseMultimodal\nimport paddle\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@MULTIMODAL.register()\nclass ActBert(BaseMultimodal):\n \"\"\"ActBert model framework.\"\"\"\n def forward_net(self, text_ids, action_feat, image_feat, image_loc,\n token_type_ids, text_mask, image_mask, action_mask):\n pred = self.backbone(text_ids, action_feat, image_feat, image_loc,\n token_type_ids, text_mask, image_mask, action_mask)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/actbert.py:1-27"
+ },
+ "7369": {
+ "file_id": 536,
+ "content": "This code snippet defines the ActBert class, which is a multimodal model framework. It registers the model under MULTIMODAL in the registry and includes a forward_net method for processing text, action, and image data. The self.backbone function is used to make predictions based on this data. The code also includes import statements, variable definitions, and a get_logger function call for logging purposes.",
+ "type": "comment"
+ },
+ "7370": {
+ "file_id": 536,
+ "content": " return pred\n def train_step(self, data_batch):\n \"\"\"For ActBert Dataset. Define how the model is going to train, from input to output.\n \"\"\"\n text_ids, action_feat, image_feat, image_loc, \\\n token_type_ids, text_mask, image_mask, action_mask, \\\n text_labels, action_label, next_sentence_label, image_label, image_target = data_batch\n loss_metrics = dict()\n pred = self.backbone(text_ids, action_feat, image_feat, image_loc,\n token_type_ids, text_mask, image_mask, action_mask)\n prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score = pred\n total_loss = self.loss(prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score, \\\n text_labels, image_label, image_target, action_label, next_sentence_label)\n loss_metrics['loss'] = paddle.mean(total_loss)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"For ActBert Dataset. Define how the model is going to val, from input to output.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/actbert.py:28-46"
+ },
+ "7371": {
+ "file_id": 536,
+ "content": "This code defines a train_step and val_step for ActBert Dataset. In the train_step, it takes input data (text_ids, action_feat, image_feat, etc.), passes them through the backbone model to get prediction scores, calculates loss, and returns a loss metric dictionary. The val_step does not appear to have any additional functionality.",
+ "type": "comment"
+ },
+ "7372": {
+ "file_id": 536,
+ "content": " \"\"\"\n return self.train_step(data_batch)\n def test_step(self, data_batch):\n \"\"\"For MSR-VTT Dataset. Define how the model is going to test, from input to output.\"\"\"\n text_ids, action_feat, image_feat, image_loc, token_type_ids, text_mask, image_mask, action_mask = data_batch[:\n -1]\n action_feat = action_feat.squeeze(0)\n image_feat = image_feat.squeeze(0)\n image_loc = image_loc.squeeze(0)\n image_mask = image_mask.squeeze(0)\n action_mask = action_mask.squeeze(0)\n prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score = self.forward_net(text_ids, \\\n action_feat, image_feat, image_loc, token_type_ids, text_mask, image_mask, action_mask)\n return prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score\n def infer_step(self, data_batch):\n pass",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/actbert.py:47-64"
+ },
+ "7373": {
+ "file_id": 536,
+ "content": "The code defines a model that takes in multiple inputs like text_ids, action_feat, image_feat, and more. It performs testing with the test_step() function and returns prediction scores for text, video, and action, along with the sequence relationship score. The infer_step() function is not implemented yet.",
+ "type": "comment"
+ },
+ "7374": {
+ "file_id": 537,
+ "content": "/paddlevideo/modeling/framework/multimodal/base.py",
+ "type": "filepath"
+ },
+ "7375": {
+ "file_id": 537,
+ "content": "This code defines a base class for multimodal models in PaddleVideo, requiring subclasses to override train_step, valid_step, test_step, and define abstract methods for validating, testing, and inference steps.",
+ "type": "summary"
+ },
+ "7376": {
+ "file_id": 537,
+ "content": "from abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseMultimodal(nn.Layer):\n \"\"\"Base class for Multimodal.\n All Multimodal model should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Head to process feature.\n loss(dict): Loss function.\n \"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__()\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/base.py:1-32"
+ },
+ "7377": {
+ "file_id": 537,
+ "content": "This code defines a base class for multimodal models in PaddleVideo. It requires subclasses to override train_step, valid_step, and test_step methods. The constructor accepts optional backbone, head, and loss parameters which are built using the builder module. If provided, the backbone is initialized with its init_weights method.",
+ "type": "comment"
+ },
+ "7378": {
+ "file_id": 537,
+ "content": " if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n if loss is not None:\n self.loss = builder.build_loss(loss)\n else:\n self.loss = None\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/base.py:33-63"
+ },
+ "7379": {
+ "file_id": 537,
+ "content": "The code defines a base class for multimodal models, with an initializer to set up the head and loss functions. The `forward` method selects the appropriate step function based on the given mode (train, valid, test, or infer). The abstract `train_step` method must be implemented in subclasses for training.",
+ "type": "comment"
+ },
+ "7380": {
+ "file_id": 537,
+ "content": " @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/multimodal/base.py:65-81"
+ },
+ "7381": {
+ "file_id": 537,
+ "content": "This code defines three abstract methods: val_step, test_step, and infer_step. These methods represent validating, testing, and inference steps respectively. The methods are not yet implemented and will need to be filled by subclasses according to the specific requirements of the model being developed.",
+ "type": "comment"
+ },
+ "7382": {
+ "file_id": 538,
+ "content": "/paddlevideo/modeling/framework/partitioners/__init__.py",
+ "type": "filepath"
+ },
+ "7383": {
+ "file_id": 538,
+ "content": "This code is the initialization file of PaddleVideo's partitioner module. It imports base and TransNetV2Partitioner classes, and declares them as part of the public interface (__all__). The license and copyright information are also included.",
+ "type": "summary"
+ },
+ "7384": {
+ "file_id": 538,
+ "content": "# copyright (c) 2020 paddlepaddle authors. all rights reserved.\n#\n# licensed under the apache license, version 2.0 (the \"license\"\n# you may not use this file except in compliance with the license.\n# you may obtain a copy of the license at\n#\n# http://www.apache.org/licenses/license-2.0\n#\n# unless required by applicable law or agreed to in writing, software\n# distributed under the license is distributed on an \"as is\" basis,\n# without warranties or conditions of any kind, either express or implied.\n# see the license for the specific language governing permissions and\n# limitations under the license.\nfrom .base import BasePartitioner\nfrom .transnetv2_partitioner import TransNetV2Partitioner\n__all__ = ['BasePartitioner', 'TransNetV2Partitioner']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/__init__.py:1-18"
+ },
+ "7385": {
+ "file_id": 538,
+ "content": "This code is the initialization file of PaddleVideo's partitioner module. It imports base and TransNetV2Partitioner classes, and declares them as part of the public interface (__all__). The license and copyright information are also included.",
+ "type": "comment"
+ },
+ "7386": {
+ "file_id": 539,
+ "content": "/paddlevideo/modeling/framework/partitioners/base.py",
+ "type": "filepath"
+ },
+ "7387": {
+ "file_id": 539,
+ "content": "This Python class is part of PaddleVideo's modeling framework, serving as a base for partitioners and initializing partitioned models. It includes backbone and head components initialization, optional weight initialization, and defines a forward function. A base class for model partitioners is also defined with methods for train, validate, test, and infer steps, leaving the actual implementation to subclasses.",
+ "type": "summary"
+ },
+ "7388": {
+ "file_id": 539,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom abc import abstractmethod\nimport paddle.nn as nn\nfrom ... import builder\nclass BasePartitioner(nn.Layer):\n \"\"\"Base class for Partition.\n All partitioner should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, define your train step.\n - Methods:``valid_step``, define your valid step, always the same as train_step.\n - Methods:``test_step``, define your test step.\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/base.py:1-27"
+ },
+ "7389": {
+ "file_id": 539,
+ "content": "This code is a Python class for base partitioner in PaddleVideo's modeling framework. It is an abstract class that serves as the foundation for all partitioners and requires its subclasses to define specific methods like train_step, valid_step, and test_step.",
+ "type": "comment"
+ },
+ "7390": {
+ "file_id": 539,
+ "content": " def __init__(self, backbone=None, head=None):\n super().__init__()\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n def init_weights(self):\n \"\"\"Initialize the model network weights. \"\"\"\n if getattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n pass\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/base.py:28-55"
+ },
+ "7391": {
+ "file_id": 539,
+ "content": "This code initializes a partitioned model by building backbone and head components. It also includes an option to initialize weights for these components, and provides a forward function defining the model's execution path depending on the provided mode.",
+ "type": "comment"
+ },
+ "7392": {
+ "file_id": 539,
+ "content": " \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step. input_data_batch -> loss_metric\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating setp. input_data_batch -> loss_metric\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Tets setp. to get acc in test data. input_data_batch -> output\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/base.py:56-84"
+ },
+ "7393": {
+ "file_id": 539,
+ "content": "The code defines a base class for model partitioners, which includes methods for train, validate, test, and infer steps. Each step takes a data batch as input and returns either a loss metric or the output. If an unsupported mode is provided, it raises a NotImplementedError. The actual implementation of these steps is left to subclasses.",
+ "type": "comment"
+ },
+ "7394": {
+ "file_id": 540,
+ "content": "/paddlevideo/modeling/framework/partitioners/transnetv2_partitioner.py",
+ "type": "filepath"
+ },
+ "7395": {
+ "file_id": 540,
+ "content": "TransNetV2 Partitioner in PaddleVideo framework defines a model partitioner, includes forwarding methods for image processing and computing loss metrics. It has three methods: \"loss_metrics\", \"test_step\", and \"infer_step\" for training, testing, and inference phases respectively.",
+ "type": "summary"
+ },
+ "7396": {
+ "file_id": 540,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import PARTITIONERS\nfrom .base import BasePartitioner\nimport paddle\n@PARTITIONERS.register()\nclass TransNetV2Partitioner(BasePartitioner):\n \"\"\"TransNetV2 Partitioner framework\n \"\"\"\n def forward_net(self, imgs):\n one_hot_pred = self.backbone(imgs)\n return one_hot_pred\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n frame_sequence = data_batch[0]\n one_hot_gt, many_hot_gt = data_batch[1:]\n one_hot_pred = self.forward_net(frame_sequence)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/transnetv2_partitioner.py:1-32"
+ },
+ "7397": {
+ "file_id": 540,
+ "content": "TransNetV2 Partitioner class for PaddleVideo framework, with forward_net and train_step methods for image processing and model training.",
+ "type": "comment"
+ },
+ "7398": {
+ "file_id": 540,
+ "content": " dict_ = {}\n if isinstance(one_hot_pred, tuple):\n one_hot_pred, dict_ = one_hot_pred\n many_hot_pred = dict_.get(\"many_hot\", None)\n comb_reg_loss = dict_.get(\"comb_reg_loss\", None)\n loss_metrics = self.head.loss(one_hot_pred, one_hot_gt,\n many_hot_pred, many_hot_gt,\n reg_losses={\"comb_reg\": comb_reg_loss})\n return loss_metrics\n def val_step(self, data_batch):\n frame_sequence = data_batch[0]\n one_hot_gt, many_hot_gt = data_batch[1:]\n one_hot_pred = self.forward_net(frame_sequence)\n dict_ = {}\n if isinstance(one_hot_pred, tuple):\n one_hot_pred, dict_ = one_hot_pred\n many_hot_pred = dict_.get(\"many_hot\", None)\n comb_reg_loss = dict_.get(\"comb_reg_loss\", None)\n loss_metrics = self.head.loss(one_hot_pred, one_hot_gt,\n many_hot_pred, many_hot_gt,\n reg_losses={\"comb_reg\": comb_reg_loss})",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/transnetv2_partitioner.py:33-54"
+ },
+ "7399": {
+ "file_id": 540,
+ "content": "Code defines a model partitioner for TransNetV2. It returns loss metrics from the validation step by forwarding frame sequences through the model, extracting one-hot and many-hot predictions and ground truths, and applying losses based on provided dictionaries.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/74.json b/docs/data/74.json
new file mode 100644
index 000000000..ba8ec7f64
--- /dev/null
+++ b/docs/data/74.json
@@ -0,0 +1,541 @@
+{
+ "7400": {
+ "file_id": 540,
+ "content": " return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n # NOTE: (shipping) when testing, the net won't call head.loss, we deal with the test processing in /paddlevideo/metrics\n frame_sequence = data_batch[0]\n one_hot_pred = self.forward_net(frame_sequence)\n return one_hot_pred\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n frame_sequence = data_batch[0]\n one_hot_pred = self.forward_net(frame_sequence)\n return one_hot_pred",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/partitioners/transnetv2_partitioner.py:55-68"
+ },
+ "7401": {
+ "file_id": 540,
+ "content": "The code defines three methods: \"loss_metrics\" returns loss and metrics for training, \"test_step\" performs testing by forwarding frames through the net without calculating loss, and \"infer_step\" also performs testing with forwarding frames but without specifying if it's for a test or inference phase.",
+ "type": "comment"
+ },
+ "7402": {
+ "file_id": 541,
+ "content": "/paddlevideo/modeling/framework/recognizers/__init__.py",
+ "type": "filepath"
+ },
+ "7403": {
+ "file_id": 541,
+ "content": "This code file in the PaddleVideo library imports various recognizer classes for video recognition tasks, including 1D, 2D, 3D, transformer-based, GCN, MRI, and MoViNet frame-based recognizers. These models are used for action recognition and motion estimation tasks.",
+ "type": "summary"
+ },
+ "7404": {
+ "file_id": 541,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom .base import BaseRecognizer\nfrom .recognizer1d import Recognizer1D, RecognizerAction\nfrom .recognizer2d import Recognizer2D\nfrom .recognizer3d import Recognizer3D\nfrom .recognizer_transformer import RecognizerTransformer\nfrom .recognizer_gcn import RecognizerGCN\nfrom .recognizerMRI import RecognizerMRI\nfrom .recognizer3dMRI import Recognizer3DMRI\nfrom .recognizer_transformer_MRI import RecognizerTransformer_MRI\nfrom .recognizer_movinet_frame import MoViNetRecognizerFrame\nfrom .recognizerDistillation import RecognizerDistillation",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/__init__.py:1-23"
+ },
+ "7405": {
+ "file_id": 541,
+ "content": "This code file imports various recognizer classes from different modules within the PaddleVideo framework for video recognition tasks. These recognizers include 1D, 2D, 3D, transformer-based, GCN, MRI, 3D MRI, and MoViNet frame-based recognizers, as well as a distillation-based recognizer. Each recognizer is designed for specific types of recognition tasks in video analysis.",
+ "type": "comment"
+ },
+ "7406": {
+ "file_id": 541,
+ "content": "__all__ = [\n 'BaseRecognizer', 'Recognizer1D', 'Recognizer2D', 'Recognizer3D',\n 'RecognizerTransformer', 'RecognizerGCN', 'RecognizerMRI',\n 'Recognizer3DMRI', 'RecognizerTransformer_MRI', 'MoViNetRecognizerFrame',\n 'RecognizerAction', 'RecognizerDistillation'\n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/__init__.py:25-30"
+ },
+ "7407": {
+ "file_id": 541,
+ "content": "This code snippet in the PaddleVideo library defines various recognizer models including BaseRecognizer, Recognizer1D, Recognizer2D, and more. These classes are used for video recognition tasks like action recognition and motion estimation.",
+ "type": "comment"
+ },
+ "7408": {
+ "file_id": 542,
+ "content": "/paddlevideo/modeling/framework/recognizers/base.py",
+ "type": "filepath"
+ },
+ "7409": {
+ "file_id": 542,
+ "content": "This code initializes a model's head, defines modes of operation, and provides abstract methods for training, validation, and inference steps. It serves as a base class for recognizer models in PaddleVideo and raises NotImplementedError if subclasses don't implement these steps.",
+ "type": "summary"
+ },
+ "7410": {
+ "file_id": 542,
+ "content": "from abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseRecognizer(nn.Layer):\n \"\"\"Base class for recognizers.\n All recognizers should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Classification head to process feature.\n \"\"\"\n def __init__(self, backbone=None, head=None, runtime_cfg=None):\n super().__init__()\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/base.py:1-33"
+ },
+ "7411": {
+ "file_id": 542,
+ "content": "Base class for recognizers: Subclasses should override train_step, valid_step, and test_step methods. Builds backbone and head using builder if provided.",
+ "type": "comment"
+ },
+ "7412": {
+ "file_id": 542,
+ "content": " self.head.init_weights()\n else:\n self.head = None\n # Settings when the model is running,\n # such as 'avg_type'\n self.runtime_cfg = runtime_cfg\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/base.py:34-66"
+ },
+ "7413": {
+ "file_id": 542,
+ "content": "This code initializes a model's head, defines the mode of operation (train, valid, test, infer), and provides abstract methods for training and validation steps. If the mode is 'infer', it saves the inference model.",
+ "type": "comment"
+ },
+ "7414": {
+ "file_id": 542,
+ "content": " \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/base.py:67-81"
+ },
+ "7415": {
+ "file_id": 542,
+ "content": "This code snippet in PaddleVideo defines abstract methods for validating, testing, and inferring steps. It serves as a base class for recognizer models and expects subclasses to implement these methods. The NotImplementedError is raised to ensure that subclasses provide their own implementation for these steps.",
+ "type": "comment"
+ },
+ "7416": {
+ "file_id": 543,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer1d.py",
+ "type": "filepath"
+ },
+ "7417": {
+ "file_id": 543,
+ "content": "The code defines a 1D recognizer model in PaddleVideo, processing both image and audio data for training, validation, testing, and inference. It includes forward pass, loss computation, metrics calculations and handles RGB and audio data batches.",
+ "type": "summary"
+ },
+ "7418": {
+ "file_id": 543,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\n@RECOGNIZERS.register()\nclass Recognizer1D(BaseRecognizer):\n \"\"\"1D recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n lstm_logit, lstm_output = self.head(imgs)\n return lstm_logit, lstm_output\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n rgb_data, rgb_len, rgb_mask, audio_data, audio_len, audio_mask, labels = data_batch",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer1d.py:1-29"
+ },
+ "7419": {
+ "file_id": 543,
+ "content": "This code defines a 1D recognizer model framework in PaddleVideo. It includes the forward_net function to define how the model trains from input to output and the train_step function for training steps. The data batch contains rgb_data, rgb_len, rgb_mask, audio_data, audio_len, audio_mask, and labels.",
+ "type": "comment"
+ },
+ "7420": {
+ "file_id": 543,
+ "content": " imgs = [(rgb_data, rgb_len, rgb_mask),\n (audio_data, audio_len, audio_mask)]\n # call forward\n lstm_logit, lstm_output = self.forward_net(imgs)\n loss = self.head.loss(lstm_logit, labels)\n hit_at_one, perr, gap = self.head.metric(lstm_output, labels)\n loss_metrics = dict()\n loss_metrics['loss'] = loss\n loss_metrics['hit_at_one'] = hit_at_one\n loss_metrics['perr'] = perr\n loss_metrics['gap'] = gap\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n return self.train_step(data_batch)\n def test_step(self, data_batch):\n \"\"\"Testing setp.\n \"\"\"\n return self.train_step(data_batch)\n def infer_step(self, data_batch):\n \"\"\"Infering setp.\n \"\"\"\n rgb_data, rgb_len, rgb_mask, audio_data, audio_len, audio_mask = data_batch\n imgs = [(rgb_data, rgb_len, rgb_mask),\n (audio_data, audio_len, audio_mask)]\n # call forward",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer1d.py:30-61"
+ },
+ "7421": {
+ "file_id": 543,
+ "content": "The code defines a recognizer1d model that processes both image and audio data. It includes methods for forward pass, validation, testing, and inference steps. In the forward pass, it takes input images and calculates logits and output from the LSTM network. The loss is then computed based on these logits and labels, and metrics such as hit_at_one, perr, and gap are calculated using the output and labels. The validation and testing steps perform similar calculations to those in the training step. In the inference step, only image and audio data are processed to produce output for each input.",
+ "type": "comment"
+ },
+ "7422": {
+ "file_id": 543,
+ "content": " lstm_logit, _ = self.forward_net(imgs)\n return lstm_logit\n@RECOGNIZERS.register()\nclass RecognizerAction(BaseRecognizer):\n \"\"\"1D recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n lstm_logit, lstm_output = self.head(imgs)\n return lstm_logit, lstm_output\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n rgb_data, rgb_len, rgb_mask, audio_data, audio_len, audio_mask, labels, labels_iou = data_batch\n imgs = [(rgb_data, rgb_len, rgb_mask),\n (audio_data, audio_len, audio_mask)]\n # call forward\n output_logit, output_iou = self.forward_net(imgs)\n loss = self.head.loss(output_logit, output_iou, labels, labels_iou)\n top1, top5 = self.head.metric(output_logit, labels)\n loss_metrics = dict()\n loss_metrics['loss'] = loss\n loss_metrics['top1'] = top1\n loss_metrics['top5'] = top5\n return loss_metrics",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer1d.py:62-91"
+ },
+ "7423": {
+ "file_id": 543,
+ "content": "This code defines a 1D recognizer model framework, which includes a forward_net function to define how the model trains from input to output and a train_step function for the training process. It takes in data batches, including both RGB and audio data, and outputs loss metrics including loss, top1, and top5.",
+ "type": "comment"
+ },
+ "7424": {
+ "file_id": 543,
+ "content": " def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n return self.train_step(data_batch)\n def test_step(self, data_batch):\n \"\"\"Testing setp.\n \"\"\"\n return self.train_step(data_batch)\n def infer_step(self, data_batch):\n \"\"\"Infering setp.\n \"\"\"\n rgb_data, rgb_len, rgb_mask, audio_data, audio_len, audio_mask = data_batch\n imgs = [(rgb_data, rgb_len, rgb_mask),\n (audio_data, audio_len, audio_mask)]\n # call forward\n output_logit, output_iou = self.forward_net(imgs)\n return output_logit, output_iou",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer1d.py:93-111"
+ },
+ "7425": {
+ "file_id": 543,
+ "content": "The code contains three methods: `val_step`, `test_step`, and `infer_step`. These steps perform validating, testing, and inference, respectively. In all three cases, the data batch is passed to the `train_step` method, suggesting a shared implementation between these steps. The `infer_step` specifically expects certain types of data: RGB data with length and mask, as well as audio data with its respective length and mask, in a tuple format. It then processes this data using `forward_net`, returning output logits and IOU values.",
+ "type": "comment"
+ },
+ "7426": {
+ "file_id": 544,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer2d.py",
+ "type": "filepath"
+ },
+ "7427": {
+ "file_id": 544,
+ "content": "Recognizer2D is a 2D model in PaddleVideo for video analysis. It requires num_segs and includes functions for processing, training/validating, and testing the model. The Recognizer2D class defines forward_net and infer_step methods for classification scores.",
+ "type": "summary"
+ },
+ "7428": {
+ "file_id": 544,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nimport paddle\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass Recognizer2D(BaseRecognizer):\n \"\"\"2D recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n # NOTE: As the num_segs is an attribute of dataset phase, and didn't pass to build_head phase, should obtain it from imgs(paddle.Tensor) now, then call self.head method.\n num_segs = imgs.shape[\n 1] # imgs.shape=[N,T,C,H,W], for most commonly case",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer2d.py:1-27"
+ },
+ "7429": {
+ "file_id": 544,
+ "content": "Recognizer2D is a 2D recognizer model framework in PaddleVideo, inheriting from BaseRecognizer. It requires the number of segments (num_segs) which can be obtained from the shape of input images. The forward_net function performs image recognition using this model framework.",
+ "type": "comment"
+ },
+ "7430": {
+ "file_id": 544,
+ "content": " imgs = paddle.reshape_(imgs, [-1] + list(imgs.shape[2:]))\n if self.backbone is not None:\n feature = self.backbone(imgs)\n else:\n feature = imgs\n if self.head is not None:\n cls_score = self.head(feature, num_segs)\n else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n loss_metrics = self.head.loss(cls_score, labels)\n return loss_metrics\n def val_step(self, data_batch):\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n loss_metrics = self.head.loss(cls_score, labels, valid_mode=True)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n # NOTE: (s",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer2d.py:28-60"
+ },
+ "7431": {
+ "file_id": 544,
+ "content": "The code defines a recognizer2D model for video analysis. It consists of three main parts: the forward_net function that processes images, the train_step function for training the model using input data, and the val_step and test_step functions for validating and testing the trained model respectively. The forward_net function reshapes the images and passes them through a backbone network if one is defined, then to a head network if one is defined as well. It returns the classification scores. The train_step calculates the loss metrics using the provided labels, while the val_step does the same but in validation mode. The test_step computes the loss metrics without providing any labels.",
+ "type": "comment"
+ },
+ "7432": {
+ "file_id": 544,
+ "content": "hipping) when testing, the net won't call head.loss, we deal with the test processing in /paddlevideo/metrics\n imgs = data_batch[0]\n cls_score = self.forward_net(imgs)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n cls_score = self.forward_net(imgs)\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer2d.py:60-69"
+ },
+ "7433": {
+ "file_id": 544,
+ "content": "The code defines a Recognizer2D class with two methods: forward_net and infer_step. The forward_net method takes in images (imgs) and returns the classification scores (cls_score). The infer_step method is used for testing and follows the same process as forward_net to return cls_score.",
+ "type": "comment"
+ },
+ "7434": {
+ "file_id": 545,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer3d.py",
+ "type": "filepath"
+ },
+ "7435": {
+ "file_id": 545,
+ "content": "Recognizer3D defines a model framework with forward_net, train/val_step for training and validation in recognition models. It handles image processing based on backbone and calculates loss metrics. The code defines test_step and infer_step methods for testing and inference.",
+ "type": "summary"
+ },
+ "7436": {
+ "file_id": 545,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass Recognizer3D(BaseRecognizer):\n \"\"\"3D Recognizer model framework.\n \"\"\"\n def forward_net(self, imgs):\n \"\"\"Define how the model is going to run, from input to output.\n \"\"\"\n feature = self.backbone(imgs)\n cls_score = self.head(feature)\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Training step.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer3d.py:1-33"
+ },
+ "7437": {
+ "file_id": 545,
+ "content": "Recognizer3D is a 3D Recognizer model framework, which defines how the model runs from input to output. It includes forward_net method for model execution and train_step method for training.",
+ "type": "comment"
+ },
+ "7438": {
+ "file_id": 545,
+ "content": " \"\"\"\n if self.backbone.__class__.__name__ == 'ResNet3dSlowOnly':\n imgs = data_batch[0]\n labels = data_batch[1:]\n if imgs.dim() == 6:\n imgs = imgs.reshape([-1] + imgs.shape[2:])\n else:\n imgs = data_batch[0:2]\n labels = data_batch[2:]\n # call forward\n cls_score = self.forward_net(imgs)\n loss_metrics = self.head.loss(cls_score, labels)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n if self.backbone.__class__.__name__ == 'ResNet3dSlowOnly':\n imgs = data_batch[0]\n labels = data_batch[1:]\n if imgs.dim() == 6:\n imgs = imgs.reshape([-1] + imgs.shape[2:])\n else:\n imgs = data_batch[0:2]\n labels = data_batch[2:]\n # call forward\n cls_score = self.forward_net(imgs)\n loss_metrics = self.head.loss(cls_score, labels, valid_mode=True)\n return loss_metrics",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer3d.py:34-64"
+ },
+ "7439": {
+ "file_id": 545,
+ "content": "The code is defining two methods, `train_step` and `val_step`, which are used for training and validation steps respectively in a recognition model. If the backbone of the model is 'ResNet3dSlowOnly', it reshapes the images to have a specific dimension before processing. For other backbones, it separates the images and labels from the data batch accordingly. Both methods then forward the images through the `forward_net` and calculate loss metrics with or without validation mode depending on the step type. The final output is the loss metrics.",
+ "type": "comment"
+ },
+ "7440": {
+ "file_id": 545,
+ "content": " def test_step(self, data_batch):\n \"\"\"Test step.\n \"\"\"\n if self.backbone.__class__.__name__ == 'ResNet3dSlowOnly':\n imgs = data_batch[0]\n if imgs.dim() == 6:\n imgs = imgs.reshape([-1] + imgs.shape[2:])\n else:\n imgs = data_batch[0:2]\n # call forward\n cls_score = self.forward_net(imgs)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Infer step.\n \"\"\"\n if self.backbone.__class__.__name__ == 'ResNet3dSlowOnly':\n imgs = data_batch[0]\n # call forward\n imgs = imgs.reshape([-1] + imgs.shape[2:])\n cls_score = self.forward_net(imgs)\n else:\n imgs = data_batch[0:2]\n # call forward\n cls_score = self.forward_net(imgs)\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer3d.py:66-93"
+ },
+ "7441": {
+ "file_id": 545,
+ "content": "The code defines two methods: `test_step` and `infer_step`. In the `test_step`, if the backbone is a 'ResNet3dSlowOnly', it reshapes the input images, then calls the forward pass to get class scores. Otherwise, it takes the first two elements of the data batch for inference. The `infer_step` follows similar logic but without the condition on backbone type. Both methods return the class scores.",
+ "type": "comment"
+ },
+ "7442": {
+ "file_id": 546,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer3dMRI.py",
+ "type": "filepath"
+ },
+ "7443": {
+ "file_id": 546,
+ "content": "The code defines a 3D Recognizer model and framework in PaddleVideo, with classes and methods for training, validation, and testing. It includes two methods, \"test_step\" and \"infer_step\", used for testing or inferring on limited data batches.",
+ "type": "summary"
+ },
+ "7444": {
+ "file_id": 546,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nfrom paddlevideo.utils import get_logger\nimport paddle\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass Recognizer3DMRI(BaseRecognizer):\n \"\"\"3D Recognizer model framework.\n \"\"\"\n def forward_net(self, imgs):\n \"\"\"Define how the model is going to run, from input to output.\n \"\"\"\n imgs[0] = paddle.cast(imgs[0], \"float32\")\n imgs[1] = paddle.cast(imgs[1], \"float32\")\n imgs[0] = imgs[0].unsqueeze(1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer3dMRI.py:1-31"
+ },
+ "7445": {
+ "file_id": 546,
+ "content": "This code defines a 3D Recognizer model framework that takes in input images, casts them to float32 type and unsqueeze the first image for dimension alignment. The Recognizer3DMRI class inherits from BaseRecognizer and has a forward_net method for defining how the model should run from input to output.",
+ "type": "comment"
+ },
+ "7446": {
+ "file_id": 546,
+ "content": " imgs[1] = imgs[1].unsqueeze(1)\n feature = self.backbone(imgs)\n cls_score = self.head(feature)\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n imgs = data_batch[0:2]\n labels = data_batch[2:]\n # call forward\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score, labels, if_top5=False)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n imgs = data_batch[0:2]\n labels = data_batch[2:]\n # call forward\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score,\n labels,\n valid_mode=True,\n if_top5=False)\n return loss_metrics\n def test_step(self, data_batch):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer3dMRI.py:32-65"
+ },
+ "7447": {
+ "file_id": 546,
+ "content": "This code defines a recognizer3dMRI model in the PaddleVideo framework. It has three methods: train_step, val_step, and test_step for training, validating, and testing the model, respectively. In each step, it processes image data batches, calls the forward function to generate class scores using a forward_net, applies sigmoid activation, and calculates losses using the head's loss function.",
+ "type": "comment"
+ },
+ "7448": {
+ "file_id": 546,
+ "content": " \"\"\"Test step.\n \"\"\"\n imgs = data_batch[0:2]\n # call forward\n cls_score = self.forward_net(imgs)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Infer step.\n \"\"\"\n imgs = data_batch[0:2]\n # call forward\n cls_score = self.forward_net(imgs)\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer3dMRI.py:66-81"
+ },
+ "7449": {
+ "file_id": 546,
+ "content": "This code defines two methods, \"test_step\" and \"infer_step\", which both take a data batch as input and return the class score after calling the forward function in the forward_net object. These steps seem to be used for testing or inferring on a limited subset of the data batch (the first two images).",
+ "type": "comment"
+ },
+ "7450": {
+ "file_id": 547,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py",
+ "type": "filepath"
+ },
+ "7451": {
+ "file_id": 547,
+ "content": "The code introduces a RecognizerDistillation class for recognizer distillation in PaddleVideo's framework, and includes model selection, modes like training and validation, loss functions, accuracy functions, and forward pass capabilities.",
+ "type": "summary"
+ },
+ "7452": {
+ "file_id": 547,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\nfrom ...registry import RECOGNIZERS\nfrom ... import builder\nfrom paddlevideo.utils import get_logger, get_dist_info\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerDistillation(nn.Layer):\n \"\"\"recognizer Distillation framework.\"\"\"\n def __init__(self,\n freeze_params_list=None,\n models=None,\n loss=None,\n **kargs):\n \"\"\"\n Args:\n freeze_params_list: list, set each model is trainable or not",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:1-34"
+ },
+ "7453": {
+ "file_id": 547,
+ "content": "This code defines a RecognizerDistillation class that inherits from nn.Layer in PaddleVideo's framework. It implements recognizer distillation, which is a machine learning framework for object recognition tasks. The class takes optional arguments such as freeze_params_list (a list to set models trainable/not), models, and loss. It is registered under RECOGNIZERS and uses logger from paddlevideo's utils.",
+ "type": "comment"
+ },
+ "7454": {
+ "file_id": 547,
+ "content": " models: config of distillaciton model.\n loss: config of loss list\n \"\"\"\n super().__init__()\n self.model_list = []\n self.model_name_list = []\n self.loss_cfgs = loss\n if freeze_params_list is None:\n freeze_params_list = [False] * len(models)\n assert len(freeze_params_list) == len(models)\n # build Teacher and Student model\n for idx, model_config in enumerate(models):\n assert len(model_config) == 1\n key = list(model_config.keys())[0] #Teacher or Student\n model_config = model_config[key]\n model_name = model_config['backbone']['name']\n backbone, head = None, None\n if model_config.get('backbone'):\n backbone = builder.build_backbone(model_config['backbone'])\n if hasattr(backbone, 'init_weights'):\n backbone.init_weights()\n if model_config.get('head'):\n head = builder.build_head(model_config['head'])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:35-60"
+ },
+ "7455": {
+ "file_id": 547,
+ "content": "This code initializes an instance of a distillation model. It takes in a list of models and loss configurations, as well as a freeze_params_list (optional). It checks the lengths of the input lists, builds teacher and student models, and initializes backbone and head if they exist in the configurations.",
+ "type": "comment"
+ },
+ "7456": {
+ "file_id": 547,
+ "content": " if hasattr(head, 'init_weights'):\n head.init_weights()\n model = nn.Sequential(backbone, head)\n logger.info('build distillation {} model done'.format(key))\n # for add all parameters in nn.Layer class\n self.model_list.append(self.add_sublayer(key, model))\n self.model_name_list.append({model_name: key})\n # set model trainable or not\n if freeze_params_list[idx]:\n for param in model.parameters():\n param.trainable = False\n # build loss: support for loss list\n self.loss_func_list = []\n mode_keys = list(loss.keys())\n for mode in mode_keys:\n loss_cfgs = loss[mode]\n for loss_cfg in loss_cfgs:\n loss_func_dict = {}\n model_name_pairs = loss_cfg.pop('model_name_pairs')\n loss_func = builder.build_loss(loss_cfg)\n loss_func_dict['mode'] = mode\n loss_func_dict['loss_func'] = loss_func",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:61-85"
+ },
+ "7457": {
+ "file_id": 547,
+ "content": "Builds a distillation model by appending a head to the backbone, initializes weights for the head if possible, and sets trainable parameters based on freeze_params_list. Constructs loss functions using builder.build_loss().",
+ "type": "comment"
+ },
+ "7458": {
+ "file_id": 547,
+ "content": " loss_func_dict['model_name_pairs'] = model_name_pairs\n self.loss_func_list.append(loss_func_dict)\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n def get_loss(self, output, labels, mode):\n \"\"\"\n Args:\n output: dict, output name and its value\n labels: label of data\n mode: str, 'Train' or 'Val'\n \"\"\"\n output['GroundTruth'] = labels\n loss_list = []",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:86-114"
+ },
+ "7459": {
+ "file_id": 547,
+ "content": "This code defines a class for handling different modes of operation (train, valid, test, infer) and includes methods to handle each mode. It also contains a method to calculate the loss based on output and labels in 'Train' or 'Val' mode. The code is likely used in a model framework and the class might be used to control the flow and operations of the model depending on the mode it runs in.",
+ "type": "comment"
+ },
+ "7460": {
+ "file_id": 547,
+ "content": " for loss_func_dict in self.loss_func_list:\n if mode == loss_func_dict['mode']:\n model_name_pairs = loss_func_dict['model_name_pairs']\n loss_func = loss_func_dict['loss_func']\n loss_val = loss_func(output[model_name_pairs[0]],\n output[model_name_pairs[1]])\n loss_list.append(loss_val)\n total_loss = paddle.add_n(loss_list)\n return total_loss\n def get_acc(self, scores, labels, mode='Train'):\n def _get_acc(score, label, mode='Train'):\n top1 = paddle.metric.accuracy(input=score, label=label, k=1)\n top5 = paddle.metric.accuracy(input=score, label=label, k=5)\n _, world_size = get_dist_info()\n # Deal with multi cards validate\n if world_size > 1 and mode == 'Val': #reduce sum when valid\n top1 = paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM) / world_size\n top5 = paddle.distributed.all_reduce(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:116-136"
+ },
+ "7461": {
+ "file_id": 547,
+ "content": "This code is iterating over a list of loss function dictionaries to find the appropriate loss function based on the input mode. It then calculates the loss value and appends it to a list. Finally, it adds up all the loss values to get the total loss. In the `get_acc` method, it defines an inner function that calculates top-1 and top-5 accuracy scores using PaddlePaddle's `metric.accuracy` function. It also handles multi-card validation by reducing the sum of top-1 and top-5 accuracy scores across multiple cards.",
+ "type": "comment"
+ },
+ "7462": {
+ "file_id": 547,
+ "content": " top5, op=paddle.distributed.ReduceOp.SUM) / world_size\n return top1, top5\n if len(labels) == 1:\n label = labels[0]\n return _get_acc(scores, label)\n # Deal with VideoMix\n elif len(labels) == 3:\n label_a, label_b, lam = labels\n top1a, top5a = _get_acc(scores, label_a, mode)\n top1b, top5b = _get_acc(scores, label_b, mode)\n top1 = lam * top1a + (1 - lam) * top1b\n top5 = lam * top5a + (1 - lam) * top5b\n return top1, top5\n def forward_model(self, imgs, model_name, model):\n if model_name in ['PPTSM_v2', 'ResNetTweaksTSM']:\n # [N,T,C,H,W] -> [N*T,C,H,W]\n imgs = paddle.reshape(imgs, [-1] + list(imgs.shape[2:]))\n return model(imgs)\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n out = {}\n loss_metrics = {}\n imgs = data_batch[0]\n labels = data_batch[1:]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:137-165"
+ },
+ "7463": {
+ "file_id": 547,
+ "content": "The code snippet contains a recognizerDistillation function, which calculates accuracy based on given scores and labels. It also includes a forward_model function for reshaping images and applying model operations. The train_step function defines the training process from input to output, including loss metrics calculation.",
+ "type": "comment"
+ },
+ "7464": {
+ "file_id": 547,
+ "content": " for idx, item in enumerate(self.model_name_list):\n model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n out[model_type] = self.forward_model(imgs, model_name, model)\n # out_student, out_teacher\n loss = self.get_loss(out, labels, 'Train')\n loss_metrics['loss'] = loss\n # calculate acc with student output\n top1, top5 = self.get_acc(out['Student'], labels)\n loss_metrics['top1'] = top1\n loss_metrics['top5'] = top5\n return loss_metrics\n def val_step(self, data_batch):\n out = {}\n loss_metrics = {}\n imgs = data_batch[0]\n labels = data_batch[1:]\n for idx, item in enumerate(self.model_name_list):\n model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n out[model_type] = self.forward_model(imgs, model_name, model)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:167-193"
+ },
+ "7465": {
+ "file_id": 547,
+ "content": "This code defines a class that implements a recognizer for distillation. The model takes in an image and a model name, and returns the output from both the student and teacher models. It calculates loss using the student and teacher outputs, as well as top-1 and top-5 accuracy metrics from the student's output only. This is used for both training (train_step) and validation (val_step). The class utilizes a list of model names and corresponding models for both types, student and teacher, and iterates over them to apply the forward pass and calculate loss and metrics.",
+ "type": "comment"
+ },
+ "7466": {
+ "file_id": 547,
+ "content": " # Loss of student with gt: out_student, label\n loss = self.get_loss(out, labels, 'Val')\n loss_metrics['loss'] = loss\n top1, top5 = self.get_acc(out['Student'], labels, 'Val')\n loss_metrics['top1'] = top1\n loss_metrics['top5'] = top5\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n # Use Student to test\n for idx, item in enumerate(self.model_name_list):\n model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n if model_type == \"Student\":\n out = self.forward_model(imgs, model_name, model)\n return out\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n # Use Student to infer\n for idx, item in enumerate(self.model_name_list):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:195-224"
+ },
+ "7467": {
+ "file_id": 547,
+ "content": "In this code snippet, the get_loss and get_acc functions are used to calculate loss and accuracy metrics for a \"Student\" model. The test_step function tests the Student model using forward_model function, and the infer_step function is not implemented here. This code seems related to evaluating the performance of a student model in image recognition tasks.",
+ "type": "comment"
+ },
+ "7468": {
+ "file_id": 547,
+ "content": " model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n if model_type == \"Student\":\n out = self.forward_model(imgs, model_name, model)\n return out",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py:225-231"
+ },
+ "7469": {
+ "file_id": 547,
+ "content": "The code selects a model from the model_list based on the idx, and assigns its name to model_name. If the model type is \"Student\", it calls forward_model function passing imgs, model_name, and model as parameters, and returns the output.",
+ "type": "comment"
+ },
+ "7470": {
+ "file_id": 548,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizerMRI.py",
+ "type": "filepath"
+ },
+ "7471": {
+ "file_id": 548,
+ "content": "The code creates a 2D image classifier model using PaddleVideo's RecognizerMRI, with train_step and val_step calculating loss metrics, and test_step for testing without calling head.loss during inference.",
+ "type": "summary"
+ },
+ "7472": {
+ "file_id": 548,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nimport paddle\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerMRI(BaseRecognizer):\n \"\"\"2D recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n # NOTE: As the num_segs is an attribute of dataset phase, and didn't pass to build_head phase, should obtain it from imgs(paddle.Tensor) now, then call self.head method.\n num_segs = imgs.shape[\n 1] # imgs.shape=[N,T,C,H,W], for most commonly case",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerMRI.py:1-27"
+ },
+ "7473": {
+ "file_id": 548,
+ "content": "Code is from PaddleVideo's RecognizerMRI class, a 2D recognizer model framework. It has a forward_net method that takes imgs as input and returns the output of the network. The number of segments is obtained from the image shape and used to call the self.head method.",
+ "type": "comment"
+ },
+ "7474": {
+ "file_id": 548,
+ "content": " imgs = paddle.reshape_(imgs, [-1] + list(imgs.shape[2:]))\n imgs = paddle.cast(imgs, \"float32\") #############\n imgs = imgs.unsqueeze(1)\n if self.backbone != None:\n feature = self.backbone(imgs)\n else:\n feature = imgs\n if self.head != None:\n cls_score = self.head(feature, num_segs)\n else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score, labels, if_top5=False)\n return loss_metrics\n def val_step(self, data_batch):\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerMRI.py:28-59"
+ },
+ "7475": {
+ "file_id": 548,
+ "content": "This code defines a model for image classification. It first reshapes and casts the input images to float32 type, then passes them through a backbone network if one is defined. After that, it sends the resulting feature map through a head network (if defined) to produce class scores. The train_step function uses these class scores to calculate loss metrics during training, while the val_step function performs similar operations but does not compute losses.",
+ "type": "comment"
+ },
+ "7476": {
+ "file_id": 548,
+ "content": " labels,\n valid_mode=True,\n if_top5=False)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n # NOTE: (shipping) when testing, the net won't call head.loss, we deal with the test processing in /paddlevideo/metrics\n imgs = data_batch[0]\n cls_score = self.forward_net(imgs)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n cls_score = self.forward_net(imgs)\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizerMRI.py:60-76"
+ },
+ "7477": {
+ "file_id": 548,
+ "content": "The code defines a test_step and infer_step function for a model, which takes in data_batch as input and returns the classification scores from the forward_net function. The test_step specifically mentions that during testing, the net won't call head.loss.",
+ "type": "comment"
+ },
+ "7478": {
+ "file_id": 549,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer_gcn.py",
+ "type": "filepath"
+ },
+ "7479": {
+ "file_id": 549,
+ "content": "The code introduces a GCN Recognizer model framework for PaddleVideo, classifying images through forward pass definition, training step loss calculation, and validation. A RecognizerGCN model is defined with test_step and infer_step functions.",
+ "type": "summary"
+ },
+ "7480": {
+ "file_id": 549,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerGCN(BaseRecognizer):\n \"\"\"GCN Recognizer model framework.\n \"\"\"\n def __init__(self,\n backbone=None,\n head=None,\n runtime_cfg=None,\n if_top5=True):\n \"\"\"\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Classification head to process feature.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_gcn.py:1-33"
+ },
+ "7481": {
+ "file_id": 549,
+ "content": "This code defines a GCN Recognizer model framework for PaddleVideo. It has an initialization method that takes arguments for backbone, head, runtime_cfg, and if_top5. The GCN Recognizer is registered with the RECOGNIZERS registry and extends BaseRecognizer class.",
+ "type": "comment"
+ },
+ "7482": {
+ "file_id": 549,
+ "content": " is_top5 (bool): Whether to display top-5 accuracy during training/validation steps.\n \"\"\"\n super(RecognizerGCN, self).__init__(backbone, head, runtime_cfg)\n self.if_top5 = if_top5\n def forward_net(self, data):\n \"\"\"Define how the model is going to run, from input to output.\n \"\"\"\n feature = self.backbone(data)\n cls_score = self.head(feature)\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n data = data_batch[0]\n label = data_batch[1:]\n # call forward\n cls_score = self.forward_net(data)\n loss_metrics = self.head.loss(cls_score, label, if_top5=self.if_top5)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n data = data_batch[0]\n label = data_batch[1:]\n # call forward\n cls_score = self.forward_net(data)\n loss_metrics = self.head.loss(cls_score,\n label,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_gcn.py:34-66"
+ },
+ "7483": {
+ "file_id": 549,
+ "content": "RecognizerGCN is a model that performs image classification. It has a backbone for feature extraction and a head for classification. Forward_net defines the forward pass. Train_step calculates loss and metrics during training, taking into account if_top5 flag. Val_step performs validation by forward pass and loss calculation.",
+ "type": "comment"
+ },
+ "7484": {
+ "file_id": 549,
+ "content": " valid_mode=True,\n if_top5=self.if_top5)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Test step.\n \"\"\"\n data = data_batch[0]\n # call forward\n cls_score = self.forward_net(data)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Infer step.\n \"\"\"\n data = data_batch[0]\n # call forward\n cls_score = self.forward_net(data)\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_gcn.py:67-87"
+ },
+ "7485": {
+ "file_id": 549,
+ "content": "The code defines a RecognizerGCN model and provides test_step and infer_step functions to classify data by forwarding it through the network and returning class scores.",
+ "type": "comment"
+ },
+ "7486": {
+ "file_id": 550,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer_movinet_frame.py",
+ "type": "filepath"
+ },
+ "7487": {
+ "file_id": 550,
+ "content": "The MoViNetRecognizerFrame class, extending BaseRecognizer, has forward_net and train_step methods for training steps. Three functions - forward_net, test_step, and infer_step are defined for model's testing or inference process.",
+ "type": "summary"
+ },
+ "7488": {
+ "file_id": 550,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nfrom paddlevideo.utils import get_logger\nfrom .base import BaseRecognizer\nfrom ...registry import RECOGNIZERS\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass MoViNetRecognizerFrame(BaseRecognizer):\n def forward_net(self, imgs):\n \"\"\"Define how the model is going to run, from input to output.\n \"\"\"\n self.backbone.clean_activation_buffers()\n outputs = self.backbone(imgs)\n cls_score = self.head(outputs)\n return cls_score\n def train_step(self, data_batch):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_movinet_frame.py:1-33"
+ },
+ "7489": {
+ "file_id": 550,
+ "content": "The code is defining a class named \"MoViNetRecognizerFrame\" which extends the BaseRecognizer class. It has two methods, forward_net and train_step. The forward_net method defines how the model will run from input to output by first cleaning activation buffers in the backbone and then passing the inputs through it to get outputs. Finally, the head is applied on these outputs to get class scores. The train_step method defines a training step for this model.",
+ "type": "comment"
+ },
+ "7490": {
+ "file_id": 550,
+ "content": " \"\"\"Training step.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1] #.astype(\"int64\")\n data = paddle.transpose(imgs, perm=[0, 2, 1, 3, 4])\n # call forward\n cls_score = self.forward_net(data)\n loss_metrics = self.head.loss_func(cls_score, labels)\n top1 = paddle.metric.accuracy(input=cls_score, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=cls_score, label=labels, k=5)\n output = {'loss': loss_metrics, 'top1': top1, 'top5': top5}\n return output\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1] #.astype(\"int64\")\n data = paddle.transpose(imgs, perm=[0, 2, 1, 3, 4])\n # call forward\n cls_score = self.forward_net(data)\n loss_metrics = self.head.loss_func(cls_score, labels)\n top1 = paddle.metric.accuracy(input=cls_score, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=cls_score, label=labels, k=5)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_movinet_frame.py:34-57"
+ },
+ "7491": {
+ "file_id": 550,
+ "content": "Training step: Implements a training step for the model, taking data_batch as input. Extracts images and labels, transposes data, applies forward pass in the network, calculates loss metrics, and computes top-1 and top-5 accuracy scores. Returns output with 'loss', 'top1', and 'top5' keys.\nValidating step: Implements a validating step for the model, similar to training step but used to validate the model on unseen data. Computes top-1 and top-5 accuracy scores along with loss metrics.",
+ "type": "comment"
+ },
+ "7492": {
+ "file_id": 550,
+ "content": " output = {'loss': loss_metrics, 'top1': top1, 'top5': top5}\n return output\n def test_step(self, data_batch):\n \"\"\"Test step.\n \"\"\"\n imgs = data_batch[0]\n data = paddle.transpose(imgs, perm=[0, 2, 1, 3, 4])\n # call forward\n cls_score = self.forward_net(data)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Infer step.\n \"\"\"\n imgs = data_batch[0]\n # call forward\n data = paddle.transpose(imgs, perm=[0, 2, 1, 3, 4])\n cls_score = self.forward_net(data)\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_movinet_frame.py:58-78"
+ },
+ "7493": {
+ "file_id": 550,
+ "content": "This code defines three functions: `forward_net`, `test_step`, and `infer_step`. The `forward_net` function is the core of the model, responsible for forward propagation. The `test_step` and `infer_step` functions both take in a data batch, transpose the images, call the `forward_net` function to get class scores, and return these scores. These steps are likely part of a deep learning model's testing or inference process.",
+ "type": "comment"
+ },
+ "7494": {
+ "file_id": 551,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer.py",
+ "type": "filepath"
+ },
+ "7495": {
+ "file_id": 551,
+ "content": "The code defines a RecognizerTransformer class for implementing a transformer-based recognizer model, which includes feature extraction, training, validation, and testing steps. It also defines a model for inferring image results from multiple views using forward_net function and averaging based on 'avg_type'.",
+ "type": "summary"
+ },
+ "7496": {
+ "file_id": 551,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddlevideo.utils import get_logger\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerTransformer(BaseRecognizer):\n \"\"\"Transformer's recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n # imgs.shape=[N,C,T,H,W], for transformer case\n if self.backbone is not None:\n feature = self.backbone(imgs)\n else:\n feature = imgs",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer.py:1-31"
+ },
+ "7497": {
+ "file_id": 551,
+ "content": "This code defines a RecognizerTransformer class that inherits from BaseRecognizer and implements a transformer-based recognizer model framework. It takes in an input tensor imgs of shape [N,C,T,H,W] where N is the batch size, C is the number of channels, T is the temporal length, H is the height, and W is the width. If a backbone is specified, it applies the backbone to the images for feature extraction; otherwise, it uses the input images directly. The resulting feature tensor is returned.",
+ "type": "comment"
+ },
+ "7498": {
+ "file_id": 551,
+ "content": " if self.head is not None:\n cls_score = self.head(feature)\n else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n loss_metrics = self.head.loss(cls_score, labels)\n return loss_metrics\n def val_step(self, data_batch):\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n loss_metrics = self.head.loss(cls_score, labels, valid_mode=True)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n imgs = data_batch[0]\n num_views = imgs.shape[2] // self.runtime_cfg.test.num_seg\n cls_score = []\n for i in range(num_views):\n view = imgs[:, :, i * self.runtime_cfg.test.num_seg:(i + 1) *",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer.py:33-62"
+ },
+ "7499": {
+ "file_id": 551,
+ "content": "The code defines a model's training, validation, and testing steps. The train_step calculates the loss between predicted class scores and actual labels. The val_step is similar but marks some samples as valid in validation mode. The test_step infers by processing views of images and stores class scores in a list.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/75.json b/docs/data/75.json
new file mode 100644
index 000000000..c75692bfe
--- /dev/null
+++ b/docs/data/75.json
@@ -0,0 +1,547 @@
+{
+ "7500": {
+ "file_id": 551,
+ "content": " self.runtime_cfg.test.num_seg]\n cls_score.append(self.forward_net(view))\n cls_score = self._average_view(cls_score,\n self.runtime_cfg.test.avg_type)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n imgs = data_batch[0]\n num_views = imgs.shape[2] // self.runtime_cfg.test.num_seg\n cls_score = []\n for i in range(num_views):\n view = imgs[:, :, i * self.runtime_cfg.test.num_seg:(i + 1) *\n self.runtime_cfg.test.num_seg]\n cls_score.append(self.forward_net(view))\n cls_score = self._average_view(cls_score,\n self.runtime_cfg.test.avg_type)\n return cls_score\n def _average_view(self, cls_score, avg_type='score'):\n \"\"\"Combine the predicted results of different views\n Args:\n cls_score (list): results of multiple views",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer.py:63-86"
+ },
+ "7501": {
+ "file_id": 551,
+ "content": "This code defines a model for inferring the results from multiple views of images. The `forward_net` function is used to process each view, and then the results are averaged using the `_average_view` function based on the specified average type. This allows the model to make predictions from different perspectives of an image and combine them for a more accurate result.",
+ "type": "comment"
+ },
+ "7502": {
+ "file_id": 551,
+ "content": " avg_type (str, optional): Average calculation method. Defaults to 'score'.\n \"\"\"\n assert avg_type in ['score', 'prob'], \\\n f\"Currently only the average of 'score' or 'prob' is supported, but got {avg_type}\"\n if avg_type == 'score':\n return paddle.add_n(cls_score) / len(cls_score)\n elif avg_type == 'prob':\n return paddle.add_n(\n [F.softmax(score, axis=-1)\n for score in cls_score]) / len(cls_score)\n else:\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer.py:87-98"
+ },
+ "7503": {
+ "file_id": 551,
+ "content": "This code defines a class method with an optional 'avg_type' parameter for average calculation. It checks if the input is either 'score' or 'prob'. If 'score', it returns the sum of 'cls_score' divided by its length. If 'prob', it applies softmax to each element in 'cls_score', then averages their sum and length. Otherwise, it raises a NotImplementedError.",
+ "type": "comment"
+ },
+ "7504": {
+ "file_id": 552,
+ "content": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py",
+ "type": "filepath"
+ },
+ "7505": {
+ "file_id": 552,
+ "content": "The code imports libraries, defines the RecognizerTransformer_MRI model class with forward method and training/validation steps, using loss metrics. It includes two inference methods: 'test_step' and 'infer_step', which split input into multiple views for classification score generation. The average_view function combines these scores across views, using either 'score' or 'prob' averaging types.",
+ "type": "summary"
+ },
+ "7506": {
+ "file_id": 552,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddlevideo.utils import get_logger\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerTransformer_MRI(BaseRecognizer):\n \"\"\"Transformer's recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n # imgs.shape=[N,C,T,H,W], for transformer case\n imgs = paddle.cast(imgs, \"float32\") #############\n imgs = imgs.unsqueeze(1)\n if self.backbone != None:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py:1-32"
+ },
+ "7507": {
+ "file_id": 552,
+ "content": "This code imports necessary libraries, defines a class for the RecognizerTransformer_MRI model, and sets the input image shape. The forward_net method preprocesses input images by casting them to float32 type and adding an extra dimension for compatibility with the transformer architecture.",
+ "type": "comment"
+ },
+ "7508": {
+ "file_id": 552,
+ "content": " feature = self.backbone(imgs)\n else:\n feature = imgs\n if self.head != None:\n cls_score = self.head(feature)\n else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score, labels, if_top5=False)\n return loss_metrics\n def val_step(self, data_batch):\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score,\n labels,\n valid_mode=True,\n if_top5=False)\n return loss_metrics",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py:33-63"
+ },
+ "7509": {
+ "file_id": 552,
+ "content": "This code defines a recognizer transformer model for image classification. The `forward_net` method processes images and returns class scores, while the `train_step` and `val_step` methods perform training and validation steps by passing data batches to the model and computing loss metrics using sigmoid activation and the head's loss function.",
+ "type": "comment"
+ },
+ "7510": {
+ "file_id": 552,
+ "content": " def test_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n imgs = data_batch[0]\n num_views = imgs.shape[2] // self.backbone.seg_num\n cls_score = []\n for i in range(num_views):\n view = imgs[:, :, i * self.backbone.seg_num:(i + 1) *\n self.backbone.seg_num]\n cls_score.append(self.forward_net(view))\n cls_score = self.average_view(cls_score)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n imgs = data_batch[0]\n num_views = imgs.shape[2] // self.backbone.seg_num\n cls_score = []\n for i in range(num_views):\n view = imgs[:, :, i * self.backbone.seg_num:(i + 1) *\n self.backbone.seg_num]\n cls_score.append(self.forward_net(view))\n cls_score = self.average_view(cls_score)\n return cls_score\n def average_view(self, cls_score, average_type='score'):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py:65-89"
+ },
+ "7511": {
+ "file_id": 552,
+ "content": "The code defines two methods, 'test_step' and 'infer_step', for the model to infer from input to output. It splits the input into multiple views based on the number of segments in each view. For each view, it applies the forward network to generate a set of classification scores. Finally, it averages the scores across all views using the average_view method.",
+ "type": "comment"
+ },
+ "7512": {
+ "file_id": 552,
+ "content": " \"\"\"Combine the scores of different views\n Args:\n cls_score (list): Scores of multiple views\n average_type (str, optional): Average calculation method. Defaults to 'score'.\n \"\"\"\n assert average_type in ['score', 'prob'], \\\n f\"Currently only the average of 'score' or 'prob' is supported, but got {average_type}\"\n if average_type == 'score':\n return paddle.add_n(cls_score) / len(cls_score)\n elif average_type == 'avg':\n return paddle.add_n([F.softmax(score)\n for score in cls_score]) / len(cls_score)\n else:\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py:90-104"
+ },
+ "7513": {
+ "file_id": 552,
+ "content": "This function combines the scores of multiple views, taking two arguments: a list of cls_scores and an optional average_type. It asserts that average_type is either 'score' or 'prob'. If 'score', it adds all scores in the list and divides by the count. If 'avg', it first applies softmax to each score, then adds them and divides by the count. Otherwise, it raises a NotImplementedError.",
+ "type": "comment"
+ },
+ "7514": {
+ "file_id": 553,
+ "content": "/paddlevideo/modeling/framework/segment/__init__.py",
+ "type": "filepath"
+ },
+ "7515": {
+ "file_id": 553,
+ "content": "This code file contains the Python implementation of segment models in PaddleVideo, including BaseSegment and CFBI classes. It is licensed under the Apache License, Version 2.0. The __all__ variable lists the available segments: BaseSegment and CFBI.",
+ "type": "summary"
+ },
+ "7516": {
+ "file_id": 553,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom .base import BaseSegment\nfrom .cfbi import CFBI\n__all__ = ['BaseSegment', 'CFBI']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/__init__.py:1-16"
+ },
+ "7517": {
+ "file_id": 553,
+ "content": "This code file contains the Python implementation of segment models in PaddleVideo, including BaseSegment and CFBI classes. It is licensed under the Apache License, Version 2.0. The __all__ variable lists the available segments: BaseSegment and CFBI.",
+ "type": "comment"
+ },
+ "7518": {
+ "file_id": 554,
+ "content": "/paddlevideo/modeling/framework/segment/base.py",
+ "type": "filepath"
+ },
+ "7519": {
+ "file_id": 554,
+ "content": "This code defines a semi-Video Object Segmentation abstract base class with train_step, valid_step, and test_step methods for different modes (train, valid, test, or infer). Subclasses must implement 4 methods for model training and evaluation.",
+ "type": "summary"
+ },
+ "7520": {
+ "file_id": 554,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseSegment(nn.Layer):\n \"\"\"Base class for semi-Video Object Segmentation.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Head to process feature.\n loss(dict): Loss function.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/base.py:1-29"
+ },
+ "7521": {
+ "file_id": 554,
+ "content": "This code defines an abstract base class for semi-Video Object Segmentation. It has three required methods: train_step, valid_step, and test_step. The class also contains backbone, head, and loss modules to extract feature, process feature, and define the loss function respectively.",
+ "type": "comment"
+ },
+ "7522": {
+ "file_id": 554,
+ "content": " \"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__()\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n if loss is not None:\n self.loss = builder.build_loss(loss)\n else:\n self.loss = None\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/base.py:30-57"
+ },
+ "7523": {
+ "file_id": 554,
+ "content": "This code initializes a segment model by building its backbone, head, and loss based on the provided parameters. The forward method defines how the model runs in different modes (train, valid, test, or infer). If running in train mode, the model performs training operations.",
+ "type": "comment"
+ },
+ "7524": {
+ "file_id": 554,
+ "content": " return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\"\n raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/base.py:58-90"
+ },
+ "7525": {
+ "file_id": 554,
+ "content": "The code defines an abstract class with four methods (train_step, val_step, test_step, and infer_step) that must be implemented by subclasses. The method name is chosen based on the mode input parameter for different phases of model training or evaluation. If an unsupported mode is passed, a NotImplementedError is raised.",
+ "type": "comment"
+ },
+ "7526": {
+ "file_id": 555,
+ "content": "/paddlevideo/modeling/framework/segment/cfbi.py",
+ "type": "filepath"
+ },
+ "7527": {
+ "file_id": 555,
+ "content": "This Python class initializes the CFBI model in PaddleVideo library for image segmentation and video processing using AI techniques, with instance-level attention via previous frame embeddings and labels.",
+ "type": "summary"
+ },
+ "7528": {
+ "file_id": 555,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom .utils import foreground2background, global_matching_for_eval, local_matching, calculate_attention_head_for_eval\nfrom ...registry import SEGMENT\nfrom .base import BaseSegment\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@SEGMENT.register()\nclass CFBI(BaseSegment):\n \"\"\"CFBI model framework.\"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__(backbone, head, loss)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:1-30"
+ },
+ "7529": {
+ "file_id": 555,
+ "content": "This code is a Python class for the CFBI model in the PaddleVideo library. It initializes the model and inherits from the BaseSegment class, allowing it to use other classes like backbone, head, and loss.",
+ "type": "comment"
+ },
+ "7530": {
+ "file_id": 555,
+ "content": " x1 = paddle.zeros([3, 1, 1, 1])\n self.bg_bias = paddle.create_parameter(\n shape=x1.shape,\n dtype=x1.dtype,\n default_initializer=nn.initializer.Assign(x1))\n self.fg_bias = paddle.create_parameter(\n shape=x1.shape,\n dtype=x1.dtype,\n default_initializer=nn.initializer.Assign(x1))\n self.epsilon = 1e-05\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\n \"\"\"\n self.test_mode = True\n ref_embeddings, ref_masks, prev_embedding, prev_mask, current_frame, pred_size, gt_ids = data_batch\n current_frame_embedding_4x, current_frame_embedding_8x, current_frame_embedding_16x, \\\n current_low_level = self.backbone(current_frame)\n current_frame_embedding = [\n current_frame_embedding_4x, current_frame_embedding_8x,\n current_frame_embedding_16x\n ]\n if prev_embedding is None:\n return None, current_frame_embedding",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:31-56"
+ },
+ "7531": {
+ "file_id": 555,
+ "content": "This code defines a class with a `test_step` method that performs testing on input data. It initializes some parameters and returns None if there is no previous embedding. The backbone function extracts multiple frame embeddings, which are stored in the `current_frame_embedding` list.",
+ "type": "comment"
+ },
+ "7532": {
+ "file_id": 555,
+ "content": " else:\n bs, c, h, w = current_frame_embedding_4x.shape\n tmp_dic, _ = self.before_seghead_process(\n ref_embeddings,\n prev_embedding,\n current_frame_embedding,\n ref_masks,\n prev_mask,\n gt_ids,\n current_low_level=current_low_level,\n )\n all_pred = []\n for i in range(bs):\n pred = tmp_dic[i]\n pred = F.interpolate(pred,\n size=[pred_size[0], pred_size[1]],\n mode='bilinear',\n align_corners=True)\n all_pred.append(pred)\n all_pred = paddle.concat(all_pred, axis=0)\n all_pred = F.softmax(all_pred, axis=1)\n return all_pred, current_frame_embedding\n def before_seghead_process(self,\n ref_frame_embeddings=None,\n previous_frame_embeddings=None,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:57-84"
+ },
+ "7533": {
+ "file_id": 555,
+ "content": "The code is in PaddleVideo framework, and it contains an else block that executes when a condition is not met. The function first defines the shape of current_frame_embedding_4x. It then processes reference embeddings, previous embeddings, and current frame embedding with other parameters such as masks and IDs. It interpolates predictions and concatenates them along the specified axis. Finally, it applies softmax to all_pred on the specified axis before returning both all_pred and current_frame_embedding.",
+ "type": "comment"
+ },
+ "7534": {
+ "file_id": 555,
+ "content": " current_frame_embeddings=None,\n ref_frame_labels=None,\n previous_frame_mask=None,\n gt_ids=None,\n current_low_level=None):\n \"\"\" process befor segmentation head\"\"\"\n TEST_GLOBAL_MATCHING_CHUNK = [4, 1, 1]\n TEST_GLOBAL_ATROUS_RATE = [2, 1, 1]\n TRAIN_LOCAL_ATROUS_RATE = [2, 1, 1]\n TEST_LOCAL_ATROUS_RATE = [2, 1, 1]\n MODEL_FLOAT16_MATCHING = False\n TEST_GLOBAL_MATCHING_MIN_PIXEL = 100\n MODEL_MULTI_LOCAL_DISTANCE = [[4, 8, 12, 16, 20, 24],\n [2, 4, 6, 8, 10, 12], [2, 4, 6, 8, 10]]\n TRAIN_LOCAL_PARALLEL = True\n TEST_LOCAL_PARALLEL = True\n MODEL_MATCHING_BACKGROUND = True\n MODEL_SEMANTIC_MATCHING_DIM = [32, 64, 128]\n dic_tmp = []\n boards = {}\n scale_ref_frame_labels = []\n scale_previous_frame_labels = []\n for current_frame_embedding in current_frame_embeddings:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:85-108"
+ },
+ "7535": {
+ "file_id": 555,
+ "content": "The code initializes various constants and variables for the segmentation head process. It includes settings for matching, atroous rates, and parallel processing, as well as defining arrays for scale reference frame labels and previous frame labels.",
+ "type": "comment"
+ },
+ "7536": {
+ "file_id": 555,
+ "content": " bs, c, h, w = current_frame_embedding.shape\n if not self.test_mode:\n raise NotImplementedError\n else:\n ref_frame_embeddings = list(zip(*ref_frame_embeddings))\n all_scale_ref_frame_label = []\n for ref_frame_label in ref_frame_labels:\n scale_ref_frame_label = paddle.cast(F.interpolate(\n paddle.cast(ref_frame_label, dtype=\"float32\"),\n size=(h, w),\n mode='nearest'),\n dtype=\"int32\")\n all_scale_ref_frame_label.append(scale_ref_frame_label)\n scale_ref_frame_labels.append(all_scale_ref_frame_label)\n scale_previous_frame_label = paddle.cast(F.interpolate(\n paddle.cast(previous_frame_mask, dtype=\"float32\"),\n size=(h, w),\n mode='nearest'),\n dtype=\"int32\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:109-127"
+ },
+ "7537": {
+ "file_id": 555,
+ "content": "Resizing ref_frame_label and previous_frame_mask to match current frame size for nearest mode interpolation in PaddleVideo model.",
+ "type": "comment"
+ },
+ "7538": {
+ "file_id": 555,
+ "content": " scale_previous_frame_labels.append(scale_previous_frame_label)\n for n in range(bs):\n ref_obj_ids = paddle.reshape(\n paddle.cast(paddle.arange(0,\n np.array(gt_ids)[n] + 1),\n dtype=\"int32\"), [-1, 1, 1, 1])\n obj_num = ref_obj_ids.shape[0]\n low_level_feat = paddle.unsqueeze(current_low_level[n], axis=0)\n all_CE_input = []\n all_attention_head = []\n for scale_idx, current_frame_embedding, ref_frame_embedding, previous_frame_embedding, \\\n scale_ref_frame_label, scale_previous_frame_label in zip(range(3), \\\n current_frame_embeddings, ref_frame_embeddings, previous_frame_embeddings, \\\n scale_ref_frame_labels, scale_previous_frame_labels):\n #Prepare\n seq_current_frame_embedding = current_frame_embedding[n]\n seq_prev_frame_embedding = previous_frame_embedding[n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:128-144"
+ },
+ "7539": {
+ "file_id": 555,
+ "content": "The code is iterating over the input data and for each frame, it prepares the current_frame_embedding and previous_frame_embedding by reshaping, unsqueezing, and extracting the specific frames from their respective arrays. It then adds these embeddings to separate lists for later use in calculating attention scores and computing cross-entropy loss.",
+ "type": "comment"
+ },
+ "7540": {
+ "file_id": 555,
+ "content": " seq_previous_frame_label = paddle.cast(\n (paddle.cast(scale_previous_frame_label[n], dtype=\"int32\")\n == ref_obj_ids),\n dtype=\"float32\")\n if np.array(gt_ids)[n] > 0:\n dis_bias = paddle.concat([\n paddle.unsqueeze(self.bg_bias[scale_idx], axis=0),\n paddle.expand(\n paddle.unsqueeze(self.fg_bias[scale_idx], axis=0),\n [np.array(gt_ids)[n], -1, -1, -1])\n ],\n axis=0)\n else:\n dis_bias = paddle.unsqueeze(self.bg_bias[scale_idx], axis=0)\n #Global FG map\n matching_dim = MODEL_SEMANTIC_MATCHING_DIM[scale_idx]\n seq_current_frame_embedding_for_matching = paddle.transpose(\n seq_current_frame_embedding[:matching_dim], [1, 2, 0])\n if not self.test_mode:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:145-164"
+ },
+ "7541": {
+ "file_id": 555,
+ "content": "This code calculates the distance bias for each frame in a sequence and prepares it for matching. It checks if the object ID is greater than 0, then assigns the corresponding background or foreground distance bias. It also transposes the current frame embedding for matching in case it's not in test mode.",
+ "type": "comment"
+ },
+ "7542": {
+ "file_id": 555,
+ "content": " raise NotImplementedError\n else:\n all_scale_ref_frame_label = scale_ref_frame_label\n all_ref_frame_embedding = ref_frame_embedding\n all_reference_embeddings = []\n all_reference_labels = []\n seq_ref_frame_labels = []\n count = 0\n for idx in range(len(all_scale_ref_frame_label)):\n ref_frame_embedding = all_ref_frame_embedding[idx]\n scale_ref_frame_label = all_scale_ref_frame_label[idx]\n seq_ref_frame_embedding = ref_frame_embedding[n]\n seq_ref_frame_embedding = paddle.transpose(\n seq_ref_frame_embedding, [1, 2, 0])\n seq_ref_frame_label = paddle.cast(\n (paddle.cast(scale_ref_frame_label[n],\n dtype=\"int32\") == ref_obj_ids),\n dtype=\"float32\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:165-184"
+ },
+ "7543": {
+ "file_id": 555,
+ "content": "The code raises a NotImplementedError if the condition is met and otherwise creates variables for storing reference frame embeddings, labels, and sequence-specific values. It then iterates through the provided labels and embeddings to prepare them for use in the model's segmentation process.",
+ "type": "comment"
+ },
+ "7544": {
+ "file_id": 555,
+ "content": " seq_ref_frame_labels.append(seq_ref_frame_label)\n seq_ref_frame_label = paddle.transpose(\n paddle.squeeze(seq_ref_frame_label, axis=1),\n [1, 2, 0])\n all_reference_embeddings.append(\n seq_ref_frame_embedding[:, :, :matching_dim])\n all_reference_labels.append(seq_ref_frame_label)\n global_matching_fg = global_matching_for_eval(\n all_reference_embeddings=all_reference_embeddings,\n query_embeddings=\n seq_current_frame_embedding_for_matching,\n all_reference_labels=all_reference_labels,\n n_chunks=TEST_GLOBAL_MATCHING_CHUNK[scale_idx],\n dis_bias=dis_bias,\n atrous_rate=TEST_GLOBAL_ATROUS_RATE[scale_idx],\n use_float16=MODEL_FLOAT16_MATCHING,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:185-200"
+ },
+ "7545": {
+ "file_id": 555,
+ "content": "This code appears to be part of a computer vision model. It is appending reference frame labels, transposing them, and adding the reference embeddings to a list. Then it calls a function called \"global_matching_fg\" with the reference embeddings, query embeddings, reference labels, number of chunks, distance bias, and atrous rate as arguments. The function is likely used for global matching evaluation in the context of this model.",
+ "type": "comment"
+ },
+ "7546": {
+ "file_id": 555,
+ "content": " atrous_obj_pixel_num=TEST_GLOBAL_MATCHING_MIN_PIXEL)\n # Local FG map\n seq_prev_frame_embedding_for_matching = paddle.transpose(\n seq_prev_frame_embedding[:matching_dim], [1, 2, 0])\n seq_previous_frame_label_for_matching = paddle.transpose(\n paddle.squeeze(seq_previous_frame_label, axis=1), [1, 2, 0])\n local_matching_fg = local_matching(\n prev_frame_embedding=seq_prev_frame_embedding_for_matching,\n query_embedding=seq_current_frame_embedding_for_matching,\n prev_frame_labels=seq_previous_frame_label_for_matching,\n multi_local_distance=MODEL_MULTI_LOCAL_DISTANCE[scale_idx],\n dis_bias=dis_bias,\n atrous_rate=TRAIN_LOCAL_ATROUS_RATE[scale_idx] if\n not self.test_mode else TEST_LOCAL_ATROUS_RATE[scale_idx],\n use_float16=MODEL_FLOAT16_MATCHING,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:201-216"
+ },
+ "7547": {
+ "file_id": 555,
+ "content": "This code block prepares input for a local matching function to compare previous and current frames. It transposes the embeddings and labels, sets atrous rate based on test mode, and uses float16 if needed.",
+ "type": "comment"
+ },
+ "7548": {
+ "file_id": 555,
+ "content": " allow_downsample=False,\n allow_parallel=TRAIN_LOCAL_PARALLEL\n if not self.test_mode else TEST_LOCAL_PARALLEL)\n #Aggregate Pixel-level Matching\n to_cat_global_matching_fg = paddle.transpose(\n paddle.squeeze(global_matching_fg, axis=0), [2, 3, 0, 1])\n to_cat_local_matching_fg = paddle.transpose(\n paddle.squeeze(local_matching_fg, axis=0), [2, 3, 0, 1])\n all_to_cat = [\n to_cat_global_matching_fg, to_cat_local_matching_fg,\n seq_previous_frame_label\n ]\n #Global and Local BG map\n if MODEL_MATCHING_BACKGROUND:\n to_cat_global_matching_bg = foreground2background(\n to_cat_global_matching_fg,\n np.array(gt_ids)[n] + 1)\n reshaped_prev_nn_feature_n = paddle.unsqueeze(\n paddle.transpose(to_cat_local_matching_fg,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:217-237"
+ },
+ "7549": {
+ "file_id": 555,
+ "content": "This code performs pixel-level matching and global/local background subtraction for image segmentation. It transposes and squeezes the global and local matching results, concatenates them with previous frame labels, and if using background modeling, computes global and local background maps.",
+ "type": "comment"
+ },
+ "7550": {
+ "file_id": 555,
+ "content": " [0, 2, 3, 1]),\n axis=1)\n to_cat_local_matching_bg = foreground2background(\n reshaped_prev_nn_feature_n,\n np.array(gt_ids)[n] + 1)\n to_cat_local_matching_bg = paddle.squeeze(paddle.transpose(\n to_cat_local_matching_bg, [0, 4, 2, 3, 1]),\n axis=-1)\n all_to_cat += [\n to_cat_local_matching_bg, to_cat_global_matching_bg\n ]\n to_cat_current_frame_embedding = paddle.expand(\n paddle.unsqueeze(current_frame_embedding[n], axis=0),\n [obj_num, -1, -1, -1])\n to_cat_prev_frame_embedding = paddle.expand(\n paddle.unsqueeze(previous_frame_embedding[n], axis=0),\n [obj_num, -1, -1, -1])\n to_cat_prev_frame_embedding_fg = to_cat_prev_frame_embedding * seq_previous_frame_label",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:238-256"
+ },
+ "7551": {
+ "file_id": 555,
+ "content": "This code segment appears to be part of a computer vision model that handles segmentation for video frames. It seems to be working with object instances, their previous and current frame embeddings, and global/local matching backgrounds. The code is performing reshaping operations and expansions on tensors, and calculating local and global matching backgrounds for the current frame's object instance. Overall, it appears to be a complex segment of a larger AI-based video processing pipeline.",
+ "type": "comment"
+ },
+ "7552": {
+ "file_id": 555,
+ "content": " to_cat_prev_frame_embedding_bg = to_cat_prev_frame_embedding * (\n 1 - seq_previous_frame_label)\n all_to_cat += [\n to_cat_current_frame_embedding,\n to_cat_prev_frame_embedding_fg,\n to_cat_prev_frame_embedding_bg\n ]\n CE_input = paddle.concat(all_to_cat, axis=1)\n #Instance-level Attention\n if not self.test_mode:\n raise NotImplementedError\n else:\n attention_head = calculate_attention_head_for_eval(\n all_ref_frame_embedding,\n seq_ref_frame_labels,\n paddle.expand(\n paddle.unsqueeze(previous_frame_embedding[n],\n axis=0), [obj_num, -1, -1, -1]),\n seq_previous_frame_label,\n epsilon=self.epsilon)\n all_CE_input.append(CE_input)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:257-279"
+ },
+ "7553": {
+ "file_id": 555,
+ "content": "This code calculates attention for instance-level using previous frame embeddings and labels. It concatenates current, previous frame embedding (for foreground and background), and then applies attention on all frames in non-test mode. In test mode, it raises a NotImplementedError.",
+ "type": "comment"
+ },
+ "7554": {
+ "file_id": 555,
+ "content": " all_attention_head.append(attention_head)\n #Collaborative Ensembler\n pred = self.head(all_CE_input, all_attention_head, low_level_feat)\n dic_tmp.append(pred)\n return dic_tmp, boards",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/cfbi.py:280-286"
+ },
+ "7555": {
+ "file_id": 555,
+ "content": "This code snippet is part of a machine learning model. It appends the \"attention_head\" to the list \"all_attention_head\", then passes the combined inputs along with \"low_level_feat\" to a \"head\" function, and appends its output to \"dic_tmp\". Finally, it returns both \"dic_tmp\" and \"boards\".",
+ "type": "comment"
+ },
+ "7556": {
+ "file_id": 556,
+ "content": "/paddlevideo/modeling/framework/segment/utils.py",
+ "type": "filepath"
+ },
+ "7557": {
+ "file_id": 556,
+ "content": "This code uses PaddleVideo for video segment matching, ASPP-based deep learning models for object size determination and feature extraction, handles padding, computes distances, prepares data, performs feature selection and masking, and utilizes parallel processing in PaddlePaddle.",
+ "type": "summary"
+ },
+ "7558": {
+ "file_id": 556,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\ndef foreground2background(dis, obj_num):\n if obj_num == 1:\n return dis\n bg_dis = []\n for i in range(obj_num):\n obj_back = []\n for j in range(obj_num):\n if i == j:\n continue\n obj_back.append(paddle.unsqueeze(dis[j], axis=0))\n obj_back = paddle.concat(x=obj_back, axis=1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:1-31"
+ },
+ "7559": {
+ "file_id": 556,
+ "content": "This code defines a function \"foreground2background\" that takes distance (dis) and object number (obj_num) as inputs. It returns the background distances for each foreground object when obj_num is greater than 1 by concatenating the unsqueezed distance of other objects along axis 1.",
+ "type": "comment"
+ },
+ "7560": {
+ "file_id": 556,
+ "content": " obj_back = paddle.min(x=obj_back, axis=1, keepdim=True)\n bg_dis.append(obj_back)\n bg_dis = paddle.concat(x=bg_dis, axis=0)\n return bg_dis\nWRONG_LABEL_PADDING_DISTANCE = 5e4\n#GLOBAL_DIST_MAP\ndef _pairwise_distances(x, x2, y, y2):\n \"\"\"\n Computes pairwise squared l2 distances between tensors x and y.\n Args:\n x: [n, feature_dim].\n y: [m, feature_dim].\n Returns:\n d: [n, m].\n \"\"\"\n xs = x2\n ys = y2\n xs = paddle.unsqueeze(xs, axis=1)\n ys = paddle.unsqueeze(ys, axis=0)\n d = xs + ys - 2. * paddle.matmul(x, y, transpose_y=True)\n return d\ndef _flattened_pairwise_distances(reference_embeddings, ref_square,\n query_embeddings, query_square):\n \"\"\"\n Calculates flattened tensor of pairwise distances between ref and query.\n Args:\n reference_embeddings: [..., embedding_dim],\n the embedding vectors for the reference frame\n query_embeddings: [..., embedding_dim],\n the embedding vectors for the query frames.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:32-68"
+ },
+ "7561": {
+ "file_id": 556,
+ "content": "This function calculates the pairwise squared L2 distances between tensors x and y, returns them in a matrix d. The function takes x and y as input, which are [n, feature_dim] and [m, feature_dim] respectively. It then performs matrix calculations to compute the pairwise distances and returns d, which is of size [n, m].",
+ "type": "comment"
+ },
+ "7562": {
+ "file_id": 556,
+ "content": " Returns:\n dists: [reference_embeddings.size / embedding_dim, query_embeddings.size / embedding_dim]\n \"\"\"\n dists = _pairwise_distances(query_embeddings, query_square,\n reference_embeddings, ref_square)\n return dists\ndef _nn_features_per_object_for_chunk(reference_embeddings, ref_square,\n query_embeddings, query_square,\n wrong_label_mask):\n \"\"\"Extracts features for each object using nearest neighbor attention.\n Args:\n reference_embeddings: [n_chunk, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings: [m_chunk, embedding_dim],\n the embedding vectors for the query frames.\n wrong_label_mask: [n_objects, n_chunk],\n the mask for pixels not used for matching.\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m_chunk, n_objects, n_chunk].\n \"\"\"\n if reference_embeddings.dtype == \"float16\":",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:69-92"
+ },
+ "7563": {
+ "file_id": 556,
+ "content": "This code computes pairwise distances between query and reference embeddings, then extracts features for each object using nearest neighbor attention. It takes embedding vectors for the reference frame and query frames as input, along with a mask for pixels not used for matching. The output is a tensor of nearest neighbor features shape [m_chunk, n_objects, n_chunk]. The code also checks the dtype of reference_embeddings to handle float16 data.",
+ "type": "comment"
+ },
+ "7564": {
+ "file_id": 556,
+ "content": " wrong_label_mask = paddle.cast(wrong_label_mask, dtype=\"float16\")\n else:\n wrong_label_mask = paddle.cast(wrong_label_mask, dtype=\"float32\")\n reference_embeddings_key = reference_embeddings\n query_embeddings_key = query_embeddings\n dists = _flattened_pairwise_distances(reference_embeddings_key, ref_square,\n query_embeddings_key, query_square)\n dists = (paddle.unsqueeze(dists, axis=1) +\n paddle.unsqueeze(wrong_label_mask, axis=0) *\n WRONG_LABEL_PADDING_DISTANCE)\n features = paddle.min(dists, axis=2, keepdim=True)\n return features\ndef _nearest_neighbor_features_per_object_in_chunks(reference_embeddings_flat,\n query_embeddings_flat,\n reference_labels_flat,\n n_chunks):\n \"\"\"Calculates the nearest neighbor features per object in chunks to save mem.\n Uses chunking to bound the memory use.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:93-113"
+ },
+ "7565": {
+ "file_id": 556,
+ "content": "This function calculates the nearest neighbor features per object in chunks to save memory. It takes reference embeddings, query embeddings, and reference labels as inputs. The function first casts the wrong_label_mask based on its type (float16 or float32). Then it calculates pairwise distances between reference and query embeddings. Distances for incorrect matches are set to a specific padding distance using wrong_label_mask. Finally, it returns the features by taking the minimum value across chunks in each dimension.",
+ "type": "comment"
+ },
+ "7566": {
+ "file_id": 556,
+ "content": " Args:\n reference_embeddings_flat: [n, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings_flat: [m, embedding_dim],\n the embedding vectors for the query frames.\n reference_labels_flat: [n, n_objects],\n the class labels of the reference frame.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: [m, n_objects, n].\n \"\"\"\n feature_dim, embedding_dim = query_embeddings_flat.shape\n chunk_size = int(np.ceil(float(feature_dim) / n_chunks))\n wrong_label_mask = reference_labels_flat < 0.1\n wrong_label_mask = paddle.transpose(x=wrong_label_mask, perm=[1, 0])\n ref_square = paddle.sum(paddle.pow(reference_embeddings_flat, 2), axis=1)\n query_square = paddle.sum(paddle.pow(query_embeddings_flat, 2), axis=1)\n all_features = []\n for n in range(n_chunks):\n if n_chunks == 1:\n query_embeddings_flat_chunk = query_embeddings_flat",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:114-138"
+ },
+ "7567": {
+ "file_id": 556,
+ "content": "This function computes the features for a set of query frames against a reference frame. It takes in embedding vectors for reference and query frames, as well as their respective class labels. The function uses chunking to handle large feature dimensions, with the number of chunks adjustable by the user. It returns a tensor of shape [m, n_objects, n] which represents the features for each query frame.",
+ "type": "comment"
+ },
+ "7568": {
+ "file_id": 556,
+ "content": " query_square_chunk = query_square\n chunk_start = 0\n else:\n chunk_start = n * chunk_size\n chunk_end = (n + 1) * chunk_size\n query_square_chunk = query_square[chunk_start:chunk_end]\n if query_square_chunk.shape[0] == 0:\n continue\n query_embeddings_flat_chunk = query_embeddings_flat[\n chunk_start:chunk_end]\n features = _nn_features_per_object_for_chunk(\n reference_embeddings_flat, ref_square, query_embeddings_flat_chunk,\n query_square_chunk, wrong_label_mask)\n all_features.append(features)\n if n_chunks == 1:\n nn_features = all_features[0]\n else:\n nn_features = paddle.concat(all_features, axis=0)\n return nn_features\ndef global_matching(reference_embeddings,\n query_embeddings,\n reference_labels,\n n_chunks=100,\n dis_bias=0.,\n ori_size=None,\n atrous_rate=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:139-167"
+ },
+ "7569": {
+ "file_id": 556,
+ "content": "This function is performing global matching on query embeddings and reference embeddings. It breaks down the embeddings into chunks, calculates features for each chunk, and concatenates these features to get the final nn_features. The number of chunks is determined by n_chunks, which default to 100. If n_chunks = 1, it returns the features from the only chunk.",
+ "type": "comment"
+ },
+ "7570": {
+ "file_id": 556,
+ "content": " use_float16=True,\n atrous_obj_pixel_num=0):\n \"\"\"\n Calculates the distance to the nearest neighbor per object.\n For every pixel of query_embeddings calculate the distance to the\n nearest neighbor in the (possibly subsampled) reference_embeddings per object.\n Args:\n reference_embeddings: [height, width, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings: [height, width,\n embedding_dim], the embedding vectors for the query frames.\n reference_labels: [height, width, obj_nums],\n the class labels of the reference frame.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n dis_bias: [n_objects], foreground and background bias\n ori_size: (ori_height, ori_width),\n the original spatial size. If \"None\", (ori_height, ori_width) = (height, width).\n atrous_rate: Integer, the atrous rate of reference_embeddings.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:168-186"
+ },
+ "7571": {
+ "file_id": 556,
+ "content": "This code calculates the distance to the nearest neighbor per object for query_embeddings and reference_embeddings, given class labels and other parameters. It uses chunks to save memory and takes into account the atrous rate of reference_embeddings.",
+ "type": "comment"
+ },
+ "7572": {
+ "file_id": 556,
+ "content": " use_float16: Bool, if \"True\", use float16 type for matching.\n Returns:\n nn_features: [1, ori_height, ori_width, n_objects, feature_dim].\n \"\"\"\n assert (reference_embeddings.shape[:2] == reference_labels.shape[:2])\n if use_float16:\n query_embeddings = paddle.cast(query_embeddings, dtype=\"float16\")\n reference_embeddings = paddle.cast(reference_embeddings,\n dtype=\"float16\")\n h, w, embedding_dim = query_embeddings.shape\n obj_nums = reference_labels.shape[2]\n if atrous_rate > 1:\n h_pad = (atrous_rate - h % atrous_rate) % atrous_rate\n w_pad = (atrous_rate - w % atrous_rate) % atrous_rate\n selected_points = paddle.zeros([h + h_pad, w + w_pad])\n selected_points = selected_points.view(\n (h + h_pad) // atrous_rate, atrous_rate, (w + w_pad) // atrous_rate,\n atrous_rate)\n selected_points[:, 0, :, 0] = 1.\n selected_points = paddle.reshape(selected_points,\n [h + h_pad, w + w_pad, 1])[:h, :w]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:187-209"
+ },
+ "7573": {
+ "file_id": 556,
+ "content": "This code snippet calculates and pads the selected points for spatial pyramid pooling in a segmentation model. It checks if float16 is used, then prepares padding based on the atrous rate. The resulting tensor of selected points is reshaped to match the input shape before returning it.",
+ "type": "comment"
+ },
+ "7574": {
+ "file_id": 556,
+ "content": " is_big_obj = (paddle.sum(\n reference_labels,\n axis=(0, 1))) > (atrous_obj_pixel_num * atrous_rate**2)\n reference_labels[:, :,\n is_big_obj] = reference_labels[:, :,\n is_big_obj] * selected_points\n reference_embeddings_flat = paddle.reshape(reference_embeddings,\n [-1, embedding_dim])\n reference_labels_flat = paddle.reshape(reference_labels, [-1, obj_nums])\n query_embeddings_flat = paddle.reshape(query_embeddings,\n [-1, embedding_dim])\n all_ref_fg = paddle.sum(reference_labels_flat, axis=1, keepdim=True) > 0.9\n reference_labels_flat = paddle.reshape(\n paddle.masked_select(reference_labels_flat,\n paddle.expand(all_ref_fg, [-1, obj_nums])),\n [-1, obj_nums])\n if reference_labels_flat.shape[0] == 0:\n return paddle.ones([1, h, w, obj_nums, 1])\n reference_embeddings_flat = paddle.reshape(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:210-230"
+ },
+ "7575": {
+ "file_id": 556,
+ "content": "The code is implementing a segmentation method in the PaddleVideo library. It first determines if an object is big or small based on the sum of reference labels. Then, it reshapes the reference embeddings and labels for further processing. It checks if any reference labels are present and returns default values if none are found.",
+ "type": "comment"
+ },
+ "7576": {
+ "file_id": 556,
+ "content": " paddle.masked_select(reference_embeddings_flat,\n paddle.expand(all_ref_fg, [-1, embedding_dim])),\n [-1, embedding_dim])\n nn_features = _nearest_neighbor_features_per_object_in_chunks(\n reference_embeddings_flat, query_embeddings_flat, reference_labels_flat,\n n_chunks)\n nn_features_reshape = paddle.reshape(nn_features, [1, h, w, obj_nums, 1])\n nn_features_reshape = (\n F.sigmoid(nn_features_reshape +\n paddle.reshape(dis_bias, [1, 1, 1, -1, 1])) - 0.5) * 2\n #TODO: ori_size is not None\n if use_float16:\n nn_features_reshape = paddle.cast(nn_features_reshape, dtype=\"float32\")\n return nn_features_reshape\ndef global_matching_for_eval(all_reference_embeddings,\n query_embeddings,\n all_reference_labels,\n n_chunks=20,\n dis_bias=0.,\n ori_size=None,\n atrous_rate=1,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:231-257"
+ },
+ "7577": {
+ "file_id": 556,
+ "content": "This function performs nearest neighbor feature extraction for video segment matching using reference and query embeddings, reference labels, and other parameters such as number of chunks, displacement bias, original size, and atrous rate. It returns the normalized nearest neighbor features in a reshaped format.",
+ "type": "comment"
+ },
+ "7578": {
+ "file_id": 556,
+ "content": " use_float16=True,\n atrous_obj_pixel_num=0):\n \"\"\"\n Calculates the distance to the nearest neighbor per object.\n For every pixel of query_embeddings calculate the distance to the\n nearest neighbor in the (possibly subsampled) reference_embeddings per object.\n Args:\n all_reference_embeddings: A list of reference_embeddings,\n each with size [height, width, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings: [n_query_images, height, width,\n embedding_dim], the embedding vectors for the query frames.\n all_reference_labels: A list of reference_labels,\n each with size [height, width, obj_nums],\n the class labels of the reference frame.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n dis_bias: [n_objects], foreground and background bias\n ori_size: (ori_height, ori_width),\n the original spatial size. If \"None\", (ori_height, ori_width) = (height, width).",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:258-277"
+ },
+ "7579": {
+ "file_id": 556,
+ "content": "This code calculates the distance to the nearest neighbor per object for query embeddings in a list of reference embeddings, considering potentially subsampled frames. It takes query_embeddings of size [n_query_images, height, width, embedding_dim], all_reference_embeddings and all_reference_labels lists with size [height, width, obj_nums] each, n_chunks, dis_bias, and ori_size as input arguments.",
+ "type": "comment"
+ },
+ "7580": {
+ "file_id": 556,
+ "content": " atrous_rate: Integer, the atrous rate of reference_embeddings.\n use_float16: Bool, if \"True\", use float16 type for matching.\n Returns:\n nn_features: [n_query_images, ori_height, ori_width, n_objects, feature_dim].\n \"\"\"\n h, w, embedding_dim = query_embeddings.shape\n obj_nums = all_reference_labels[0].shape[2]\n all_reference_embeddings_flat = []\n all_reference_labels_flat = []\n ref_num = len(all_reference_labels)\n n_chunks *= ref_num\n if atrous_obj_pixel_num > 0:\n if atrous_rate > 1:\n h_pad = (atrous_rate - h % atrous_rate) % atrous_rate\n w_pad = (atrous_rate - w % atrous_rate) % atrous_rate\n selected_points = paddle.zeros([h + h_pad, w + w_pad])\n selected_points = paddle.reshape(\n selected_points, [(h + h_pad) // atrous_rate, atrous_rate,\n (w + w_pad) // atrous_rate, atrous_rate])\n selected_points[:, 0, :, 0] = 1.\n selected_points = paddle.reshape(selected_points,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:278-299"
+ },
+ "7581": {
+ "file_id": 556,
+ "content": "This function is responsible for creating a tensor of reference embeddings and labels for a given set of query embeddings, based on the provided atrous rate. The function first calculates the shape of the input tensors, then initializes empty lists for flat versions of reference embeddings and labels. It then determines the padding needed to match the atrous rate, creates a selection matrix with ones at the selected points, and reshapes it according to the atrous rate. Finally, it prepares the tensor for matching by flattening the reference embeddings and labels lists.",
+ "type": "comment"
+ },
+ "7582": {
+ "file_id": 556,
+ "content": " [h + h_pad, w + w_pad, 1])[:h, :w]\n for reference_embeddings, reference_labels, idx in zip(\n all_reference_embeddings, all_reference_labels, range(ref_num)):\n if atrous_rate > 1:\n is_big_obj = paddle.sum(\n reference_labels,\n axis=(0, 1)) > (atrous_obj_pixel_num * atrous_rate**2)\n is_big_obj = list(np.array(is_big_obj))\n for j in range(len(is_big_obj)):\n if is_big_obj[j] == True:\n reference_labels[:, :, j:j +\n 1] = reference_labels[:, :, j:j +\n 1] * selected_points\n reference_embeddings_flat = paddle.reshape(reference_embeddings,\n [-1, embedding_dim])\n reference_labels_flat = paddle.reshape(reference_labels,\n [-1, obj_nums])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:300-318"
+ },
+ "7583": {
+ "file_id": 556,
+ "content": "This code segment appears to be a part of image segmentation or object detection algorithm. It processes reference embeddings and labels, potentially for each detected object in the image. The atrous rate determines if an object is big or small, with larger objects being processed separately by multiplying selected points to corresponding regions in reference_labels. The embeddings are flattened into 1D arrays, as well as reference_labels.",
+ "type": "comment"
+ },
+ "7584": {
+ "file_id": 556,
+ "content": " all_reference_embeddings_flat.append(reference_embeddings_flat)\n all_reference_labels_flat.append(reference_labels_flat)\n reference_embeddings_flat = paddle.concat(\n x=all_reference_embeddings_flat, axis=0)\n reference_labels_flat = paddle.concat(x=all_reference_labels_flat,\n axis=0)\n else:\n if ref_num == 1:\n reference_embeddings, reference_labels = all_reference_embeddings[\n 0], all_reference_labels[0]\n if atrous_rate > 1:\n h_pad = (atrous_rate - h % atrous_rate) % atrous_rate\n w_pad = (atrous_rate - w % atrous_rate) % atrous_rate\n if h_pad > 0 or w_pad > 0:\n reference_embeddings = F.pad(reference_embeddings,\n [0, h_pad, 0, w_pad, 0, 0])\n reference_labels = F.pad(reference_labels,\n [0, h_pad, 0, w_pad, 0, 0])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:320-338"
+ },
+ "7585": {
+ "file_id": 556,
+ "content": "The code concatenates reference embeddings and labels, then pads them if necessary based on the atrous rate. If there is only one reference, it directly selects the first item from all_reference_embeddings and all_reference_labels lists.",
+ "type": "comment"
+ },
+ "7586": {
+ "file_id": 556,
+ "content": " reference_embeddings = paddle.reshape(\n reference_embeddings,\n [(h + h_pad) // atrous_rate, atrous_rate,\n (w + w_pad) // atrous_rate, atrous_rate, 32])\n reference_labels = paddle.reshape(\n reference_labels,\n [(h + h_pad) // atrous_rate, atrous_rate,\n (w + w_pad) // atrous_rate, atrous_rate, -1])\n reference_embeddings = paddle.reshape(\n reference_embeddings[:, 0, :, 0, :],\n reference_embeddings[:, 0, :, 0, :].shape)\n reference_labels = paddle.reshape(\n reference_labels[:, 0, :, 0, :],\n reference_labels[:, 0, :, 0, :].shape)\n reference_embeddings_flat = paddle.reshape(reference_embeddings,\n [-1, embedding_dim])\n reference_labels_flat = paddle.reshape(reference_labels,\n [-1, obj_nums])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:339-356"
+ },
+ "7587": {
+ "file_id": 556,
+ "content": "This code reshapes the reference embeddings and labels to match a specific pattern, then flattens the reference embeddings while preserving their data type and shape.",
+ "type": "comment"
+ },
+ "7588": {
+ "file_id": 556,
+ "content": " else:\n for reference_embeddings, reference_labels, idx in zip(\n all_reference_embeddings, all_reference_labels,\n range(ref_num)):\n if atrous_rate > 1:\n h_pad = (atrous_rate - h % atrous_rate) % atrous_rate\n w_pad = (atrous_rate - w % atrous_rate) % atrous_rate\n if h_pad > 0 or w_pad > 0:\n reference_embeddings = F.pad(reference_embeddings,\n [0, h_pad, 0, w_pad, 0, 0])\n reference_labels = F.pad(reference_labels,\n [0, h_pad, 0, w_pad, 0, 0])\n reference_embeddings = paddle.reshape(\n reference_embeddings,\n [(h + h_pad) // atrous_rate, atrous_rate,\n (w + w_pad) // atrous_rate, atrous_rate, -1])\n reference_labels = paddle.reshape(\n reference_labels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:357-375"
+ },
+ "7589": {
+ "file_id": 556,
+ "content": "This code segment handles the case where atrous_rate is greater than 1. It pads reference embeddings and labels if needed, then reshapes them to have a shape compatible with Atrous Spatial Pyramid Pooling (ASPP) in deep learning models for image classification or detection tasks.",
+ "type": "comment"
+ },
+ "7590": {
+ "file_id": 556,
+ "content": " [(h + h_pad) // atrous_rate, atrous_rate,\n (w + w_pad) // atrous_rate, atrous_rate, -1])\n reference_embeddings = paddle.reshape(\n reference_embeddings[:, 0, :, 0, :],\n reference_embeddings[:, 0, :, 0, :].shape)\n reference_labels = paddle.reshape(\n reference_labels[:, 0, :, 0, :],\n reference_labels[:, 0, :, 0, :].shape)\n reference_embeddings_flat = paddle.reshape(\n reference_embeddings, [-1, embedding_dim])\n reference_labels_flat = paddle.reshape(reference_labels,\n [-1, obj_nums])\n all_reference_embeddings_flat.append(reference_embeddings_flat)\n all_reference_labels_flat.append(reference_labels_flat)\n reference_embeddings_flat = paddle.concat(\n all_reference_embeddings_flat, axis=0)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:376-394"
+ },
+ "7591": {
+ "file_id": 556,
+ "content": "This code reshapes the reference embeddings and labels into a flattened format, appends them to lists, and then concatenates all the flattened reference embeddings along axis 0. This is likely for use in a deep learning model that requires the data in this specific format for training or prediction.",
+ "type": "comment"
+ },
+ "7592": {
+ "file_id": 556,
+ "content": " reference_labels_flat = paddle.concat(all_reference_labels_flat,\n axis=0)\n query_embeddings_flat = paddle.reshape(query_embeddings,\n [-1, embedding_dim])\n all_ref_fg = paddle.sum(reference_labels_flat, axis=1, keepdim=True) > 0.9\n reference_labels_flat = paddle.reshape(\n paddle.masked_select(reference_labels_flat,\n paddle.expand(all_ref_fg, [-1, obj_nums])),\n [-1, obj_nums])\n if reference_labels_flat.shape[0] == 0:\n return paddle.ones([1, h, w, obj_nums, 1])\n reference_embeddings_flat = paddle.reshape(\n paddle.masked_select(reference_embeddings_flat,\n paddle.expand(all_ref_fg, [-1, embedding_dim])),\n [-1, embedding_dim])\n if use_float16:\n query_embeddings_flat = paddle.cast(query_embeddings_flat,\n dtype=\"float16\")\n reference_embeddings_flat = paddle.cast(reference_embeddings_flat,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:395-415"
+ },
+ "7593": {
+ "file_id": 556,
+ "content": "This code segment performs feature selection and reshaping of query and reference embeddings for the segment matching process. It concatenates all reference labels, flattens the query embeddings, masks the selected reference labels and embeddings based on a threshold, and finally reshapes them before returning a tensor of ones if no references are found or casting the embeddings to float16 datatype if specified.",
+ "type": "comment"
+ },
+ "7594": {
+ "file_id": 556,
+ "content": " dtype=\"float16\")\n nn_features = _nearest_neighbor_features_per_object_in_chunks(\n reference_embeddings_flat, query_embeddings_flat, reference_labels_flat,\n n_chunks)\n nn_features_reshape = paddle.reshape(nn_features, [1, h, w, obj_nums, 1])\n nn_features_reshape = (\n F.sigmoid(nn_features_reshape +\n paddle.reshape(dis_bias, [1, 1, 1, -1, 1])) - 0.5) * 2\n # TODO: ori_size is not None\n if use_float16:\n nn_features_reshape = paddle.cast(nn_features_reshape, dtype=\"float32\")\n return nn_features_reshape\n#LOCAL_DIST_MAP\ndef local_pairwise_distances(x,\n y,\n max_distance=9,\n atrous_rate=1,\n allow_downsample=False):\n \"\"\"Computes pairwise squared l2 distances using a local search window.\n Use for-loop for saving memory.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:416-442"
+ },
+ "7595": {
+ "file_id": 556,
+ "content": "This code calculates pairwise squared L2 distances using a local search window, and then computes the nearest neighbor features for each object in image chunks. The result is reshaped into an appropriate format and can be used for further processing or analysis.",
+ "type": "comment"
+ },
+ "7596": {
+ "file_id": 556,
+ "content": " y: Float32 tensor of shape [height, width, feature_dim].\n max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n atrous_rate: Integer, the atrous rate of local matching.\n allow_downsample: Bool, if \"True\", downsample x and y\n with a stride of 2.\n Returns:\n Float32 distances tensor of shape [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n if allow_downsample:\n ori_height = x.shape[0]\n ori_width = x.shape[1]\n x = paddle.unsqueeze(paddle.transpose(x, [2, 0, 1]), axis=0)\n y = paddle.unsqueeze(paddle.transpose(y, [2, 0, 1]), axis=0)\n down_size = (int(ori_height / 2) + 1, int(ori_width / 2) + 1)\n x = F.interpolate(x,\n size=down_size,\n mode='bilinear',\n align_corners=True)\n y = F.interpolate(y,\n size=down_size,\n mode='bilinear',",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:443-464"
+ },
+ "7597": {
+ "file_id": 556,
+ "content": "This function takes in a tensor 'x' and 'y', along with parameters such as max_distance, atrous_rate, and allow_downsample. It returns a distances tensor of shape [height, width, (2 * max_distance + 1) ** 2]. If downsampling is allowed, the original height and width are saved and the tensors 'x' and 'y' are reshaped. Then, using bilinear interpolation, 'x' and 'y' are downsampled to half their size while preserving values at borders.",
+ "type": "comment"
+ },
+ "7598": {
+ "file_id": 556,
+ "content": " align_corners=True)\n x = paddle.unsqueeze(paddle.transpose(x, [1, 2, 0]), axis=0)\n y = paddle.unsqueeze(paddle.transpose(y, [1, 2, 0]), axis=0)\n pad_max_distance = max_distance - max_distance % atrous_rate\n # no change pad\n padded_y = F.pad(y, (0, 0, pad_max_distance, pad_max_distance,\n pad_max_distance, pad_max_distance),\n value=WRONG_LABEL_PADDING_DISTANCE)\n height, width, _ = x.shape\n dists = []\n for y in range(2 * pad_max_distance // atrous_rate + 1):\n y_start = y * atrous_rate\n y_end = y_start + height\n y_slice = padded_y[y_start:y_end]\n for x in range(2 * max_distance + 1):\n x_start = x * atrous_rate\n x_end = x_start + width\n offset_y = y_slice[:, x_start:x_end]\n dist = paddle.sum(paddle.pow((x - offset_y), 2), axis=2)\n dists.append(dist)\n dists = paddle.stack(dists, axis=2)\n return dists\ndef local_pairwise_distances_parallel(x,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:465-492"
+ },
+ "7599": {
+ "file_id": 556,
+ "content": "This code computes local pairwise distances between the input tensors x and y, accounting for atrous dilation. It first pads y with wrong label padding distance to match the size of x. Then it loops through the range of possible offsets for each pixel and calculates the sum of squared differences between the current pixel and all potential offsets in y. These distances are then stacked along the channel axis before being returned.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/76.json b/docs/data/76.json
new file mode 100644
index 000000000..333fe9ce9
--- /dev/null
+++ b/docs/data/76.json
@@ -0,0 +1,545 @@
+{
+ "7600": {
+ "file_id": 556,
+ "content": " y,\n max_distance=9,\n atrous_rate=1,\n allow_downsample=True):\n \"\"\"Computes pairwise squared l2 distances using a local search window.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].\n y: Float32 tensor of shape [height, width, feature_dim].\n max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n atrous_rate: Integer, the atrous rate of local matching.\n allow_downsample: Bool, if \"True\", downsample x and y\n with a stride of 2.\n Returns:\n Float32 distances tensor of shape [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n ori_height, ori_width, _ = x.shape\n x = paddle.unsqueeze(paddle.transpose(x, [2, 0, 1]), axis=0)\n y = paddle.unsqueeze(paddle.transpose(y, [2, 0, 1]), axis=0)\n if allow_downsample:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:493-513"
+ },
+ "7601": {
+ "file_id": 556,
+ "content": "This function computes pairwise squared L2 distances using a local search window. It takes two tensors x and y of shape [height, width, feature_dim] as input. The maximum distance (max\\_distance) in pixel coordinates per dimension is considered in the search window. Atrous rate determines the local matching rate. If downsampling is allowed, the function downsamples the tensors with a stride of 2. It returns a float32 distances tensor of shape [height, width, (2 * max_distance + 1) ** 2].",
+ "type": "comment"
+ },
+ "7602": {
+ "file_id": 556,
+ "content": " down_size = (int(ori_height / 2) + 1, int(ori_width / 2) + 1)\n x = F.interpolate(x,\n size=down_size,\n mode='bilinear',\n align_corners=True)\n y = F.interpolate(y,\n size=down_size,\n mode='bilinear',\n align_corners=True)\n _, channels, height, width = x.shape\n x2 = paddle.reshape(paddle.sum(paddle.pow(x, 2), axis=1),\n [height, width, 1])\n y2 = paddle.reshape(paddle.sum(paddle.pow(y, 2), axis=1),\n [1, 1, height, width])\n pad_max_distance = max_distance - max_distance % atrous_rate\n # no change pad\n padded_y = F.pad(y, (pad_max_distance, pad_max_distance, pad_max_distance,\n pad_max_distance))\n padded_y2 = F.pad(y2, (pad_max_distance, pad_max_distance, pad_max_distance,\n pad_max_distance),\n value=WRONG_LABEL_PADDING_DISTANCE)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:514-537"
+ },
+ "7603": {
+ "file_id": 556,
+ "content": "The code resizes the input tensors x and y to a downsized version of half the original size using bilinear interpolation. It then calculates the squared values for x and y, reshapes them, pads the tensors with WRONG_LABEL_PADDING_DISTANCE to match the atrous rate, and assigns them to padded_y and padded_y2 variables.",
+ "type": "comment"
+ },
+ "7604": {
+ "file_id": 556,
+ "content": " offset_y = paddle.transpose(\n paddle.reshape(\n F.unfold(x=padded_y,\n kernel_sizes=[height, width],\n strides=[atrous_rate, atrous_rate]),\n [channels, height * width, -1]), [1, 0, 2])\n offset_y2 = paddle.reshape(\n F.unfold(padded_y2,\n kernel_sizes=[height, width],\n strides=[atrous_rate, atrous_rate]), [height, width, -1])\n x = paddle.transpose(paddle.reshape(x, [channels, height * width, -1]),\n [1, 2, 0])\n dists = x2 + offset_y2 - 2. * paddle.reshape(paddle.matmul(x, offset_y),\n [height, width, -1])\n return dists\ndef local_matching(prev_frame_embedding,\n query_embedding,\n prev_frame_labels,\n dis_bias=0.,\n multi_local_distance=[15],\n ori_size=None,\n atrous_rate=1,\n use_float16=True,\n allow_downsample=True,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:539-566"
+ },
+ "7605": {
+ "file_id": 556,
+ "content": "This code snippet calculates the distance between embeddings of frames to measure similarity. It takes in two frame embeddings, their corresponding labels, and several optional parameters, including an atomic number for local distances, a size parameter, and whether to allow downsampling. The function uses Paddle's unfold operation to reshape the data, then calculates distances between these reshaped embeddings using a formula that involves matmul (matrix multiplication) operations. The final step is returning the calculated distances.",
+ "type": "comment"
+ },
+ "7606": {
+ "file_id": 556,
+ "content": " allow_parallel=True):\n \"\"\"Computes nearest neighbor features while only allowing local matches.\n Args:\n prev_frame_embedding: [height, width, embedding_dim],\n the embedding vectors for the last frame.\n query_embedding: [height, width, embedding_dim],\n the embedding vectors for the query frames.\n prev_frame_labels: [height, width, n_objects],\n the class labels of the previous frame.\n multi_local_distance: A list of Integer,\n a list of maximum distance allowed for local matching.\n ori_size: (ori_height, ori_width),\n the original spatial size. If \"None\", (ori_height, ori_width) = (height, width).\n atrous_rate: Integer, the atrous rate of local matching.\n use_float16: Bool, if \"True\", use float16 type for matching.\n allow_downsample: Bool, if \"True\", downsample prev_frame_embedding and query_embedding\n with a stride of 2.\n allow_parallel: Bool, if \"True\", do matching in a parallel way. If \"False\", do matching in",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:567-584"
+ },
+ "7607": {
+ "file_id": 556,
+ "content": "This code computes nearest neighbor features for local matching in video segmentation. It takes embedding vectors, class labels, and a list of maximum distances as input. The function allows downsampling and parallel processing.",
+ "type": "comment"
+ },
+ "7608": {
+ "file_id": 556,
+ "content": " a for-loop way, which will save GPU memory.\n Returns:\n nn_features: A float32 np.array of nearest neighbor features of shape\n [1, height, width, n_objects, 1].\n \"\"\"\n max_distance = multi_local_distance[-1]\n if ori_size is None:\n height, width = prev_frame_embedding.shape[:2]\n ori_size = (height, width)\n obj_num = prev_frame_labels.shape[2]\n pad = paddle.ones([1]) * WRONG_LABEL_PADDING_DISTANCE\n if use_float16:\n query_embedding = paddle.cast(query_embedding, dtype=\"float16\")\n prev_frame_embedding = paddle.cast(prev_frame_embedding,\n dtype=\"float16\")\n pad = paddle.cast(pad, dtype=\"float16\")\n if allow_parallel:\n d = local_pairwise_distances_parallel(query_embedding,\n prev_frame_embedding,\n max_distance=max_distance,\n atrous_rate=atrous_rate,\n allow_downsample=allow_downsample)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:585-609"
+ },
+ "7609": {
+ "file_id": 556,
+ "content": "This function calculates nearest neighbor features by using local pairwise distances in a parallel manner, with options to cast data types and allow downsampling. It takes query and previous frame embeddings as input, and returns nearest neighbor features of shape [1, height, width, n_objects, 1].",
+ "type": "comment"
+ },
+ "7610": {
+ "file_id": 556,
+ "content": " else:\n d = local_pairwise_distances(query_embedding,\n prev_frame_embedding,\n max_distance=max_distance,\n atrous_rate=atrous_rate,\n allow_downsample=allow_downsample)\n height, width = d.shape[:2]\n labels = paddle.unsqueeze(paddle.transpose(prev_frame_labels, [2, 0, 1]), 1)\n labels = paddle.unsqueeze(paddle.transpose(prev_frame_labels, [2, 0, 1]),\n axis=1)\n if (height, width) != ori_size:\n labels = F.interpolate(labels, size=(height, width), mode='nearest')\n pad_max_distance = max_distance - max_distance % atrous_rate\n atrous_max_distance = pad_max_distance // atrous_rate\n #no change pad\n padded_labels = F.pad(labels, (\n pad_max_distance,\n pad_max_distance,\n pad_max_distance,\n pad_max_distance,\n ),\n mode='constant',\n value=0)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:610-635"
+ },
+ "7611": {
+ "file_id": 556,
+ "content": "This code calculates pairwise distances between query and previous frame embeddings. If the shape of the distances doesn't match the original size, it interpolates labels using nearest neighbor mode. The code then pads the labels with zeros to match the maximum distance considering the atrous rate.",
+ "type": "comment"
+ },
+ "7612": {
+ "file_id": 556,
+ "content": " offset_masks = paddle.transpose(\n paddle.reshape(\n F.unfold(padded_labels,\n kernel_sizes=[height, width],\n strides=[atrous_rate, atrous_rate]),\n [obj_num, height, width, -1]), [1, 2, 3, 0]) > 0.9\n d_tiled = paddle.expand(paddle.unsqueeze(\n d, axis=-1), [-1, -1, -1, obj_num]) # h, w, num_local_pos, obj_num\n d_masked = paddle.where(offset_masks, d_tiled, pad)\n dists = paddle.min(d_masked, axis=2)\n multi_dists = [\n paddle.unsqueeze(paddle.transpose(dists, [2, 0, 1]), axis=1)\n ] # n_objects, num_multi_local, h, w\n reshaped_d_masked = paddle.reshape(d_masked, [\n height, width, 2 * atrous_max_distance + 1, 2 * atrous_max_distance + 1,\n obj_num\n ])\n for local_dis in multi_local_distance[:-1]:\n local_dis = local_dis // atrous_rate\n start_idx = atrous_max_distance - local_dis\n end_idx = atrous_max_distance + local_dis + 1\n new_d_masked = paddle.reshape(\n reshaped_d_masked[:, :, start_idx:end_idx, start_idx:end_idx, :],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:637-662"
+ },
+ "7613": {
+ "file_id": 556,
+ "content": "This code segment applies atrous spatial pyramid pooling in a PaddlePaddle implementation. It creates offset masks, performs element-wise masking, computes minimum distances, and reshapes the data for each local distance level to perform feature extraction at different scales.",
+ "type": "comment"
+ },
+ "7614": {
+ "file_id": 556,
+ "content": " reshaped_d_masked[:, :, start_idx:end_idx,\n start_idx:end_idx, :].shape)\n new_d_masked = paddle.reshape(new_d_masked,\n [height, width, -1, obj_num])\n new_dists = paddle.min(new_d_masked, axis=2)\n new_dists = paddle.unsqueeze(paddle.transpose(new_dists, [2, 0, 1]),\n axis=1)\n multi_dists.append(new_dists)\n multi_dists = paddle.concat(multi_dists, axis=1)\n multi_dists = (F.sigmoid(multi_dists +\n paddle.reshape(dis_bias, [-1, 1, 1, 1])) - 0.5) * 2\n if use_float16:\n multi_dists = paddle.cast(multi_dists, dtype=\"float32\")\n if (height, width) != ori_size:\n multi_dists = F.interpolate(multi_dists,\n size=ori_size,\n mode='bilinear',\n align_corners=True)\n multi_dists = paddle.transpose(multi_dists, perm=[2, 3, 0, 1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:663-684"
+ },
+ "7615": {
+ "file_id": 556,
+ "content": "This code performs image segmentation by reshaping and resizing the distance matrix, calculating minimum distances, and applying sigmoid activation. It also handles cases where input size is not the original size.",
+ "type": "comment"
+ },
+ "7616": {
+ "file_id": 556,
+ "content": " multi_dists = paddle.reshape(multi_dists,\n [1, ori_size[0], ori_size[1], obj_num, -1])\n return multi_dists\ndef calculate_attention_head(ref_embedding,\n ref_label,\n prev_embedding,\n prev_label,\n epsilon=1e-5):\n ref_head = ref_embedding * ref_label\n ref_head_pos = paddle.sum(ref_head, axis=(2, 3))\n ref_head_neg = paddle.sum(ref_embedding, axis=(2, 3)) - ref_head_pos\n ref_pos_num = paddle.sum(ref_label, axis=(2, 3))\n ref_neg_num = paddle.sum(1. - ref_label, axis=(2, 3))\n ref_head_pos = ref_head_pos / (ref_pos_num + epsilon)\n ref_head_neg = ref_head_neg / (ref_neg_num + epsilon)\n prev_head = prev_embedding * prev_label\n prev_head_pos = paddle.sum(prev_head, axis=(2, 3))\n prev_head_neg = paddle.sum(prev_embedding, axis=(2, 3)) - prev_head_pos\n prev_pos_num = paddle.sum(prev_label, axis=(2, 3))\n prev_neg_num = paddle.sum(1. - prev_label, axis=(2, 3))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:685-709"
+ },
+ "7617": {
+ "file_id": 556,
+ "content": "This function calculates the attention heads for each object in a given scene. It takes reference and previous embeddings and labels as input, along with an optional epsilon value. It then computes the positional and negative head values, divides them by their respective counts (positive and negative labels), and returns the resulting attention heads for each object. The epsilon is added to avoid division by zero in the calculations.",
+ "type": "comment"
+ },
+ "7618": {
+ "file_id": 556,
+ "content": " prev_head_pos = prev_head_pos / (prev_pos_num + epsilon)\n prev_head_neg = prev_head_neg / (prev_neg_num + epsilon)\n total_head = paddle.concat(\n x=[ref_head_pos, ref_head_neg, prev_head_pos, prev_head_neg], axis=1)\n return total_head\ndef calculate_attention_head_for_eval(ref_embeddings,\n ref_labels,\n prev_embedding,\n prev_label,\n epsilon=1e-5):\n total_ref_head_pos = 0.\n total_ref_head_neg = 0.\n total_ref_pos_num = 0.\n total_ref_neg_num = 0.\n for idx in range(len(ref_embeddings)):\n ref_embedding = ref_embeddings[idx]\n ref_label = ref_labels[idx]\n ref_head = ref_embedding * ref_label\n ref_head_pos = paddle.sum(ref_head, axis=(2, 3))\n ref_head_neg = paddle.sum(ref_embedding, axis=(2, 3)) - ref_head_pos\n ref_pos_num = paddle.sum(ref_label, axis=(2, 3))\n ref_neg_num = paddle.sum(1. - ref_label, axis=(2, 3))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:710-736"
+ },
+ "7619": {
+ "file_id": 556,
+ "content": "This code calculates the attention head values for evaluation, where it sums up reference embeddings multiplied by their corresponding labels. It also accounts for positive and negative instances of reference embeddings by subtracting them from total sums. The final total_head is returned as a concatenated matrix.",
+ "type": "comment"
+ },
+ "7620": {
+ "file_id": 556,
+ "content": " total_ref_head_pos = total_ref_head_pos + ref_head_pos\n total_ref_head_neg = total_ref_head_neg + ref_head_neg\n total_ref_pos_num = total_ref_pos_num + ref_pos_num\n total_ref_neg_num = total_ref_neg_num + ref_neg_num\n ref_head_pos = total_ref_head_pos / (total_ref_pos_num + epsilon)\n ref_head_neg = total_ref_head_neg / (total_ref_neg_num + epsilon)\n prev_head = prev_embedding * prev_label\n prev_head_pos = paddle.sum(prev_head, axis=(2, 3))\n prev_head_neg = paddle.sum(prev_embedding, axis=(2, 3)) - prev_head_pos\n prev_pos_num = paddle.sum(prev_label, axis=(2, 3))\n prev_neg_num = paddle.sum(1. - prev_label, axis=(2, 3))\n prev_head_pos = prev_head_pos / (prev_pos_num + epsilon)\n prev_head_neg = prev_head_neg / (prev_neg_num + epsilon)\n total_head = paddle.concat(\n x=[ref_head_pos, ref_head_neg, prev_head_pos, prev_head_neg], axis=1)\n return total_head",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segment/utils.py:737-754"
+ },
+ "7621": {
+ "file_id": 556,
+ "content": "This code calculates and returns a total head value by accumulating reference (ref) head values and previous (prev) head values, then normalizing them. It handles potential zero-division cases with a small epsilon for stability. The resulting total head consists of reference positive (pos), reference negative (neg), previous positive (pos), and previous negative (neg) head components concatenated along axis 1.",
+ "type": "comment"
+ },
+ "7622": {
+ "file_id": 557,
+ "content": "/paddlevideo/modeling/framework/segmenters/__init__.py",
+ "type": "filepath"
+ },
+ "7623": {
+ "file_id": 557,
+ "content": "This code is part of the PaddleVideo library and defines a segmenter module. It includes three classes: BaseSegmenter, MSTCN, and ASRF. These classes are used for video frame-level feature extraction, semantic segmentation, and audio source separation respectively. The __all__ variable lists all exported names in this package.",
+ "type": "summary"
+ },
+ "7624": {
+ "file_id": 557,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom .base import BaseSegmenter\nfrom .ms_tcn import MSTCN\nfrom .asrf import ASRF\n__all__ = ['BaseSegmenter', 'MSTCN', 'ASRF']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/__init__.py:1-17"
+ },
+ "7625": {
+ "file_id": 557,
+ "content": "This code is part of the PaddleVideo library and defines a segmenter module. It includes three classes: BaseSegmenter, MSTCN, and ASRF. These classes are used for video frame-level feature extraction, semantic segmentation, and audio source separation respectively. The __all__ variable lists all exported names in this package.",
+ "type": "comment"
+ },
+ "7626": {
+ "file_id": 558,
+ "content": "/paddlevideo/modeling/framework/segmenters/asrf.py",
+ "type": "filepath"
+ },
+ "7627": {
+ "file_id": 558,
+ "content": "The PaddleVideo framework's ASRF segmentation model uses a backbone for feature extraction and head network for classification. It performs forward passes, post-processing, inference, validates using loss and F1@0.50 score, and extracts class outputs for results.",
+ "type": "summary"
+ },
+ "7628": {
+ "file_id": 558,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import SEGMENTERS\nfrom .base import BaseSegmenter\nimport paddle\nimport paddle.nn.functional as F\nfrom .utils import ASRFPostProcessing\n@SEGMENTERS.register()\nclass ASRF(BaseSegmenter):\n \"\"\"ASRF model framework.\"\"\"\n def __init__(self,\n postprocessing_method,\n boundary_threshold,\n backbone=None,\n head=None,\n loss=None):\n super().__init__(backbone=backbone, head=head, loss=loss)\n self.postprocessing_method = postprocessing_method",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/asrf.py:1-33"
+ },
+ "7629": {
+ "file_id": 558,
+ "content": "Class ASRF is a segmenter model in PaddleVideo framework. It takes arguments like postprocessing_method, boundary_threshold, backbone, head, and loss for initialization.",
+ "type": "comment"
+ },
+ "7630": {
+ "file_id": 558,
+ "content": " self.boundary_threshold = boundary_threshold\n def forward_net(self, video_feature):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n if self.backbone is not None:\n feature = self.backbone(video_feature)\n else:\n feature = video_feature\n if self.head is not None:\n network_outputs = self.head(feature)\n else:\n network_outputs = None\n return network_outputs\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n feature, label, boundary = data_batch\n # call forward\n outputs_cls, outputs_boundary = self.forward_net(feature)\n # transfer data\n outputs_cls_np = outputs_cls[-1].numpy()\n outputs_boundary_np = outputs_boundary[-1].numpy()\n # caculate loss\n if self.loss is not None:\n output_loss = self.loss(feature, outputs_cls, label,\n outputs_boundary, boundary)\n else:\n output_loss = None",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/asrf.py:34-67"
+ },
+ "7631": {
+ "file_id": 558,
+ "content": "The code defines a model for segmentation, which has a forward function and train step. It uses a backbone for feature extraction and a head network for classification. The train_step calculates loss using the defined loss function if it's not None.",
+ "type": "comment"
+ },
+ "7632": {
+ "file_id": 558,
+ "content": " # predict post process\n predicted = ASRFPostProcessing(outputs_cls_np, outputs_boundary_np,\n self.postprocessing_method)\n predicted = paddle.squeeze(predicted)\n loss_metrics = dict()\n loss_metrics['loss'] = output_loss\n loss_metrics['F1@0.50'] = self.head.get_F1_score(predicted, label)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n feature, label, boundary = data_batch\n # call forward\n outputs_cls, outputs_boundary = self.forward_net(feature)\n # transfer data\n outputs_cls_np = outputs_cls[-1].numpy()\n outputs_boundary_np = outputs_boundary[-1].numpy()\n ## caculate loss\n if self.loss is not None:\n output_loss = self.loss(feature, outputs_cls, label,\n outputs_boundary, boundary)\n else:\n output_loss = None\n # predict post process\n predicted = ASRFPostProcessing(outputs_cls_np, outputs_boundary_np,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/asrf.py:69-100"
+ },
+ "7633": {
+ "file_id": 558,
+ "content": "The code snippet represents the ASRF model's validation step. It predicts the outputs for the given inputs, calculates loss if applicable, and performs post-processing using ASRFPostProcessing function. The function then returns a dictionary of metrics including the 'loss' value and the 'F1@0.50' score.",
+ "type": "comment"
+ },
+ "7634": {
+ "file_id": 558,
+ "content": " self.postprocessing_method)\n predicted = paddle.squeeze(predicted)\n outputs_dict = dict()\n outputs_dict['loss'] = output_loss\n outputs_dict['F1@0.50'] = self.head.get_F1_score(predicted, label)\n return outputs_dict\n def test_step(self, data_batch):\n \"\"\"Testing setp.\n \"\"\"\n feature, _, _ = data_batch\n outputs_dict = dict()\n # call forward\n outputs_cls, outputs_boundary = self.forward_net(feature)\n # transfer data\n outputs_cls_np = outputs_cls[-1].numpy()\n outputs_boundary_np = outputs_boundary[-1].numpy()\n # predict post process\n predicted = ASRFPostProcessing(outputs_cls_np, outputs_boundary_np,\n self.postprocessing_method)\n outputs_dict['predict'] = paddle.to_tensor(predicted[0, :])\n outputs_dict['output_np'] = F.sigmoid(outputs_cls[-1])\n return outputs_dict\n def infer_step(self, data_batch):\n \"\"\"Infering setp.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/asrf.py:101-129"
+ },
+ "7635": {
+ "file_id": 558,
+ "content": "This code is for a model that performs segmentation using ASRF (Adaptive Sparsely Represented Field) method. It consists of functions for forward pass, post processing, and inference steps. The forward_net function takes input features and returns predicted classes and boundaries. The test_step performs testing by calling the forward_net function and applying post-processing to the results. The infer_step performs inference on data_batch using ASRFPostProcessing. It outputs the predicted segmentation, sigmoid-transformed output, and returns them in a dictionary for further processing or evaluation.",
+ "type": "comment"
+ },
+ "7636": {
+ "file_id": 558,
+ "content": " \"\"\"\n feature = data_batch[0]\n # call forward\n outputs_cls, outputs_boundary = self.forward_net(feature)\n # transfer data\n outputs_cls_np = outputs_cls[-1]\n outputs_boundary_np = outputs_boundary[-1]\n outputs = [\n outputs_cls_np, outputs_boundary_np,\n F.sigmoid(outputs_cls[-1])\n ]\n return outputs",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/asrf.py:130-143"
+ },
+ "7637": {
+ "file_id": 558,
+ "content": "This code segment performs the forward pass on a feature, then extracts last outputs for class and boundary, applies sigmoid to the last output of class, and returns all in a list as results.",
+ "type": "comment"
+ },
+ "7638": {
+ "file_id": 559,
+ "content": "/paddlevideo/modeling/framework/segmenters/base.py",
+ "type": "filepath"
+ },
+ "7639": {
+ "file_id": 559,
+ "content": "The BaseSegmenter class serves as a foundation for PaddleVideo segmenters, handling training, validation, testing, and inference with a mode parameter. Subclasses must implement train_step, valid_step, test_step, and feature extraction modules.",
+ "type": "summary"
+ },
+ "7640": {
+ "file_id": 559,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseSegmenter(nn.Layer):\n \"\"\"Base class for segementers.\n All segementers should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Classification head to process feature.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/base.py:1-30"
+ },
+ "7641": {
+ "file_id": 559,
+ "content": "The code is defining a BaseSegmenter class, which serves as the base class for all segmenters. It requires subclasses to override train_step, valid_step, and test_step methods. The class also accepts backbone and head modules to extract features and process them respectively.",
+ "type": "comment"
+ },
+ "7642": {
+ "file_id": 559,
+ "content": " \"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__()\n # build backbone\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n # build head\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n # build loss\n if loss is not None:\n self.loss_name = loss.name\n self.loss = builder.build_loss(loss)\n if hasattr(self.loss, 'init_weights'):\n self.loss.init_weights()\n else:\n self.loss = None\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/base.py:32-63"
+ },
+ "7643": {
+ "file_id": 559,
+ "content": "This code defines a segmenter base class for PaddleVideo. It initializes the backbone, head, and loss layers based on user input. The `forward` method specifies how the model processes data in either infer or train mode. Initializing weights is optional but can be called if the layer supports it.",
+ "type": "comment"
+ },
+ "7644": {
+ "file_id": 559,
+ "content": " 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/base.py:64-99"
+ },
+ "7645": {
+ "file_id": 559,
+ "content": "This code defines a base class for segmenters that supports training, validation, testing, and inference steps. The `mode` parameter determines which step to execute, and abstract methods must be implemented by subclasses for each step.",
+ "type": "comment"
+ },
+ "7646": {
+ "file_id": 559,
+ "content": " raise NotImplementedError",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/base.py:100-100"
+ },
+ "7647": {
+ "file_id": 559,
+ "content": "This code block raises a NotImplementedError, indicating that the current implementation of the function or method is not complete and requires further development.",
+ "type": "comment"
+ },
+ "7648": {
+ "file_id": 560,
+ "content": "/paddlevideo/modeling/framework/segmenters/ms_tcn.py",
+ "type": "filepath"
+ },
+ "7649": {
+ "file_id": 560,
+ "content": "The MSTCN model is a video segmentation tool that extends BaseSegmenter class, includes an optional backbone and head, and defines training/validation steps with loss calculation. The code includes three functions: forward_net for training, test_step for testing, and infer_step for inference.",
+ "type": "summary"
+ },
+ "7650": {
+ "file_id": 560,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import SEGMENTERS\nfrom .base import BaseSegmenter\nimport paddle\nimport paddle.nn.functional as F\n@SEGMENTERS.register()\nclass MSTCN(BaseSegmenter):\n \"\"\"MS-TCN model framework.\"\"\"\n def forward_net(self, video_feature):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n if self.backbone is not None:\n feature = self.backbone(video_feature)\n else:\n feature = video_feature\n if self.head is not None:\n cls_score = self.head(feature)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/ms_tcn.py:1-33"
+ },
+ "7651": {
+ "file_id": 560,
+ "content": "Class MSTCN defines a model for video segmentation, extending BaseSegmenter class. It contains an optional backbone and head for feature extraction and classification. The forward_net function maps input to output through these components if present.",
+ "type": "comment"
+ },
+ "7652": {
+ "file_id": 560,
+ "content": " else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n video_feat, video_gt = data_batch\n # call forward\n output = self.forward_net(video_feat)\n loss = 0.\n for i in range(len(output)):\n loss += self.head.loss(output[i], video_gt)\n predicted = paddle.argmax(output[-1], axis=1)\n predicted = paddle.squeeze(predicted)\n loss_metrics = dict()\n loss_metrics['loss'] = loss\n loss_metrics['F1@0.50'] = self.head.get_F1_score(predicted, video_gt)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n video_feat, video_gt = data_batch\n # call forward\n output = self.forward_net(video_feat)\n loss = 0.\n for i in range(len(output)):\n loss += self.head.loss(output[i], video_gt)\n predicted = paddle.argmax(output[-1], axis=1)\n predicted = paddle.squeeze(predicted)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/ms_tcn.py:34-70"
+ },
+ "7653": {
+ "file_id": 560,
+ "content": "This code defines a training step, validation step, and a method to predict the class score for video segmentation. The training step calculates the loss based on the forward network output and ground truth labels, while the validation step does the same but doesn't return a loss. Both methods return predicted results and loss metrics.",
+ "type": "comment"
+ },
+ "7654": {
+ "file_id": 560,
+ "content": " outputs_dict = dict()\n outputs_dict['loss'] = loss\n outputs_dict['F1@0.50'] = self.head.get_F1_score(predicted, video_gt)\n return outputs_dict\n def test_step(self, data_batch):\n \"\"\"Testing setp.\n \"\"\"\n video_feat, _ = data_batch\n outputs_dict = dict()\n # call forward\n output = self.forward_net(video_feat)\n predicted = paddle.argmax(output[-1], axis=1)\n predicted = paddle.squeeze(predicted)\n outputs_dict['predict'] = predicted\n outputs_dict['output_np'] = F.sigmoid(output[-1])\n return outputs_dict\n def infer_step(self, data_batch):\n \"\"\"Infering setp.\n \"\"\"\n video_feat = data_batch[0]\n # call forward\n output = self.forward_net(video_feat)\n predicted = paddle.argmax(output[-1], axis=1)\n predicted = paddle.squeeze(predicted)\n output_np = F.sigmoid(output[-1])\n return predicted, output_np",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/ms_tcn.py:72-101"
+ },
+ "7655": {
+ "file_id": 560,
+ "content": "This code defines three functions: \"forward_net\" for training, \"test_step\" for testing, and \"infer_step\" for inference. The forward pass of the model is called within each function. In the training step, the loss is calculated and an F1 score is computed using the head module. The predicted labels are also stored. For testing and inference, the predicted labels and output after sigmoid activation are returned separately.",
+ "type": "comment"
+ },
+ "7656": {
+ "file_id": 561,
+ "content": "/paddlevideo/modeling/framework/segmenters/utils.py",
+ "type": "filepath"
+ },
+ "7657": {
+ "file_id": 561,
+ "content": "The GaussianSmoothing class in PaddlePaddle applies 1D gaussian smoothing for image processing tasks, and the code initializes weights and biases for a neural network layer with Kaiming Uniform method.",
+ "type": "summary"
+ },
+ "7658": {
+ "file_id": 561,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# https://github.com/yiskw713/asrf/libs/postprocess.py\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nimport math\nclass GaussianSmoothing(nn.Layer):\n \"\"\"\n Apply gaussian smoothing on a 1d tensor.\n Filtering is performed seperately for each channel\n in the input using a depthwise convolution.\n Arguments:\n channels (int, sequence): Number of channels of the input tensors. Output will\n have this number of channels as well.\n kernel_size (int, sequence): Size of the gaussian kernel.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:1-30"
+ },
+ "7659": {
+ "file_id": 561,
+ "content": "This code defines a GaussianSmoothing class in PaddlePaddle for applying gaussian smoothing on 1D tensors. It uses depthwise convolution to filter each channel separately, with input and output channels remaining the same. The kernel size can be specified as an integer or sequence.",
+ "type": "comment"
+ },
+ "7660": {
+ "file_id": 561,
+ "content": " sigma (float, sequence): Standard deviation of the gaussian kernel.\n \"\"\"\n def __init__(self, kernel_size=15, sigma=1.0):\n super().__init__()\n self.kernel_size = kernel_size\n # The gaussian kernel is the product of the\n # gaussian function of each dimension.\n kernel = 1\n meshgrid = paddle.arange(kernel_size)\n meshgrid = paddle.cast(meshgrid, dtype='float32')\n mean = (kernel_size - 1) / 2\n kernel = kernel / (sigma * math.sqrt(2 * math.pi))\n kernel = kernel * paddle.exp(-(((meshgrid - mean) / sigma)**2) / 2)\n # Make sure sum of values in gaussian kernel equals 1.\n # kernel = kernel / paddle.max(kernel)\n self.kernel = paddle.reshape(kernel, [1, 1, -1])\n def forward(self, inputs):\n \"\"\"\n Apply gaussian filter to input.\n Arguments:\n input (paddle.Tensor): Input to apply gaussian filter on.\n Returns:\n filtered (paddle.Tensor): Filtered output.\n \"\"\"\n _, c, _ = inputs.shape",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:31-62"
+ },
+ "7661": {
+ "file_id": 561,
+ "content": "This code initializes a Gaussian kernel with specified kernel size and sigma. The kernel is then applied to the input during forward pass to filter the data.",
+ "type": "comment"
+ },
+ "7662": {
+ "file_id": 561,
+ "content": " inputs = F.pad(inputs,\n pad=((self.kernel_size - 1) // 2,\n (self.kernel_size - 1) // 2),\n mode=\"reflect\",\n data_format='NCL')\n kernel = paddle.expand(self.kernel, shape=[c, 1, self.kernel_size])\n return F.conv1d(inputs, weight=kernel, groups=c)\ndef argrelmax(prob, threshold=0.7):\n \"\"\"\n Calculate arguments of relative maxima.\n prob: np.array. boundary probability maps distributerd in [0, 1]\n prob shape is (T)\n ignore the peak whose value is under threshold\n Return:\n Index of peaks for each batch\n \"\"\"\n # ignore the values under threshold\n prob[prob < threshold] = 0.0\n # calculate the relative maxima of boundary maps\n # treat the first frame as boundary\n peak = np.concatenate(\n [\n np.ones((1), dtype=np.bool),\n (prob[:-2] < prob[1:-1]) & (prob[2:] < prob[1:-1]),\n np.zeros((1), dtype=np.bool),\n ],\n axis=0,\n )",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:63-95"
+ },
+ "7663": {
+ "file_id": 561,
+ "content": "This code defines a convolution operation and an argrelmax function for image processing. The conv1d function performs 1D convolutions on the input tensor, with padding, kernel expansion, and return as output. The argrelmax function calculates the arguments of relative maxima in boundary probability maps, ignoring values below a certain threshold. This code seems to be related to image segmentation or edge detection tasks.",
+ "type": "comment"
+ },
+ "7664": {
+ "file_id": 561,
+ "content": " peak_idx = np.where(peak)[0].tolist()\n return peak_idx\ndef is_probability(x):\n assert x.ndim == 3\n if x.shape[1] == 1:\n # sigmoid\n if x.min() >= 0 and x.max() <= 1:\n return True\n else:\n return False\n else:\n # softmax\n _sum = np.sum(x, axis=1).astype(np.float32)\n _ones = np.ones_like(_sum, dtype=np.float32)\n return np.allclose(_sum, _ones)\ndef convert2probability(x):\n \"\"\"\n Args: x (N, C, T)\n \"\"\"\n assert x.ndim == 3\n if is_probability(x):\n return x\n else:\n if x.shape[1] == 1:\n # sigmoid\n prob = 1 / (1 + np.exp(-x))\n else:\n # softmax\n prob = np.exp(x) / np.sum(np.exp(x), axis=1)\n return prob.astype(np.float32)\ndef convert2label(x):\n assert x.ndim == 2 or x.ndim == 3\n if x.ndim == 2:\n return x.astype(np.int64)\n else:\n if not is_probability(x):\n x = convert2probability(x)\n label = np.argmax(x, axis=1)\n return label.astype(np.int64)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:97-146"
+ },
+ "7665": {
+ "file_id": 561,
+ "content": "The code provides functions to convert tensors into probabilities or labels. The 'is_probability' function checks if a tensor is in the form of sigmoid or softmax outputs and returns True/False accordingly. The 'convert2probability' function converts tensors into probabilities based on whether they are sigmoid or softmax outputs. Lastly, 'convert2label' function converts tensors (2D or 3D) into labels by either casting to int64 directly for 2D or first converting the tensor to probability and then finding the index of maximum value along the appropriate axis.",
+ "type": "comment"
+ },
+ "7666": {
+ "file_id": 561,
+ "content": "def refinement_with_boundary(outputs, boundaries, boundary_threshold):\n \"\"\"\n Get segments which is defined as the span b/w two boundaries,\n and decide their classes by majority vote.\n Args:\n outputs: numpy array. shape (N, C, T)\n the model output for frame-level class prediction.\n boundaries: numpy array. shape (N, 1, T)\n boundary prediction.\n boundary_threshold: the threshold of the size of action segments. float(default=0.7)\n Return:\n preds: np.array. shape (N, T)\n final class prediction considering boundaries.\n \"\"\"\n preds = convert2label(outputs)\n boundaries = convert2probability(boundaries)\n for i, (output, pred, boundary) in enumerate(zip(outputs, preds,\n boundaries)):\n idx = argrelmax(boundary[0, :], threshold=boundary_threshold)\n # add the index of the last action ending\n T = pred.shape[0]\n idx.append(T)\n # majority vote\n for j in range(len(idx) - 1):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:149-176"
+ },
+ "7667": {
+ "file_id": 561,
+ "content": "This function refines the segmented action outputs based on boundary predictions, and performs majority vote to decide class labels. The inputs include model output (outputs) for frame-level class prediction, boundary prediction (boundaries), and an optional threshold (boundary_threshold). It converts outputs and boundaries into label and probability format respectively. For each sequence, it finds the indices of maximum boundary values above the threshold, appends the last action end index, then performs majority vote on each interval between adjacent max boundaries. The function returns the final class prediction considering boundaries in a numpy array format (preds).",
+ "type": "comment"
+ },
+ "7668": {
+ "file_id": 561,
+ "content": " count = np.bincount(pred[idx[j]:idx[j + 1]])\n modes = np.where(count == count.max())[0]\n if len(modes) == 1:\n mode = modes\n else:\n if outputs.ndim == 3:\n # if more than one majority class exist\n prob_sum_max = 0\n for m in modes:\n prob_sum = output[m, idx[j]:idx[j + 1]].sum()\n if prob_sum_max < prob_sum:\n mode = m\n prob_sum_max = prob_sum\n else:\n # decide first mode when more than one majority class\n # have the same number during oracle experiment\n mode = modes[0]\n preds[i, idx[j]:idx[j + 1]] = mode\n return preds\ndef relabeling(outputs, theta_t):\n \"\"\"\n Relabeling small action segments with their previous action segment\n Args:\n output: the results of action segmentation. (N, T) or (N, C, T)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:177-203"
+ },
+ "7669": {
+ "file_id": 561,
+ "content": "This code segment performs action segmentation by detecting the majority class in each chunk and relabeling smaller action segments with their previous action segment. It uses numpy's bincount and where functions to find majority classes, and has separate logic for cases with multiple majority classes depending on the dimension of outputs. The results are stored in preds array.",
+ "type": "comment"
+ },
+ "7670": {
+ "file_id": 561,
+ "content": " theta_t: the threshold of the size of action segments.\n Return:\n relabeled output. (N, T)\n \"\"\"\n preds = convert2label(outputs)\n for i in range(preds.shape[0]):\n # shape (T,)\n last = preds[i][0]\n cnt = 1\n for j in range(1, preds.shape[1]):\n if last == preds[i][j]:\n cnt += 1\n else:\n if cnt > theta_t:\n cnt = 1\n last = preds[i][j]\n else:\n preds[i][j - cnt:j] = preds[i][j - cnt - 1]\n cnt = 1\n last = preds[i][j]\n if cnt <= theta_t:\n preds[i][j - cnt:j] = preds[i][j - cnt - 1]\n return preds\ndef smoothing(outputs, filter_func):\n \"\"\"\n Smoothing action probabilities with gaussian filter.\n Args:\n outputs: frame-wise action probabilities. (N, C, T)\n Return:\n predictions: final prediction. (N, T)\n \"\"\"\n outputs = convert2probability(outputs)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:204-242"
+ },
+ "7671": {
+ "file_id": 561,
+ "content": "The code defines two functions: \"relabel\" and \"smoothing\". The relabel function takes predicted action segment labels, applies a threshold to merge adjacent segments with overlapping actions, and returns the relabeled output. The smoothing function applies a Gaussian filter to frame-wise action probabilities, resulting in final predictions.",
+ "type": "comment"
+ },
+ "7672": {
+ "file_id": 561,
+ "content": " outputs = filter_func(paddle.to_tensor(outputs)).numpy()\n preds = convert2label(outputs)\n return preds\ndef ASRFPostProcessing(outputs_cls,\n outputs_boundary,\n refinement_method,\n boundary_threshold=0.7,\n theta_t=15,\n kernel_size=15):\n \"\"\"\n ASRF post processing is to refine action boundary\n Args:\n outputs_cls: the results of action segmentation. (N, T) or (N, C, T)\n outputs_boundary: action boundary probability. (N, 1, T)\n refinement_method: the way of refine predict boundary and classification. str\n boundary_threshold: the threshold of the size of action segments. float(default=0.7)\n theta_t: the threshold of the size of action segments. int(default=15)\n kernel_size: Size of the gaussian kernel. int(default=15)\n Return:\n preds output. (N, T)\n \"\"\"\n func = [\n \"refinement_with_boundary\",\n \"relabeling\",\n \"smoothing\",",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:243-270"
+ },
+ "7673": {
+ "file_id": 561,
+ "content": "This code is implementing ASRF post-processing for refining action boundary and classification. It takes in outputs_cls (action segmentation results), outputs_boundary (action boundary probability), refinement_method, boundary_threshold, theta_t (threshold of the size of action segments), and kernel_size as arguments. The code applies three processing steps: \"refinement_with_boundary\", \"relabeling\", and \"smoothing\" to refine the predict boundary and classification. It returns the preds output which is a (N, T) shape.",
+ "type": "comment"
+ },
+ "7674": {
+ "file_id": 561,
+ "content": " ]\n if refinement_method == \"smoothing\":\n filter_func = GaussianSmoothing(kernel_size)\n preds = smoothing(outputs_cls, filter_func)\n elif refinement_method == \"relabeling\":\n preds = relabeling(outputs_cls, theta_t)\n elif refinement_method == \"refinement_with_boundary\":\n preds = refinement_with_boundary(outputs_cls, outputs_boundary,\n boundary_threshold)\n else:\n preds = np.zeros((1, 1))\n assert refinement_method in func\n return paddle.to_tensor(preds)\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = len(tensor.shape)\n if dimensions < 2:\n raise ValueError(\"Fan in and fan out can not be computed \\\n for tensor with fewer than 2 dimensions\")\n if dimensions == 2: # Linear\n fan_in = tensor.shape[1]\n fan_out = tensor.shape[0]\n else:\n num_input_fmaps = tensor.shape[1]\n num_output_fmaps = tensor.shape[0]\n receptive_field_size = 1\n if tensor.dim() > 2:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:271-301"
+ },
+ "7675": {
+ "file_id": 561,
+ "content": "This code segment defines a function that takes an input tensor and calculates the fan-in and fan-out. It also applies different refinement methods to outputs_cls based on the user-specified refinement method. If an invalid method is chosen, it returns a zero tensor. The code includes functions for smoothing, relabeling, and refinement with boundary.",
+ "type": "comment"
+ },
+ "7676": {
+ "file_id": 561,
+ "content": " receptive_field_size = tensor[0][0].numel()\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef calculate_gain(nonlinearity=None, a=None):\n if nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if a is not None:\n return math.sqrt(2.0 / (1 + a**2))\n else:\n return math.sqrt(2.0 / (1 + 0.01**2))\n elif nonlinearity == 'selu':\n return 3.0 / 4\n else:\n return 1\ndef KaimingUniform_like_torch(weight_npy,\n mode='fan_in',\n nonlinearity='leaky_relu'):\n fan_in, fan_out = _calculate_fan_in_and_fan_out(weight_npy)\n if mode == 'fan_in':\n fan_mode = fan_in\n else:\n fan_mode = fan_out\n a = math.sqrt(5.0)\n gain = calculate_gain(nonlinearity=nonlinearity, a=a)\n std = gain / math.sqrt(fan_mode)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:302-335"
+ },
+ "7677": {
+ "file_id": 561,
+ "content": "This code calculates the gain and fan-in/fan-out values for weight initialization in a neural network. It supports different nonlinearities such as 'tanh', 'relu', 'leaky_relu', and 'selu'. The function KaimingUniform_like_torch initializes weights using the Kaiming Uniform method with the specified nonlinearity, fan mode (fan_in or fan_out), and standard deviation of the initialization.",
+ "type": "comment"
+ },
+ "7678": {
+ "file_id": 561,
+ "content": " bound = math.sqrt(3.0) * std\n return np.random.uniform(-bound, bound, weight_npy.shape)\ndef init_bias(weight_npy, bias_npy):\n fan_in, fan_out = _calculate_fan_in_and_fan_out(weight_npy)\n bound = 1.0 / math.sqrt(fan_in)\n return np.random.uniform(-bound, bound, bias_npy.shape)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/framework/segmenters/utils.py:336-343"
+ },
+ "7679": {
+ "file_id": 561,
+ "content": "This code initializes weights and biases for a neural network layer. It calculates the fan-in and fan-out, determines bounds based on standard deviation or square root of three times the standard deviation for weights, and uses a uniform distribution within those bounds to initialize the weights and biases.",
+ "type": "comment"
+ },
+ "7680": {
+ "file_id": 562,
+ "content": "/paddlevideo/modeling/heads/__init__.py",
+ "type": "filepath"
+ },
+ "7681": {
+ "file_id": 562,
+ "content": "This code imports various head classes from different modules in the PaddleVideo library for video object detection, segmentation, or action recognition tasks, and adds them to the `__all__` list for easy access.",
+ "type": "summary"
+ },
+ "7682": {
+ "file_id": 562,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .adds_head import AddsHead\nfrom .asrf_head import ASRFHead\nfrom .attention_lstm_head import AttentionLstmHead, ActionAttentionLstmHead\nfrom .base import BaseHead\nfrom .bbox_head import BBoxHeadAVA\nfrom .cfbi_head import CollaborativeEnsemblerMS\nfrom .i3d_head import I3DHead\nfrom .movinet_head import MoViNetHead\nfrom .ms_tcn_head import MSTCNHead\nfrom .pptimesformer_head import ppTimeSformerHead\nfrom .pptsm_head import ppTSMHead",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/__init__.py:1-25"
+ },
+ "7683": {
+ "file_id": 562,
+ "content": "This code is importing various classes from different modules in the PaddleVideo library. These classes represent different types of heads used in video modeling, such as AttentionLstmHead and BBoxHeadAVA. The code also includes licenses and copyright information for the PaddlePaddle Authors.",
+ "type": "comment"
+ },
+ "7684": {
+ "file_id": 562,
+ "content": "from .pptsn_head import ppTSNHead\nfrom .roi_head import AVARoIHead\nfrom .single_straight3d import SingleRoIExtractor3D\nfrom .slowfast_head import SlowFastHead\nfrom .stgcn_head import STGCNHead\nfrom .timesformer_head import TimeSformerHead\nfrom .transnetv2_head import TransNetV2Head\nfrom .tsm_head import TSMHead\nfrom .tsn_head import TSNHead\nfrom .ms_tcn_head import MSTCNHead\nfrom .asrf_head import ASRFHead\nfrom .ctrgcn_head import CTRGCNHead\nfrom .movinet_head import MoViNetHead\nfrom .agcn2s_head import AGCN2sHead\nfrom .token_shift_head import TokenShiftHead\n__all__ = [\n 'BaseHead', 'TSNHead', 'TSMHead', 'ppTSMHead', 'ppTSNHead', 'SlowFastHead',\n 'AttentionLstmHead', 'TimeSformerHead', 'STGCNHead', 'TransNetV2Head',\n 'I3DHead', 'SingleRoIExtractor3D', 'AVARoIHead', 'BBoxHeadAVA', 'AddsHead',\n 'ppTimeSformerHead', 'CollaborativeEnsemblerMS', 'MSTCNHead', 'ASRFHead',\n 'MoViNetHead', 'CTRGCNHead', 'TokenShiftHead', 'ActionAttentionLstmHead',\n 'AGCN2sHead'\n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/__init__.py:26-49"
+ },
+ "7685": {
+ "file_id": 562,
+ "content": "This code imports various head classes from different modules and adds them to the `__all__` list, making them accessible for import when using this module. These head classes are used in video object detection, segmentation or action recognition tasks. They include ppTSNHead, TSNHead, TSMHead, ppTSMHead, SlowFastHead, TimeSformerHead and more. Each head class has its own specific functionality for different tasks.",
+ "type": "comment"
+ },
+ "7686": {
+ "file_id": 563,
+ "content": "/paddlevideo/modeling/heads/adds_head.py",
+ "type": "filepath"
+ },
+ "7687": {
+ "file_id": 563,
+ "content": "The \"AddsHead\" class in PaddleVideo handles object detection, loss calculation during training and metrics like abs_rel, rmse in inference, while supporting multi-GPU scenarios with all-reduce operations.",
+ "type": "summary"
+ },
+ "7688": {
+ "file_id": 563,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport cv2\nimport numpy as np\nimport paddle.nn as nn\nfrom paddlevideo.utils import get_dist_info\nimport paddle\nfrom ..builder import build_loss\nfrom ..registry import HEADS\nMIN_DEPTH = 1e-3\nMAX_DEPTH = 80\n@HEADS.register()\nclass AddsHead(nn.Layer):\n \"\"\"TimeSformerHead Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/adds_head.py:1-33"
+ },
+ "7689": {
+ "file_id": 563,
+ "content": "This code is part of the PaddleVideo library, defining a class called AddsHead for object detection. It uses input features with specific number of channels and classes to be classified. The class is registered in the registry under HEADS. MIN_DEPTH and MAX_DEPTH constants define the minimum and maximum depth values respectively.",
+ "type": "comment"
+ },
+ "7690": {
+ "file_id": 563,
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n avg_reprojection,\n disparity_smoothness,\n no_ssim,\n loss_cfg=dict(name='ADDSLoss'),\n max_gt_depth=60,\n pred_depth_scale_factor=1):\n super(AddsHead, self).__init__()\n loss_cfg['avg_reprojection'] = avg_reprojection\n loss_cfg['disparity_smoothness'] = disparity_smoothness\n loss_cfg['no_ssim'] = no_ssim\n self.max_gt_depth = max_gt_depth\n self.pred_depth_scale_factor = pred_depth_scale_factor\n self.loss_func = build_loss(loss_cfg)\n def forward(self):\n raise NotImplemented\n def loss(self, inputs, outputs):\n if self.training:\n return self.loss_func(inputs, outputs)\n else:\n abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3 = self.get_metrics(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/adds_head.py:34-62"
+ },
+ "7691": {
+ "file_id": 563,
+ "content": "The code represents the initialization and forward pass of a class named \"AddsHead\". The class takes in parameters like avg_reprojection, disparity_smoothness, no_ssim, etc. It builds a loss function using build_loss method with the provided configuration (loss_cfg). During training, it returns the result of the loss function on inputs and outputs. In inference mode, it uses get_metrics method to calculate metrics such as abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3.",
+ "type": "comment"
+ },
+ "7692": {
+ "file_id": 563,
+ "content": " outputs['pred_disp'], outputs['gt'])\n outputs['abs_rel'] = abs_rel\n outputs['sq_rel'] = sq_rel\n outputs['rmse'] = rmse\n outputs['rmse_log'] = rmse_log\n outputs['a1'] = a1\n outputs['a2'] = a2\n outputs['a3'] = a3\n return outputs\n def get_metrics(self, pred_disp, gt_depth):\n gt_height, gt_width = gt_depth.shape[:2]\n pred_disp = cv2.resize(pred_disp, (gt_width, gt_height))\n pred_depth = 1 / pred_disp\n mask = gt_depth > 0\n pred_depth = pred_depth[mask]\n gt_depth = gt_depth[mask]\n pred_depth *= self.pred_depth_scale_factor\n ratio = np.median(gt_depth) / np.median(pred_depth)\n pred_depth *= ratio\n pred_depth[pred_depth < MIN_DEPTH] = MIN_DEPTH\n pred_depth[pred_depth > MAX_DEPTH] = MAX_DEPTH\n mask2 = gt_depth <= self.max_gt_depth\n pred_depth = pred_depth[mask2]\n gt_depth = gt_depth[mask2]\n abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3 = self.compute_errors(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/adds_head.py:63-95"
+ },
+ "7693": {
+ "file_id": 563,
+ "content": "This code snippet defines an \"AddsHead\" class that returns a dictionary of metrics including absolute relative error, squared relative error, root mean square error, and additional error measures. The get_metrics function resizes the predicted displacement to match the ground truth depth, scales and adjusts the predicted depth based on certain factors, and then computes the specified errors using another function.",
+ "type": "comment"
+ },
+ "7694": {
+ "file_id": 563,
+ "content": " gt_depth, pred_depth)\n _, world_size = get_dist_info()\n if world_size > 1:\n # educe sum when valid\n # TODO: there are some problems with multi gpu gather code.\n abs_rel = paddle.to_tensor(abs_rel)\n sq_rel = paddle.to_tensor(sq_rel)\n rmse = paddle.to_tensor(rmse)\n rmse_log = paddle.to_tensor(rmse_log)\n a1 = paddle.to_tensor(a1)\n a2 = paddle.to_tensor(a2)\n a3 = paddle.to_tensor(a3)\n abs_rel = paddle.distributed.all_reduce(\n abs_rel, op=paddle.distributed.ReduceOp.SUM) / world_size\n sq_rel = paddle.distributed.all_reduce(\n sq_rel, op=paddle.distributed.ReduceOp.SUM) / world_size\n rmse = paddle.distributed.all_reduce(\n rmse, op=paddle.distributed.ReduceOp.SUM) / world_size\n rmse_log = paddle.distributed.all_reduce(\n rmse_log, op=paddle.distributed.ReduceOp.SUM) / world_size\n a1 = paddle.distributed.all_reduce(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/adds_head.py:96-117"
+ },
+ "7695": {
+ "file_id": 563,
+ "content": "This code is performing all-reduce operations on tensors for multi-GPU scenarios, ensuring that the sum of tensor values across GPUs is reduced and then divided by the total number of participating GPUs. This allows for accurate averaging of results when working with multiple GPUs in a distributed environment.",
+ "type": "comment"
+ },
+ "7696": {
+ "file_id": 563,
+ "content": " a1, op=paddle.distributed.ReduceOp.SUM) / world_size\n a2 = paddle.distributed.all_reduce(\n a2, op=paddle.distributed.ReduceOp.SUM) / world_size\n a3 = paddle.distributed.all_reduce(\n a3, op=paddle.distributed.ReduceOp.SUM) / world_size\n return abs_rel.item(), sq_rel.item(), rmse.item(), rmse_log.item(\n ), a1.item(), a2.item(), a3.item()\n return abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3\n def compute_errors(self, gt, pred):\n \"\"\"Computation of error metrics between predicted and ground truth depths\n \"\"\"\n thresh = np.maximum((gt / pred), (pred / gt))\n a1 = (thresh < 1.25).mean()\n a2 = (thresh < 1.25**2).mean()\n a3 = (thresh < 1.25**3).mean()\n rmse = (gt - pred)**2\n rmse = np.sqrt(rmse.mean())\n rmse_log = (np.log(gt) - np.log(pred))**2\n rmse_log = np.sqrt(rmse_log.mean())\n abs_rel = np.mean(np.abs(gt - pred) / gt)\n sq_rel = np.mean(((gt - pred)**2) / gt)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/adds_head.py:118-144"
+ },
+ "7697": {
+ "file_id": 563,
+ "content": "The code defines a function that computes error metrics between predicted and ground truth depths. It uses all-reduce operations to calculate average values and returns multiple metrics including absolute relative error, squared relative error, RMSE, log RMSE, and three averages (a1, a2, a3) indicating the percentage of thresholds below certain levels.",
+ "type": "comment"
+ },
+ "7698": {
+ "file_id": 563,
+ "content": " return abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/adds_head.py:146-146"
+ },
+ "7699": {
+ "file_id": 563,
+ "content": "This code returns six metrics: abs_rel, sq_rel, rmse, rmse_log, a1, and a2. These metrics are likely related to evaluating the performance of some model or algorithm in a regression task. The 'abs_rel' might stand for absolute relative error, 'sq_rel' could be squared relative error, 'rmse' represents root mean squared error, 'rmse_log' could be the logarithm of rmse, and 'a1', 'a2', and 'a3' are possibly other evaluation metrics.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/77.json b/docs/data/77.json
new file mode 100644
index 000000000..70763a33b
--- /dev/null
+++ b/docs/data/77.json
@@ -0,0 +1,546 @@
+{
+ "7700": {
+ "file_id": 564,
+ "content": "/paddlevideo/modeling/heads/agcn2s_head.py",
+ "type": "filepath"
+ },
+ "7701": {
+ "file_id": 564,
+ "content": "The AGCN2sHead class is a head for the AGCN2s model in PaddleVideo, with input arguments defining channels, classes, people, and dropout ratio. It registers under HEADS registry, inherits from BaseHead class, initializes base class, sets instance variables, creates a linear layer, and reshapes input for forward pass. The code takes the input tensor x, averages along axes, passes through a fully connected layer (self.fc) to produce output.",
+ "type": "summary"
+ },
+ "7702": {
+ "file_id": 564,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn as nn\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass AGCN2sHead(BaseHead):\n \"\"\"\n Head for AGCN2s model.\n Args:\n in_channels: int, input feature channels. Default: 64.\n num_classes: int, output the number of classes.\n M: int, number of people.\n drop_out: float, dropout ratio of layer. Default: 0.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/agcn2s_head.py:1-32"
+ },
+ "7703": {
+ "file_id": 564,
+ "content": "The code defines the AGCN2sHead class, a head for the AGCN2s model in PaddleVideo. It has input feature channels, number of classes, number of people, and dropout ratio as arguments. This head is registered under HEADS registry and inherits from BaseHead class.",
+ "type": "comment"
+ },
+ "7704": {
+ "file_id": 564,
+ "content": " \"\"\"\n def __init__(self, in_channels=64, num_classes=10, M=2, **kwargs):\n super().__init__(num_classes, in_channels, **kwargs)\n self.in_channels = in_channels\n self.M = M\n weight_attr = paddle.ParamAttr(\n name=\"linear_weight\",\n initializer=paddle.nn.initializer.Normal(mean=0.0,\n std=math.sqrt(\n 2. / num_classes)))\n self.fc = nn.Linear(self.in_channels * 4,\n self.num_classes,\n weight_attr=weight_attr)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n \"\"\"\n assert x.shape[\n 0] % self.M == 0, f'The first dimension of the output must be an integer multiple of the number of people M, but recieved shape[0]={x.shape[0]}, M={self.M}'\n # N*M,C,T,V\n N = x.shape[0] // self.M\n c_new = x.shape[1]\n x = x.reshape([N, self.M, c_new, -1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/agcn2s_head.py:33-56"
+ },
+ "7705": {
+ "file_id": 564,
+ "content": "Class constructor takes in_channels, num_classes, and M as parameters, initializes base class, sets instance variables, creates a linear layer with specified weights using paddle's Normal initializer, and reshapes input for forward pass.",
+ "type": "comment"
+ },
+ "7706": {
+ "file_id": 564,
+ "content": " x = x.mean(3).mean(1)\n return self.fc(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/agcn2s_head.py:57-59"
+ },
+ "7707": {
+ "file_id": 564,
+ "content": "This code takes the input tensor x, averages it along the third and first axes respectively, then passes it through a fully connected layer (self.fc) to produce an output.",
+ "type": "comment"
+ },
+ "7708": {
+ "file_id": 565,
+ "content": "/paddlevideo/modeling/heads/asrf_head.py",
+ "type": "filepath"
+ },
+ "7709": {
+ "file_id": 565,
+ "content": "The ASRFHead class is a model for action recognition using convolutional layers, and computes precision, recall, F1 score. It creates an ASRF head class for video processing with label retrieval, Levenshtein distance methods, edit scores, true positives, false positives, IoU measures, and selects the best scoring segment.",
+ "type": "summary"
+ },
+ "7710": {
+ "file_id": 565,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# https://github.com/yiskw713/asrf/libs/models/tcn.py\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom paddle import ParamAttr\nfrom ..backbones.ms_tcn import SingleStageModel\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\nfrom ..framework.segmenters.utils import init_bias, KaimingUniform_like_torch\n@HEADS.register()\nclass ASRFHead(BaseHead):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:1-32"
+ },
+ "7711": {
+ "file_id": 565,
+ "content": "The code defines a class for the ASRFHead, which is an instance of BaseHead and registered in HEADS registry. It imports necessary libraries, defines several models including SingleStageModel, and includes various utility functions from other modules. It also initializes weights with KaimingUniform_like_torch method.",
+ "type": "comment"
+ },
+ "7712": {
+ "file_id": 565,
+ "content": " def __init__(self,\n num_classes,\n num_features,\n num_stages,\n num_layers,\n num_stages_asb=None,\n num_stages_brb=None):\n super().__init__(num_classes=num_classes, in_channels=num_features)\n if not isinstance(num_stages_asb, int):\n num_stages_asb = num_stages\n if not isinstance(num_stages_brb, int):\n num_stages_brb = num_stages\n self.num_layers = num_layers\n self.num_stages_asb = num_stages_asb\n self.num_stages_brb = num_stages_brb\n self.num_features = num_features\n # cls score\n self.overlap = 0.5\n self.conv_cls = nn.Conv1D(self.num_features, self.num_classes, 1)\n self.conv_boundary = nn.Conv1D(self.num_features, 1, 1)\n # action segmentation branch\n asb = [\n SingleStageModel(self.num_layers, self.num_features,\n self.num_classes, self.num_classes)\n for _ in range(self.num_stages_asb - 1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:34-63"
+ },
+ "7713": {
+ "file_id": 565,
+ "content": "The code above initializes an object of a class representing a feature extraction and classification model for action recognition. It takes several parameters such as the number of classes, features, stages in the action segmentation branch (ASB), stages in the boundary refinement branch (BRB), and layers per stage. The object is initialized by first calling the superclass constructor and then setting up the necessary components like the convolutional layers for class scores and boundary prediction, as well as multiple SingleStageModel instances for the action segmentation branch if needed.",
+ "type": "comment"
+ },
+ "7714": {
+ "file_id": 565,
+ "content": " ]\n # boundary regression branch\n brb = [\n SingleStageModel(self.num_layers, self.num_features, 1, 1)\n for _ in range(self.num_stages_brb - 1)\n ]\n self.brb = nn.LayerList(brb)\n self.asb = nn.LayerList(asb)\n self.activation_asb = nn.Softmax(axis=1)\n self.activation_brb = nn.Sigmoid()\n def init_weights(self):\n \"\"\"\n initialize model layers' weight\n \"\"\"\n # init weight\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv1D):\n layer.weight.set_value(\n KaimingUniform_like_torch(layer.weight).astype('float32'))\n if layer.bias is not None:\n layer.bias.set_value(\n init_bias(layer.weight, layer.bias).astype('float32'))\n def forward(self, x):\n \"\"\"\n ASRF head\n \"\"\"\n out_cls = self.conv_cls(x)\n out_boundary = self.conv_boundary(x)\n outputs_cls = [out_cls]\n outputs_boundary = [out_boundary]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:64-98"
+ },
+ "7715": {
+ "file_id": 565,
+ "content": "This code defines a ASRF head model, initializes its weights and performs forward pass for classification and boundary regression tasks. It uses Conv1D layers and LayerList for flexibility. The weight initialization follows Kaiming uniform distribution and applies bias if present. The outputs of both tasks are stored separately in lists.",
+ "type": "comment"
+ },
+ "7716": {
+ "file_id": 565,
+ "content": " for as_stage in self.asb:\n out_cls = as_stage(self.activation_asb(out_cls))\n outputs_cls.append(out_cls)\n for br_stage in self.brb:\n out_boundary = br_stage(self.activation_brb(out_boundary))\n outputs_boundary.append(out_boundary)\n return outputs_cls, outputs_boundary\n def get_F1_score(self, predicted, groundTruth):\n recog_content = list(predicted.numpy())\n gt_content = list(groundTruth[0].numpy())\n # cls score\n correct = 0\n total = 0\n edit = 0\n for i in range(len(gt_content)):\n total += 1\n if gt_content[i] == recog_content[i]:\n correct += 1\n edit_num = self.edit_score(recog_content, gt_content)\n edit += edit_num\n tp, fp, fn = self.f_score(recog_content, gt_content, self.overlap)\n # cls metric\n precision = tp / float(tp + fp)\n recall = tp / float(fp + fn)\n if precision + recall > 0.0:\n f1 = 2.0 * (precision * recall) / (precision + recall)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:100-136"
+ },
+ "7717": {
+ "file_id": 565,
+ "content": "This code implements an ASRF head for a model, which takes in input and outputs classified classes and boundary scores. It also includes a get_F1_score function to calculate precision, recall, and F1 score for classification tasks. The F1 score is calculated based on the correctness of predicted class labels compared to ground truth labels.",
+ "type": "comment"
+ },
+ "7718": {
+ "file_id": 565,
+ "content": " else:\n f1 = 0.0\n f1 = np.nan_to_num(f1)\n return f1\n def get_labels_start_end_time(self, frame_wise_labels):\n labels = []\n starts = []\n ends = []\n last_label = frame_wise_labels[0]\n labels.append(frame_wise_labels[0])\n starts.append(0)\n for i in range(len(frame_wise_labels)):\n if frame_wise_labels[i] != last_label:\n labels.append(frame_wise_labels[i])\n starts.append(i)\n ends.append(i)\n last_label = frame_wise_labels[i]\n ends.append(i + 1)\n return labels, starts, ends\n def levenstein(self, p, y, norm=False):\n m_row = len(p)\n n_col = len(y)\n D = np.zeros([m_row + 1, n_col + 1], np.float)\n for i in range(m_row + 1):\n D[i, 0] = i\n for i in range(n_col + 1):\n D[0, i] = i\n for j in range(1, n_col + 1):\n for i in range(1, m_row + 1):\n if y[j - 1] == p[i - 1]:\n D[i, j] = D[i - 1, j - 1]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:137-170"
+ },
+ "7719": {
+ "file_id": 565,
+ "content": "The code defines an ASRF head class that seems to be related to video processing and includes methods for retrieving label information and calculating the Levenshtein distance between two sequences. The get_labels_start_end_time method converts frame-wise labels into a list of labels, their respective start times, and end times. The levenstein method calculates the Levenshtein distance, which is used to compare two sequences of characters.",
+ "type": "comment"
+ },
+ "7720": {
+ "file_id": 565,
+ "content": " else:\n D[i, j] = min(D[i - 1, j] + 1, D[i, j - 1] + 1,\n D[i - 1, j - 1] + 1)\n if norm:\n score = (1 - D[-1, -1] / max(m_row, n_col)) * 100\n else:\n score = D[-1, -1]\n return score\n def edit_score(self, recognized, ground_truth, norm=True):\n P, _, _ = self.get_labels_start_end_time(recognized)\n Y, _, _ = self.get_labels_start_end_time(ground_truth)\n return self.levenstein(P, Y, norm)\n def f_score(self, recognized, ground_truth, overlap):\n p_label, p_start, p_end = self.get_labels_start_end_time(recognized)\n y_label, y_start, y_end = self.get_labels_start_end_time(ground_truth)\n tp = 0\n fp = 0\n hits = np.zeros(len(y_label))\n for j in range(len(p_label)):\n intersection = np.minimum(p_end[j], y_end) - np.maximum(\n p_start[j], y_start)\n union = np.maximum(p_end[j], y_end) - np.minimum(\n p_start[j], y_start)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:171-200"
+ },
+ "7721": {
+ "file_id": 565,
+ "content": "The code contains a function to calculate the edit score between two sequences. It uses the Levenshtein distance algorithm to compare recognized and ground truth labels, considering insertions, deletions, and substitutions. The f_score function calculates true positive (tp) and false positive (fp) values based on label overlaps, and normalizes the edit score if required.",
+ "type": "comment"
+ },
+ "7722": {
+ "file_id": 565,
+ "content": " IoU = (1.0 * intersection / union) * (\n [p_label[j] == y_label[x] for x in range(len(y_label))])\n # Get the best scoring segment\n idx = np.array(IoU).argmax()\n if IoU[idx] >= overlap and not hits[idx]:\n tp += 1\n hits[idx] = 1\n else:\n fp += 1\n fn = len(y_label) - sum(hits)\n return float(tp), float(fp), float(fn)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/asrf_head.py:201-212"
+ },
+ "7723": {
+ "file_id": 565,
+ "content": "This code calculates true positives (tp), false positives (fp) and false negatives (fn). It measures IoU between predicted and actual labels, selects best scoring segment and tracks hits and misses. The method returns tp, fp, fn as float values.",
+ "type": "comment"
+ },
+ "7724": {
+ "file_id": 566,
+ "content": "/paddlevideo/modeling/heads/attention_lstm_head.py",
+ "type": "filepath"
+ },
+ "7725": {
+ "file_id": 566,
+ "content": "The code defines the AttentionLstmHead class for LSTM-based attention mechanism in PaddleVideo, performing feature extraction and softmax normalization for video and audio classification tasks.",
+ "type": "summary"
+ },
+ "7726": {
+ "file_id": 566,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn.initializer import Normal\nfrom paddle.regularizer import L2Decay\nimport paddle.nn.functional as F\nfrom ...metrics.youtube8m import eval_util as youtube8m_metrics\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\nfrom .base import BaseHead\n@HEADS.register()\nclass AttentionLstmHead(BaseHead):\n \"\"\"AttentionLstmHead.\n Args: TODO\n \"\"\"\n def __init__(self,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:1-32"
+ },
+ "7727": {
+ "file_id": 566,
+ "content": "This code defines a class called AttentionLstmHead, which is a type of head used in a neural network. It is part of the PaddleVideo library and inherits from the BaseHead class. The class uses LSTM for attention, has its own parameters (specified by ParamAttr), and utilizes weight initialization. This code also includes license information, documentation on arguments, and registration in the HEADS registry.",
+ "type": "comment"
+ },
+ "7728": {
+ "file_id": 566,
+ "content": " num_classes=3862,\n feature_num=2,\n feature_dims=[1024, 128],\n embedding_size=512,\n lstm_size=1024,\n in_channels=2048,\n loss_cfg=dict(name='CrossEntropyLoss')):\n super(AttentionLstmHead, self).__init__(num_classes, in_channels,\n loss_cfg)\n self.num_classes = num_classes\n self.feature_dims = feature_dims\n self.embedding_size = embedding_size\n self.lstm_size = lstm_size\n self.feature_num = len(self.feature_dims)\n for i in range(self.feature_num): # 0:rgb, 1:audio\n fc_feature = paddle.nn.Linear(in_features=self.feature_dims[i],\n out_features=self.embedding_size)\n self.add_sublayer(\"fc_feature{}\".format(i), fc_feature)\n bi_lstm = paddle.nn.LSTM(input_size=self.embedding_size,\n hidden_size=self.lstm_size,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:33-53"
+ },
+ "7729": {
+ "file_id": 566,
+ "content": "This code initializes an AttentionLstmHead object with specified parameters. It creates a Linear layer for each feature dimension (rgb, audio) and adds a bi-directional LSTM layer with specified sizes. The AttentionLstmHead will be used to process video frames and audio data in parallel for classification tasks.",
+ "type": "comment"
+ },
+ "7730": {
+ "file_id": 566,
+ "content": " direction=\"bidirectional\")\n self.add_sublayer(\"bi_lstm{}\".format(i), bi_lstm)\n drop_rate = 0.5\n self.dropout = paddle.nn.Dropout(drop_rate)\n att_fc = paddle.nn.Linear(in_features=self.lstm_size * 2,\n out_features=1)\n self.add_sublayer(\"att_fc{}\".format(i), att_fc)\n self.softmax = paddle.nn.Softmax()\n self.fc_out1 = paddle.nn.Linear(in_features=self.lstm_size * 4,\n out_features=8192,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.relu = paddle.nn.ReLU()\n self.fc_out2 = paddle.nn.Linear(in_features=8192,\n out_features=4096,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:54-74"
+ },
+ "7731": {
+ "file_id": 566,
+ "content": "The code initializes an LSTM layer with bidirectional capability and adds dropout for regularization. It defines a linear layer (att_fc) to map the output of the LSTM layer to 1 feature, applies softmax activation, and then defines two fully connected layers (fc_out1 and fc_out2) for further processing with specific activations and parameters.",
+ "type": "comment"
+ },
+ "7732": {
+ "file_id": 566,
+ "content": " initializer=Normal()))\n self.fc_logit = paddle.nn.Linear(in_features=4096,\n out_features=self.num_classes,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.sigmoid = paddle.nn.Sigmoid()\n def init_weights(self):\n pass\n def forward(self, inputs):\n # inputs = [(rgb_data, rgb_len, rgb_mask), (audio_data, audio_len, audio_mask)]\n # deal with features with different length\n # 1. padding to same lenght, make a tensor\n # 2. make a mask tensor with the same shpae with 1\n # 3. compute output using mask tensor, s.t. output is nothing todo with padding\n assert (len(inputs) == self.feature_num\n ), \"Input tensor does not contain {} features\".format(\n self.feature_num)\n att_outs = []",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:75-95"
+ },
+ "7733": {
+ "file_id": 566,
+ "content": "The code defines a class for an attention LSTM head in PaddleVideo. It initializes two linear layers and a sigmoid activation function. The `init_weights` method is currently empty, and the `forward` method takes inputs of different lengths and processes them before storing the results in the `att_outs` list.",
+ "type": "comment"
+ },
+ "7734": {
+ "file_id": 566,
+ "content": " for i in range(len(inputs)):\n # 1. fc\n m = getattr(self, \"fc_feature{}\".format(i))\n output_fc = m(inputs[i][0])\n output_fc = paddle.tanh(output_fc)\n # 2. bi_lstm\n m = getattr(self, \"bi_lstm{}\".format(i))\n lstm_out, _ = m(inputs=output_fc, sequence_length=inputs[i][1])\n lstm_dropout = self.dropout(lstm_out)\n # 3. att_fc\n m = getattr(self, \"att_fc{}\".format(i))\n lstm_weight = m(lstm_dropout)\n # 4. softmax replace start, for it's relevant to sum in time step\n lstm_exp = paddle.exp(lstm_weight)\n lstm_mask = paddle.mean(inputs[i][2], axis=2)\n lstm_mask = paddle.unsqueeze(lstm_mask, axis=2)\n lstm_exp_with_mask = paddle.multiply(x=lstm_exp, y=lstm_mask)\n lstm_sum_with_mask = paddle.sum(lstm_exp_with_mask, axis=1)\n exponent = -1\n lstm_denominator = paddle.pow(lstm_sum_with_mask, exponent)\n lstm_denominator = paddle.unsqueeze(lstm_denominator, axis=2)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:96-120"
+ },
+ "7735": {
+ "file_id": 566,
+ "content": "The code performs feature extraction, bi-directional LSTM processing, attention weight calculation, and finally softmax normalization on each input in a list. It uses dropout to prevent overfitting, applies masking for attention calculations, and calculates the denominator using power function.",
+ "type": "comment"
+ },
+ "7736": {
+ "file_id": 566,
+ "content": " lstm_softmax = paddle.multiply(x=lstm_exp, y=lstm_denominator)\n lstm_weight = lstm_softmax\n # softmax replace end\n lstm_scale = paddle.multiply(x=lstm_dropout, y=lstm_weight)\n # 5. sequence_pool's replace start, for it's relevant to sum in time step\n lstm_scale_with_mask = paddle.multiply(x=lstm_scale, y=lstm_mask)\n fea_lens = inputs[i][1]\n fea_len = int(fea_lens[0])\n lstm_pool = paddle.sum(lstm_scale_with_mask, axis=1)\n # sequence_pool's replace end\n att_outs.append(lstm_pool)\n att_out = paddle.concat(att_outs, axis=1)\n fc_out1 = self.fc_out1(att_out)\n fc_out1_act = self.relu(fc_out1)\n fc_out2 = self.fc_out2(fc_out1_act)\n fc_out2_act = paddle.tanh(fc_out2)\n fc_logit = self.fc_logit(fc_out2_act)\n output = self.sigmoid(fc_logit)\n return fc_logit, output\n def loss(self, lstm_logit, labels, **kwargs):\n labels.stop_gradient = True",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:121-144"
+ },
+ "7737": {
+ "file_id": 566,
+ "content": "This code performs LSTM-based attention mechanism for a sequence modeling task. It applies softmax, dropout, and mask operations on the LSTM outputs to compute the attention weights. The attention weights are then used to generate an attentive pooling of the sequence, which is passed through fully connected layers and sigmoid activation for the final output. The loss function uses labels with stop_gradient=True for training the model.",
+ "type": "comment"
+ },
+ "7738": {
+ "file_id": 566,
+ "content": " losses = dict()\n bce_logit_loss = paddle.nn.BCEWithLogitsLoss(reduction='sum')\n sum_cost = bce_logit_loss(lstm_logit, labels)\n return sum_cost\n def metric(self, lstm_output, labels):\n pred = lstm_output.numpy()\n label = labels.numpy()\n hit_at_one = youtube8m_metrics.calculate_hit_at_one(pred, label)\n perr = youtube8m_metrics.calculate_precision_at_equal_recall_rate(\n pred, label)\n gap = youtube8m_metrics.calculate_gap(pred, label)\n return hit_at_one, perr, gap\n@HEADS.register()\nclass ActionAttentionLstmHead(BaseHead):\n \"\"\"AttentionLstmHead for FootballAction\n Args: TODO\n \"\"\"\n def __init__(self,\n num_classes=8,\n feature_num=2,\n feature_dims=[2048, 1024],\n embedding_size=512,\n lstm_size=1024,\n in_channels=2048,\n loss_cfg=dict(name='CrossEntropyLoss')):\n super(ActionAttentionLstmHead, self).__init__(num_classes, in_channels,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:145-173"
+ },
+ "7739": {
+ "file_id": 566,
+ "content": "This code defines an ActionAttentionLstmHead class which is a type of BaseHead. It uses LSTM for attention and takes in various arguments like num_classes, feature_num, feature_dims, embedding_size, lstm_size, in_channels, and loss_cfg. The metric function calculates hit_at_one, perr (precision at equal recall rate), and gap values from the LSTM output and labels. The sum_cost function calculates the loss using BCEWithLogitsLoss.",
+ "type": "comment"
+ },
+ "7740": {
+ "file_id": 566,
+ "content": " loss_cfg)\n self.num_classes = num_classes\n self.feature_dims = feature_dims\n self.embedding_size = embedding_size\n self.lstm_size = lstm_size\n self.feature_num = len(self.feature_dims)\n for i in range(self.feature_num): # 0:rgb, 1:audio\n bi_lstm = paddle.nn.LSTM(input_size=self.feature_dims[i],\n hidden_size=self.feature_dims[i],\n direction=\"bidirectional\")\n self.add_sublayer(\"bi_lstm{}\".format(i), bi_lstm)\n drop_rate = 0.5\n self.dropout = paddle.nn.Dropout(drop_rate)\n att_fc = paddle.nn.Linear(in_features=self.feature_dims[i] * 2,\n out_features=1)\n self.add_sublayer(\"att_fc{}\".format(i), att_fc)\n self.softmax = paddle.nn.Softmax()\n self.fc1 = paddle.nn.Linear(in_features=2 * sum(self.feature_dims),\n out_features=8192,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:174-195"
+ },
+ "7741": {
+ "file_id": 566,
+ "content": "This code initializes a LSTM network for feature processing and attention mechanism. It defines bidirectional LSTM layers for each feature dimension (RGB, audio), followed by dropout and fully connected layers. The model has 8192 output features and is used for multimodal fusion in a video understanding task.",
+ "type": "comment"
+ },
+ "7742": {
+ "file_id": 566,
+ "content": " bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.bn1 = paddle.nn.BatchNorm(num_channels=8192)\n self.dropout1 = paddle.nn.Dropout(0.5)\n self.fc2 = paddle.nn.Linear(in_features=8192,\n out_features=4096,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.bn2 = paddle.nn.BatchNorm(num_channels=4096)\n self.dropout2 = paddle.nn.Dropout(0.5)\n self.fc3 = paddle.nn.Linear(\n in_features=4096,\n out_features=self.num_classes,\n )\n self.fc4 = paddle.nn.Linear(\n in_features=4096,\n out_features=1,\n )\n def init_weights(self):\n pass\n def forward(self, inputs):\n # inputs = [(rgb_data, rgb_len, rgb_mask), (audio_data, audio_len, audio_mask)]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:196-221"
+ },
+ "7743": {
+ "file_id": 566,
+ "content": "This code defines a class for an attention-based LSTM head in PaddleVideo. It includes several fully connected layers, batch normalization, dropout, and two linear layers. The `init_weights` function is not implemented, and the `forward` method takes input data as a tuple of (rgb_data, rgb_len, rgb_mask) and (audio_data, audio_len, audio_mask).",
+ "type": "comment"
+ },
+ "7744": {
+ "file_id": 566,
+ "content": " # deal with features with different length\n # 1. padding to same lenght, make a tensor\n # 2. make a mask tensor with the same shpae with 1\n # 3. compute output using mask tensor, s.t. output is nothing todo with padding\n assert (len(inputs) == self.feature_num\n ), \"Input tensor does not contain {} features\".format(\n self.feature_num)\n att_outs = []\n for i in range(len(inputs)):\n m = getattr(self, \"bi_lstm{}\".format(i))\n lstm_out, _ = m(inputs=inputs[i][0], sequence_length=inputs[i][1])\n lstm_dropout = self.dropout(lstm_out)\n # 3. att_fc\n m = getattr(self, \"att_fc{}\".format(i))\n lstm_weight = m(lstm_dropout)\n # 4. softmax replace start, for it's relevant to sum in time step\n lstm_exp = paddle.exp(lstm_weight)\n lstm_mask = paddle.mean(inputs[i][2], axis=2)\n lstm_mask = paddle.unsqueeze(lstm_mask, axis=2)\n lstm_exp_with_mask = paddle.multiply(x=lstm_exp, y=lstm_mask)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:222-244"
+ },
+ "7745": {
+ "file_id": 566,
+ "content": "This code handles features with varying lengths. It pads features to the same length, creates a mask tensor, and computes the output using the mask tensor, effectively ignoring padding values. It asserts that the input tensor contains the expected number of features. It iterates over each feature, performs bi-directional LSTM, applies dropout, calculates weighted sum using attention mechanism, applies softmax to the weights, multiplies by a mask, and stores the results in att_outs.",
+ "type": "comment"
+ },
+ "7746": {
+ "file_id": 566,
+ "content": " lstm_sum_with_mask = paddle.sum(lstm_exp_with_mask, axis=1)\n exponent = -1\n lstm_denominator = paddle.pow(lstm_sum_with_mask, exponent)\n lstm_denominator = paddle.unsqueeze(lstm_denominator, axis=2)\n lstm_softmax = paddle.multiply(x=lstm_exp, y=lstm_denominator)\n lstm_weight = lstm_softmax\n # softmax replace end\n lstm_scale = paddle.multiply(x=lstm_dropout, y=lstm_weight)\n # 5. sequence_pool's replace start, for it's relevant to sum in time step\n lstm_scale_with_mask = paddle.multiply(x=lstm_scale, y=lstm_mask)\n # fea_lens = inputs[i][1]\n # fea_len = int(fea_lens[0])\n lstm_pool = paddle.sum(lstm_scale_with_mask, axis=1)\n # sequence_pool's replace end\n att_outs.append(lstm_pool)\n att_out = paddle.concat(att_outs, axis=1)\n y = self.fc1(att_out)\n y = self.bn1(y)\n y = F.relu(y)\n y = self.dropout1(y)\n y = self.fc2(y)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:245-267"
+ },
+ "7747": {
+ "file_id": 566,
+ "content": "This code segment calculates the attention scores using LSTM and applies them to sequence pooling. It then passes the output through multiple layers of neural networks, including fully connected layers, batch normalization, ReLU activation, and dropout. The final result is stored in `att_out` for further processing.",
+ "type": "comment"
+ },
+ "7748": {
+ "file_id": 566,
+ "content": " y = self.bn2(y)\n y = F.relu(y)\n y = self.dropout2(y)\n out1 = self.fc3(y)\n out1 = F.softmax(out1)\n out2 = self.fc4(y)\n out2 = F.sigmoid(out2)\n return out1, out2\n def loss(self, logits, iou, labels, labels_iou, **kwargs):\n alpha = 10\n softmax_loss = F.cross_entropy(logits, labels)\n labels_iou = labels_iou.astype('float32')\n mse_loss = paddle.sum(F.square_error_cost(iou, labels_iou), axis=-1)\n sum_loss = softmax_loss + alpha * mse_loss\n return sum_loss\n def metric(self, scores, labels):\n top1 = paddle.metric.accuracy(input=scores, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=scores, label=labels, k=5)\n return top1, top5",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/attention_lstm_head.py:268-288"
+ },
+ "7749": {
+ "file_id": 566,
+ "content": "The code contains two main components: a LSTM attention head and loss/metric functions. The LSTM attention head computes attention weights for the input sequence, followed by softmax and sigmoid activation functions. The loss function calculates cross-entropy and mean squared error losses, with alpha as a weight parameter. The metric function computes top1 and top5 accuracy.",
+ "type": "comment"
+ },
+ "7750": {
+ "file_id": 567,
+ "content": "/paddlevideo/modeling/heads/base.py",
+ "type": "filepath"
+ },
+ "7751": {
+ "file_id": 567,
+ "content": "The code defines a PaddleVideo classification head base class and function for loss/accuracy calculation, supporting binary, multi-class, and specific MRI scenarios with label smoothing. It also calculates top5 accuracy, hard/soft labels, and performs all-reduce operation in distributed training.",
+ "type": "summary"
+ },
+ "7752": {
+ "file_id": 567,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..builder import build_loss\nfrom paddlevideo.utils import get_logger, get_dist_info\nlogger = get_logger(\"paddlevideo\")\nclass BaseHead(nn.Layer):\n \"\"\"Base class for head part.\n All head should subclass it.\n All subclass should overwrite:\n - Methods: ```init_weights```, initializing weights.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:1-34"
+ },
+ "7753": {
+ "file_id": 567,
+ "content": "Base class for head part, all subclass should overwrite init_weights method for initializing weights.",
+ "type": "comment"
+ },
+ "7754": {
+ "file_id": 567,
+ "content": " - Methods: ```forward```, forward function.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channels in input feature.\n loss_cfg (dict): Config for building loss. Default: dict(type='CrossEntropyLoss').\n ls_eps (float): label smoothing epsilon. Default: 0. .\n \"\"\"\n def __init__(\n self,\n num_classes=None,\n in_channels=None,\n loss_cfg=dict(\n name=\"CrossEntropyLoss\"\n ), #TODO(shipping): only pass a name or standard build cfg format.\n #multi_class=False, NOTE(shipping): not supported now.\n ls_eps=0.):\n super().__init__()\n self.num_classes = num_classes\n self.in_channels = in_channels\n self.loss_func = build_loss(loss_cfg)\n #self.multi_class = multi_class NOTE(shipping): not supported now\n self.ls_eps = ls_eps\n @abstractmethod\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n \"\"\"\n raise NotImplemented",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:35-65"
+ },
+ "7755": {
+ "file_id": 567,
+ "content": "This code is defining a base class for a classification head in PaddleVideo. It has an `__init__` method that sets the number of classes, input channels, and loss configuration. It also builds the loss function using the provided configuration. The `forward` method must be implemented by any subclasses, as it defines how the head will run during model inference.",
+ "type": "comment"
+ },
+ "7756": {
+ "file_id": 567,
+ "content": " def loss(self, scores, labels, valid_mode=False, if_top5=True, **kwargs):\n \"\"\"Calculate the loss accroding to the model output ```scores```,\n and the target ```labels```.\n Args:\n scores (paddle.Tensor): The output of the model.\n labels (paddle.Tensor): The target output of the model.\n Returns:\n losses (dict): A dict containing field 'loss'(mandatory) and 'top1_acc', 'top5_acc'(optional).\n \"\"\"\n if len(labels) == 1: #commonly case\n labels = labels[0]\n losses = dict()\n if self.ls_eps != 0. and not valid_mode: # label_smooth\n loss = self.label_smooth_loss(scores, labels, **kwargs)\n else:\n loss = self.loss_func(scores, labels, **kwargs)\n if if_top5:\n top1, top5 = self.get_acc(scores, labels, valid_mode)\n losses['top1'] = top1\n losses['top5'] = top5\n losses['loss'] = loss\n else:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:67-91"
+ },
+ "7757": {
+ "file_id": 567,
+ "content": "This function calculates the loss based on model output (scores) and target output (labels). It returns a dictionary containing 'loss', 'top1_acc', and 'top5_acc'. If labels are single, they are expanded. Label smoothing is applied if ls_eps is non-zero and not in valid mode. The loss function is used if label smoothing is not applicable. Top-1 and top-5 accuracy are also calculated if top-5 is set to True.",
+ "type": "comment"
+ },
+ "7758": {
+ "file_id": 567,
+ "content": " top1 = self.get_acc(scores, labels, valid_mode, if_top5)\n losses['top1'] = top1\n losses['loss'] = loss\n return losses\n # MRI目前二分类无top5\n elif len(labels) == 3: # mix_up\n labels_a, labels_b, lam = labels\n lam = lam[0] # get lam value\n losses = dict()\n if self.ls_eps != 0:\n loss_a = self.label_smooth_loss(scores, labels_a, **kwargs)\n loss_b = self.label_smooth_loss(scores, labels_b, **kwargs)\n else:\n loss_a = self.loss_func(scores, labels_a, **kwargs)\n loss_b = self.loss_func(scores, labels_b, **kwargs)\n loss = lam * loss_a + (1 - lam) * loss_b\n if if_top5:\n top1a, top5a = self.get_acc(scores, labels_a, valid_mode)\n top1b, top5b = self.get_acc(scores, labels_b, valid_mode)\n top1 = lam * top1a + (1 - lam) * top1b\n top5 = lam * top5a + (1 - lam) * top5b",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:92-113"
+ },
+ "7759": {
+ "file_id": 567,
+ "content": "This code handles different cases for classification tasks. For binary and multi-class tasks, it calculates top1 accuracy and loss, while for the specific case of MRI with three labels (mix_up), it applies label smoothing or regular loss function and averages results for each sample to get the final loss and top1 accuracy.",
+ "type": "comment"
+ },
+ "7760": {
+ "file_id": 567,
+ "content": " losses['top1'] = top1\n losses['top5'] = top5\n losses['loss'] = loss\n else:\n top1a = self.get_acc(scores, labels_a, valid_mode, if_top5)\n top1b = self.get_acc(scores, labels_b, valid_mode, if_top5)\n top1 = lam * top1a + (1 - lam) * top1b\n losses['top1'] = top1\n losses['loss'] = loss\n return losses\n else:\n raise NotImplemented\n def label_smooth_loss(self, scores, labels, **kwargs):\n \"\"\"\n Args:\n scores (paddle.Tensor): [N, num_classes]\n labels (paddle.Tensor): [N, ]\n Returns:\n paddle.Tensor: [1,]\n \"\"\"\n if paddle.is_compiled_with_custom_device('npu'):\n \"\"\"\n Designed for the lack of temporary operators of NPU,\n main idea is to split smooth loss into uniform distribution loss\n and hard label calculation\n \"\"\"\n hard_loss = (1.0 - self.ls_eps) * F.cross_entropy(scores, labels)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:114-143"
+ },
+ "7761": {
+ "file_id": 567,
+ "content": "Function defines a loss function for classification tasks, returning a dictionary of losses including top1 and overall loss. If valid_mode is True, calculates accuracy on validation set, otherwise on training set. Top1 accuracies for two sets are combined with specified lambda value to calculate final top1.",
+ "type": "comment"
+ },
+ "7762": {
+ "file_id": 567,
+ "content": " uniform_loss = (self.ls_eps / self.num_classes) * (\n -F.log_softmax(scores, -1).sum(-1).mean(0))\n loss = hard_loss + uniform_loss\n else:\n labels = F.one_hot(labels, self.num_classes)\n labels = F.label_smooth(labels, epsilon=self.ls_eps)\n labels = paddle.squeeze(labels, axis=1)\n loss = self.loss_func(scores, labels, soft_label=True, **kwargs)\n return loss\n def get_acc(self, scores, labels, valid_mode, if_top5=True):\n if if_top5:\n top1 = paddle.metric.accuracy(input=scores, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=scores, label=labels, k=5)\n _, world_size = get_dist_info()\n #NOTE(shipping): deal with multi cards validate\n if world_size > 1 and valid_mode: #reduce sum when valid\n paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM)\n top1 = top1 / world_size\n paddle.distributed.all_reduce(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:144-164"
+ },
+ "7763": {
+ "file_id": 567,
+ "content": "Code is for computing loss and accuracy in a classification model. If the hard label is given, it calculates uniform_loss based on scores and adds it to hard_loss for total loss. Otherwise, it computes soft labels using one-hot encoding and label smoothing, then calculates loss using the provided loss function with soft_label set to True. The get_acc function computes top1 and top5 accuracy, averages them across all cards if valid mode is on, and returns the accuracy.",
+ "type": "comment"
+ },
+ "7764": {
+ "file_id": 567,
+ "content": " top5, op=paddle.distributed.ReduceOp.SUM)\n top5 = top5 / world_size\n return top1, top5\n else:\n top1 = paddle.metric.accuracy(input=scores, label=labels, k=1)\n _, world_size = get_dist_info()\n #NOTE(shipping): deal with multi cards validate\n if world_size > 1 and valid_mode: #reduce sum when valid\n paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM)\n top1 = top1 / world_size\n return top1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/base.py:165-178"
+ },
+ "7765": {
+ "file_id": 567,
+ "content": "This code calculates the top1 and optionally top5 accuracy for a classification task. If distributed training is enabled, it performs all-reduce operation on the calculated metrics to ensure consistency across multiple cards/devices. The reduction operation used is sum, and the results are divided by the total number of devices (world_size) to obtain an average value.",
+ "type": "comment"
+ },
+ "7766": {
+ "file_id": 568,
+ "content": "/paddlevideo/modeling/heads/bbox_head.py",
+ "type": "filepath"
+ },
+ "7767": {
+ "file_id": 568,
+ "content": "The BBoxHeadAVA class generates classification targets, handles dropout, constructs labels, calculates recall/precision, computes losses, and uses a bbox_head for object detection. The code defines \"get_det_bboxes\" and \"multilabel_accuracy\" functions for detecting boxes and computing recall/precision, respectively. Loss is computed using binary cross-entropy with sigmoid activation.",
+ "type": "summary"
+ },
+ "7768": {
+ "file_id": 568,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle \nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom .. import builder\nfrom ..registry import HEADS\n@HEADS.register()\nclass BBoxHeadAVA(nn.Layer):\n \"\"\"Simplest RoI head, with only two fc layers for classification and\n regression respectively. \"\"\"\n def __init__(\n self,\n temporal_pool_type='avg',\n spatial_pool_type='max',\n in_channels=2048,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:1-32"
+ },
+ "7769": {
+ "file_id": 568,
+ "content": "This code defines a BBoxHeadAVA class, which is the simplest RoI (region of interest) head with two fully connected layers for classification and regression. The temporal_pool_type and spatial_pool_type parameters allow users to choose different pooling methods, while in_channels specifies the number of input channels for the network.",
+ "type": "comment"
+ },
+ "7770": {
+ "file_id": 568,
+ "content": " num_classes=81,# The first class is reserved, to classify bbox as pos / neg\n dropout_ratio=0,\n dropout_before_pool=True,\n topk=(3, 5),\n multilabel=True):\n super(BBoxHeadAVA, self).__init__()\n assert temporal_pool_type in ['max', 'avg']\n assert spatial_pool_type in ['max', 'avg']\n self.temporal_pool_type = temporal_pool_type\n self.spatial_pool_type = spatial_pool_type\n self.in_channels = in_channels\n self.num_classes = num_classes\n self.dropout_ratio = dropout_ratio\n self.dropout_before_pool = dropout_before_pool\n self.multilabel = multilabel\n if topk is None:\n self.topk = ()\n elif isinstance(topk, int):\n self.topk = (topk, )\n elif isinstance(topk, tuple):\n assert all([isinstance(k, int) for k in topk])\n self.topk = topk\n else:\n raise TypeError('topk should be int or tuple[int], '\n f'but get {type(topk)}')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:33-61"
+ },
+ "7771": {
+ "file_id": 568,
+ "content": "Class BBoxHeadAVA is being initialized with specified parameters including in_channels, num_classes, dropout_ratio, temporal and spatial pool types, topk values for pooling results, and multilabel flag. The code performs checks on the provided parameters to ensure their validity before assigning them to instance variables.",
+ "type": "comment"
+ },
+ "7772": {
+ "file_id": 568,
+ "content": " # Class 0 is ignored when calculaing multilabel accuracy,\n # so topk cannot be equal to num_classes\n assert all([k < num_classes for k in self.topk])\n assert self.multilabel\n in_channels = self.in_channels\n if self.temporal_pool_type == 'avg':\n self.temporal_pool = nn.AdaptiveAvgPool3D((1, None, None))\n else:\n self.temporal_pool = nn.AdaptiveMaxPool3D((1, None, None))\n if self.spatial_pool_type == 'avg':\n self.spatial_pool = nn.AdaptiveAvgPool3D((None, 1, 1))\n else:\n self.spatial_pool = nn.AdaptiveMaxPool3D((None, 1, 1))\n if dropout_ratio > 0:\n self.dropout = nn.Dropout(dropout_ratio)\n weight_attr = paddle.framework.ParamAttr(name=\"weight\",\n initializer=paddle.nn.initializer.Normal(mean=0.0, std=0.01))\n bias_attr = paddle.ParamAttr(name=\"bias\",\n initializer=paddle.nn.initializer.Constant(value=0.0))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:62-83"
+ },
+ "7773": {
+ "file_id": 568,
+ "content": "This code initializes the BBoxHead model, which is a part of PaddleVideo. It sets up different layers such as temporal and spatial pooling layers, and dropout layer if needed. The code also specifies the parameters for weights and biases in these layers.",
+ "type": "comment"
+ },
+ "7774": {
+ "file_id": 568,
+ "content": " self.fc_cls = nn.Linear(in_channels, num_classes, weight_attr=weight_attr, bias_attr=bias_attr)\n self.debug_imgs = None\n def forward(self, x,rois, rois_num):\n roi = paddle.concat(rois)\n roi_x1 = paddle.index_select(roi, index=paddle.to_tensor(0), axis=1)\n roi_x2 = paddle.index_select(roi, index=paddle.to_tensor(2), axis=1)\n roi_w = roi_x2 - roi_x1\n roi_y1 = paddle.index_select(roi, index=paddle.to_tensor(1), axis=1)\n roi_y2 = paddle.index_select(roi, index=paddle.to_tensor(3), axis=1)\n roi_h = roi_y2 - roi_y1\n roi_area = paddle.multiply(roi_w, roi_h)\n A = roi_area\n A1 = paddle.full(A.shape, 1, dtype='int32')\n A2 = paddle.where(A == 0, paddle.zeros_like(A1), A1)\n AE = paddle.expand(A2, [A.shape[0], x.shape[1]])\n rois_num = paddle.to_tensor(rois_num, dtype='int32')\n if self.dropout_before_pool and self.dropout_ratio > 0 :\n x = self.dropout(x)\n x = self.temporal_pool(x)\n x = self.spatial_pool(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:85-106"
+ },
+ "7775": {
+ "file_id": 568,
+ "content": "This code defines a bbox_head with a linear layer (fc_cls) for classification and initializes debug images. It also performs forward pass by computing ROI features, applying dropout if enabled, and pooling the features.",
+ "type": "comment"
+ },
+ "7776": {
+ "file_id": 568,
+ "content": " if not self.dropout_before_pool and self.dropout_ratio > 0 :\n x = self.dropout(x)\n x = paddle.reshape(x, [x.shape[0], -1])\n x = paddle.multiply(x, paddle.cast(AE,\"float32\"))\n cls_score = self.fc_cls(x)\n # We do not predict bbox, so return None\n return cls_score, None\n def get_targets(self, sampling_results, gt_bboxes, gt_labels, pos_weight):\n pos_proposals = [res.pos_bboxes for res in sampling_results]\n neg_proposals = [res.neg_bboxes for res in sampling_results]\n pos_gt_labels = [res.pos_gt_labels for res in sampling_results]\n cls_reg_targets = self.bbox_target(pos_proposals, neg_proposals,\n pos_gt_labels, pos_weight)\n return cls_reg_targets\n def bbox_target(self, pos_bboxes_list, neg_bboxes_list, gt_labels, pos_weight):\n \"\"\"Generate classification targets for bboxes. \"\"\"\n labels, label_weights = [], []\n pos_weight = 1.0 if pos_weight <= 0 else pos_weight",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:107-126"
+ },
+ "7777": {
+ "file_id": 568,
+ "content": "Code snippet is part of a Bounding Box (BBox) head in PaddleVideo, responsible for generating classification targets and handling dropout before pooling. The code also includes functions to generate bbox targets based on positive and negative proposals, ground truth labels, and a positional weight.",
+ "type": "comment"
+ },
+ "7778": {
+ "file_id": 568,
+ "content": " assert len(pos_bboxes_list) == len(neg_bboxes_list) == len(gt_labels)\n length = len(pos_bboxes_list)\n for i in range(length):\n pos_bboxes = pos_bboxes_list[i]\n neg_bboxes = neg_bboxes_list[i]\n gt_label = gt_labels[i]\n num_pos = pos_bboxes.shape[0]\n if neg_bboxes is not None:\n num_neg = neg_bboxes.shape[0]\n else:\n num_neg = 0\n num_samples = num_pos + num_neg\n neg_label = paddle.zeros([num_neg, gt_label.shape[1]])\n label = paddle.concat([gt_label,neg_label])\n labels.append(label)\n labels = paddle.concat(labels, 0)\n return labels\n def recall_prec(self, pred_vec, target_vec):\n correct = paddle.to_tensor(np.logical_and(pred_vec.numpy(), target_vec.numpy()))\n correct = paddle.where(correct, \n paddle.full(correct.shape,1,dtype='int32'),\n paddle.full(correct.shape,0,dtype='int32'))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:128-152"
+ },
+ "7779": {
+ "file_id": 568,
+ "content": "This code snippet is part of the PaddleVideo library's bbox_head module. It asserts that three lists have equal lengths and then iterates over each list, counting positive (pos) and negative (neg) bounding boxes. It constructs a label by concatenating ground truth labels with zero-filled negatives. The function returns the generated labels for training. The recall_prec function compares prediction vectors to target vectors, creating a correct vector before filling it with 1s or 0s based on their logical AND operation.",
+ "type": "comment"
+ },
+ "7780": {
+ "file_id": 568,
+ "content": " recall_correct = paddle.cast(paddle.sum(correct, axis=1), 'float32')\n target_vec = paddle.where(target_vec, \n paddle.full(target_vec.shape,1,dtype='int32'),\n paddle.full(target_vec.shape,0,dtype='int32'))\n recall_target = paddle.cast(paddle.sum(target_vec, axis=1),'float32')\n recall = recall_correct / recall_target\n pred_vec = paddle.where(pred_vec, \n paddle.full(pred_vec.shape,1,dtype='int32'),\n paddle.full(pred_vec.shape,0,dtype='int32'))\n prec_target = paddle.cast(paddle.sum(pred_vec, axis=1) + 1e-6, 'float32')\n prec = recall_correct / prec_target\n recall_mean = paddle.mean(recall)\n prec_mean = paddle.mean(prec)\n return recall_mean, prec_mean\n def multilabel_accuracy(self, pred, target, thr=0.5):\n pred = paddle.nn.functional.sigmoid(pred)\n pred_vec = pred > thr\n target_vec = target > 0.5",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:153-171"
+ },
+ "7781": {
+ "file_id": 568,
+ "content": "This code calculates recall and precision for multi-label classification tasks. It first computes recall and precision for each sample, then calculates the mean recall and precision across all samples. The function uses threshold values of 0.5 and 1e-6 for target and prediction vectors to ensure numerical stability.",
+ "type": "comment"
+ },
+ "7782": {
+ "file_id": 568,
+ "content": " recall_thr, prec_thr = self.recall_prec(pred_vec, target_vec)\n recalls, precs = [], []\n for k in self.topk:\n _, pred_label = paddle.topk(pred, k, 1, True, True)\n pred_vec = paddle.full(pred.shape,0,dtype='bool')\n num_sample = pred.shape[0]\n for i in range(num_sample):\n pred_vec[i, pred_label[i].numpy()] = 1 \n recall_k, prec_k = self.recall_prec(pred_vec, target_vec)\n recalls.append(recall_k)\n precs.append(prec_k)\n return recall_thr, prec_thr, recalls, precs\n def loss(self,\n cls_score,\n labels):\n losses = dict()\n if cls_score is not None:\n # Only use the cls_score\n labels = labels[:, 1:]\n pos_inds_bool = paddle.sum(labels, axis=-1) > 0\n pos_inds = paddle.where(paddle.sum(labels, axis=-1) > 0,\n paddle.full([labels.shape[0]],1,dtype='int32'),\n paddle.full([labels.shape[0]],0,dtype='int32'))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:172-195"
+ },
+ "7783": {
+ "file_id": 568,
+ "content": "Code creates a bbox_head for object detection. It computes recall and precision given predicted and target vectors, and returns the results. In loss function, it only considers cls_score if available and computes losses based on pos_inds (positive indices) and labels.",
+ "type": "comment"
+ },
+ "7784": {
+ "file_id": 568,
+ "content": " pos_inds = paddle.nonzero(pos_inds, as_tuple=False)\n cls_score = paddle.index_select(cls_score, pos_inds, axis=0)\n cls_score = cls_score[:, 1:] \n labels = paddle.index_select(labels, pos_inds, axis=0)\n bce_loss = F.binary_cross_entropy_with_logits\n loss = bce_loss(cls_score, labels, reduction='none')\n losses['loss'] = paddle.mean(loss)\n recall_thr, prec_thr, recall_k, prec_k = self.multilabel_accuracy(\n cls_score, labels, thr=0.5)\n losses['recall@thr=0.5'] = recall_thr\n losses['prec@thr=0.5'] = prec_thr\n for i, k in enumerate(self.topk):\n losses[f'recall@top{k}'] = recall_k[i]\n losses[f'prec@top{k}'] = prec_k[i]\n return losses\n def get_det_bboxes(self,\n rois,\n cls_score,\n img_shape,\n flip=False,\n crop_quadruple=None,\n cfg=None):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:196-218"
+ },
+ "7785": {
+ "file_id": 568,
+ "content": "This code defines two functions: \"get_det_bboxes\" and \"multilabel_accuracy\". The \"get_det_bboxes\" function takes ROIs, cls_score, img_shape, flip, and crop_quadruple as inputs to calculate detection boxes for each bounding box. The \"multilabel_accuracy\" function calculates recall and precision for different thresholds and top-k values from the given cls_score and labels arrays. The code also computes loss using binary cross-entropy with logits and adds it to the losses dictionary.",
+ "type": "comment"
+ },
+ "7786": {
+ "file_id": 568,
+ "content": " if isinstance(cls_score, list):\n cls_score = sum(cls_score) / float(len(cls_score))\n assert self.multilabel\n m = paddle.nn.Sigmoid()\n scores = m(cls_score)\n bboxes = rois\n return bboxes, scores",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/bbox_head.py:219-225"
+ },
+ "7787": {
+ "file_id": 568,
+ "content": "The code checks if cls_score is a list, calculates the mean of its elements if it's a list, asserts that self.multilabel is True, applies sigmoid activation to cls_score, and assigns resulting scores to variable 'scores'. It also assigns rois to bboxes and returns both bboxes and scores.",
+ "type": "comment"
+ },
+ "7788": {
+ "file_id": 569,
+ "content": "/paddlevideo/modeling/heads/cfbi_head.py",
+ "type": "filepath"
+ },
+ "7789": {
+ "file_id": 569,
+ "content": "The code introduces new layers, initializes Bottleneck with GCT, defines Convolutional Feature Fusion Block and Atrous Spatial Pyramid Pooling modules. The CollaborativeEnsemblerMS class is a neural network architecture with multiple input dimensions, transformer stages, convolutional layers, ReLU activation, and outputs foreground/background logits using ASPP modules.",
+ "type": "summary"
+ },
+ "7790": {
+ "file_id": 569,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\nclass IA_gate(nn.Layer):\n def __init__(self, in_dim, out_dim):\n super(IA_gate, self).__init__()\n self.IA = nn.Linear(in_dim, out_dim)\n def forward(self, x, IA_head):\n a = self.IA(IA_head)\n a = 1. + paddle.tanh(a)\n a = paddle.unsqueeze(paddle.unsqueeze(a, axis=-1), axis=-1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:1-32"
+ },
+ "7791": {
+ "file_id": 569,
+ "content": "This code defines a class for the IA_gate layer, which is a part of a computer vision model. It has an input and output dimension and includes a linear layer and a forward function. The forward function calculates the activation (a) by applying a tanh function to the linear layer's output and then unsqueezing it along the axis for multiplication. The result is used in the model's computation.",
+ "type": "comment"
+ },
+ "7792": {
+ "file_id": 569,
+ "content": " x = a * x\n return x\nclass GCT(nn.Layer):\n def __init__(self, num_channels, epsilon=1e-5, mode='l2', after_relu=False):\n super(GCT, self).__init__()\n x1 = paddle.zeros([1, num_channels, 1, 1])\n x2 = paddle.ones([1, num_channels, 1, 1])\n self.alpha = paddle.create_parameter(\n shape=x2.shape,\n dtype=x2.dtype,\n default_initializer=nn.initializer.Assign(x2))\n self.alpha.stop_gradient = False\n self.gamma = paddle.create_parameter(\n shape=x1.shape,\n dtype=x1.dtype,\n default_initializer=nn.initializer.Assign(x1))\n self.gamma.stop_gradient = False\n self.beta = paddle.create_parameter(\n shape=x1.shape,\n dtype=x1.dtype,\n default_initializer=nn.initializer.Assign(x1))\n self.beta.stop_gradient = False\n self.epsilon = epsilon\n self.mode = mode\n self.after_relu = after_relu\n def forward(self, x):\n if self.mode == 'l2':\n embedding = paddle.pow(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:33-65"
+ },
+ "7793": {
+ "file_id": 569,
+ "content": "This code defines a GCT layer, which is a type of normalization layer for neural networks. It initializes parameters alpha, gamma and beta with specific shapes and default values. The layer also takes in an input x, applies the mode 'l2' operation (pow) on it, and returns the result.",
+ "type": "comment"
+ },
+ "7794": {
+ "file_id": 569,
+ "content": " paddle.sum(paddle.pow(x, 2), axis=[2, 3], keepdim=True) +\n self.epsilon, 0.5) * self.alpha\n norm = self.gamma / paddle.pow(\n (paddle.mean(paddle.pow(embedding, 2), axis=1, keepdim=True) +\n self.epsilon), 0.5)\n elif self.mode == 'l1':\n if not self.after_relu:\n _x = paddle.abs(x)\n else:\n _x = x\n embedding = paddle.sum(_x, axis=(2, 3), keepdim=True) * self.alpha\n norm = self.gamma / (paddle.mean(\n paddle.abs(embedding), axis=1, keepdim=True) + self.epsilon)\n else:\n print('Unknown mode!')\n exit()\n gate = 1. + paddle.tanh(embedding * norm + self.beta)\n return x * gate\nclass Bottleneck(nn.Layer):\n def __init__(self, inplanes, outplanes, stride=1, dilation=1):\n super(Bottleneck, self).__init__()\n expansion = 4\n planes = int(outplanes / expansion)\n self.GCT1 = GCT(inplanes)\n self.conv1 = nn.Conv2D(inplanes, planes, kernel_size=1, bias_attr=False)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:66-95"
+ },
+ "7795": {
+ "file_id": 569,
+ "content": "The code initializes a Bottleneck layer in the PaddleVideo model, with GCT and convolutional layers for feature extraction. It also includes adjustable normalization and activation based on the mode parameter.",
+ "type": "comment"
+ },
+ "7796": {
+ "file_id": 569,
+ "content": " self.bn1 = nn.GroupNorm(num_groups=32, num_channels=planes)\n self.conv2 = nn.Conv2D(planes,\n planes,\n kernel_size=3,\n stride=stride,\n dilation=dilation,\n padding=dilation,\n bias_attr=False)\n self.bn2 = nn.GroupNorm(num_groups=32, num_channels=planes)\n self.conv3 = nn.Conv2D(planes,\n planes * expansion,\n kernel_size=1,\n bias_attr=False)\n self.bn3 = nn.GroupNorm(num_groups=32, num_channels=planes * expansion)\n self.relu = nn.ReLU()\n if stride != 1 or inplanes != planes * expansion:\n downsample = nn.Sequential(\n nn.Conv2D(inplanes,\n planes * expansion,\n kernel_size=1,\n stride=stride,\n bias_attr=False),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:96-119"
+ },
+ "7797": {
+ "file_id": 569,
+ "content": "This code defines a neural network layer that includes batch normalization and convolutional layers, as well as ReLU activation. It has the option for downsampling if necessary.",
+ "type": "comment"
+ },
+ "7798": {
+ "file_id": 569,
+ "content": " nn.GroupNorm(num_groups=32, num_channels=planes * expansion),\n )\n else:\n downsample = None\n self.downsample = downsample\n self.stride = stride\n self.dilation = dilation\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n def forward(self, x):\n residual = x\n out = self.GCT1(x)\n out = self.conv1(out)\n out = self.bn1(out)\n out = self.relu(out)\n out = self.conv2(out)\n out = self.bn2(out)\n out = self.relu(out)\n out = self.conv3(out)\n out = self.bn3(out)\n if self.downsample is not None:\n residual = self.downsample(x)\n out += residual\n out = self.relu(out)\n return out\nclass _ASPPModule(nn.Layer):\n def __init__(self, inplanes, planes, kernel_size, padding, dilation):\n super(_ASPPModule, self).__init__()\n self.GCT = GCT(inplanes)\n self.atrous_conv = nn.Conv2D(inplanes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:120-160"
+ },
+ "7799": {
+ "file_id": 569,
+ "content": "Code initializes a module with 3 Conv2D layers and BatchNorm2D layers. It also includes a GroupNorm layer if num_groups and num_channels are specified, otherwise sets downsample to None. Initializes sublayers and applies Kaiming Normal initialization. Forward function performs convolutions, adds residual connection if applicable, and applies ReLU activation. _ASPPModule has GCT and AtrousConv2D layers.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/78.json b/docs/data/78.json
new file mode 100644
index 000000000..9909f05b7
--- /dev/null
+++ b/docs/data/78.json
@@ -0,0 +1,543 @@
+{
+ "7800": {
+ "file_id": 569,
+ "content": " planes,\n kernel_size=kernel_size,\n stride=1,\n padding=padding,\n dilation=dilation,\n bias_attr=False)\n self.bn = nn.GroupNorm(num_groups=int(planes / 4), num_channels=planes)\n self.relu = nn.ReLU()\n self._init_weight()\n def forward(self, x):\n x = self.GCT(x)\n x = self.atrous_conv(x)\n x = self.bn(x)\n return self.relu(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass ASPP(nn.Layer):\n def __init__(self):\n super(ASPP, self).__init__()\n inplanes = 512\n dilations = [1, 6, 12, 18]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:161-193"
+ },
+ "7801": {
+ "file_id": 569,
+ "content": "The code defines a Convolutional Feature Fusion Block (CFFB) and an Atrous Spatial Pyramid Pooling (ASPP) module. The CFFB consists of group convolution, batch normalization, and ReLU activation layers. The ASPP module has four pathways with different dilation rates, each followed by a group convolution, batch normalization, and ReLU activation.",
+ "type": "comment"
+ },
+ "7802": {
+ "file_id": 569,
+ "content": " self.aspp1 = _ASPPModule(inplanes,\n 128,\n 1,\n padding=0,\n dilation=dilations[0])\n self.aspp2 = _ASPPModule(inplanes,\n 128,\n 3,\n padding=dilations[1],\n dilation=dilations[1])\n self.aspp3 = _ASPPModule(inplanes,\n 128,\n 3,\n padding=dilations[2],\n dilation=dilations[2])\n self.aspp4 = _ASPPModule(inplanes,\n 128,\n 3,\n padding=dilations[3],\n dilation=dilations[3])\n self.global_avg_pool = nn.Sequential(\n nn.AdaptiveAvgPool2D((1, 1)),\n nn.Conv2D(inplanes, 128, 1, stride=1, bias_attr=False), nn.ReLU())",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:195-218"
+ },
+ "7803": {
+ "file_id": 569,
+ "content": "This code initializes four ASPPModules and a global average pooling layer in the CFBI head model for feature extraction and pooling. The ASPPModules have different dilation rates based on the specified dilations list, while the global_avg_pool performs adaptive averaging and convolution to extract global features.",
+ "type": "comment"
+ },
+ "7804": {
+ "file_id": 569,
+ "content": " self.GCT = GCT(640)\n self.conv1 = nn.Conv2D(640, 256, 1, bias_attr=False)\n self.bn1 = nn.GroupNorm(num_groups=32, num_channels=256)\n self.relu = nn.ReLU()\n self._init_weight()\n def forward(self, x):\n x1 = self.aspp1(x)\n x2 = self.aspp2(x)\n x3 = self.aspp3(x)\n x4 = self.aspp4(x)\n x5 = self.global_avg_pool(x)\n x5 = F.interpolate(x5,\n size=x4.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat([x1, x2, x3, x4, x5], axis=1)\n x = self.GCT(x)\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n return x\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:220-251"
+ },
+ "7805": {
+ "file_id": 569,
+ "content": "The code initializes a class with multiple layers for feature extraction and processing, using Conv2D, GroupNorm, ReLU activation functions, and Global Average Pooling. The forward function combines features from different ASPP modules and passes them through GCT, convolution, batch normalization, and ReLU for final output. Initializes the weight of each layer with specific initializers.",
+ "type": "comment"
+ },
+ "7806": {
+ "file_id": 569,
+ "content": "@HEADS.register()\nclass CollaborativeEnsemblerMS(nn.Layer):\n def __init__(\n self,\n model_semantic_embedding_dim=256,\n model_multi_local_distance=[[4, 8, 12, 16, 20, 24],\n [2, 4, 6, 8, 10, 12], [2, 4, 6, 8, 10]],\n model_head_embedding_dim=256,\n model_refine_channels=64,\n model_low_level_inplanes=256,\n ):\n super(CollaborativeEnsemblerMS, self).__init__()\n in_dim_4x = model_semantic_embedding_dim * 3 + 3 + 2 * len(\n model_multi_local_distance[0])\n in_dim_8x = model_semantic_embedding_dim * 3 + 3 + 2 * len(\n model_multi_local_distance[1])\n in_dim_16x = model_semantic_embedding_dim * 3 + 3 + 2 * len(\n model_multi_local_distance[2])\n attention_dim = model_semantic_embedding_dim * 4\n embed_dim = model_head_embedding_dim\n refine_dim = model_refine_channels\n low_level_dim = model_low_level_inplanes\n IA_in_dim = attention_dim\n self.relu = nn.ReLU()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:254-279"
+ },
+ "7807": {
+ "file_id": 569,
+ "content": "The code defines a CollaborativeEnsemblerMS class within the PaddleVideo framework. It has multiple input dimensions (4x, 8x, and 16x) for semantic embedding, local distance, and attention dimension. The class also includes an instance of ReLU activation function.",
+ "type": "comment"
+ },
+ "7808": {
+ "file_id": 569,
+ "content": " # stage 1\n self.S1_IA1 = IA_gate(IA_in_dim, in_dim_4x)\n self.S1_layer1 = Bottleneck(in_dim_4x, embed_dim)\n self.S1_IA2 = IA_gate(IA_in_dim, embed_dim)\n self.S1_layer2 = Bottleneck(embed_dim, embed_dim, 1, 2)\n # stage2\n self.S2_IA1 = IA_gate(IA_in_dim, embed_dim)\n self.S2_layer1 = Bottleneck(embed_dim, embed_dim * 2, 2)\n self.S2_IA2 = IA_gate(IA_in_dim, embed_dim * 2 + in_dim_8x)\n self.S2_layer2 = Bottleneck(embed_dim * 2 + in_dim_8x, embed_dim * 2, 1,\n 2)\n self.S2_IA3 = IA_gate(IA_in_dim, embed_dim * 2)\n self.S2_layer3 = Bottleneck(embed_dim * 2, embed_dim * 2, 1, 4)\n # stage3\n self.S3_IA1 = IA_gate(IA_in_dim, embed_dim * 2)\n self.S3_layer1 = Bottleneck(embed_dim * 2, embed_dim * 2, 2)\n self.S3_IA2 = IA_gate(IA_in_dim, embed_dim * 2 + in_dim_16x)\n self.S3_layer2 = Bottleneck(embed_dim * 2 + in_dim_16x, embed_dim * 2,\n 1, 2)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:281-306"
+ },
+ "7809": {
+ "file_id": 569,
+ "content": "This code initializes multiple layers for different stages of a transformer model. Each stage consists of several IA_gate and Bottleneck layers, with varying input and output dimensions. The stages progressively increase the embedding dimension, incorporating additional inputs along the way.",
+ "type": "comment"
+ },
+ "7810": {
+ "file_id": 569,
+ "content": " self.S3_IA3 = IA_gate(IA_in_dim, embed_dim * 2)\n self.S3_layer3 = Bottleneck(embed_dim * 2, embed_dim * 2, 1, 4)\n self.ASPP_IA = IA_gate(IA_in_dim, embed_dim * 2)\n self.ASPP = ASPP()\n # Decoder\n self.GCT_sc = GCT(low_level_dim + embed_dim)\n self.conv_sc = nn.Conv2D(low_level_dim + embed_dim,\n refine_dim,\n 1,\n bias_attr=False)\n self.bn_sc = nn.GroupNorm(num_groups=int(refine_dim / 4),\n num_channels=refine_dim)\n self.relu = nn.ReLU()\n self.IA10 = IA_gate(IA_in_dim, embed_dim + refine_dim)\n self.conv1 = nn.Conv2D(embed_dim + refine_dim,\n int(embed_dim / 2),\n kernel_size=3,\n padding=1,\n bias_attr=False)\n self.bn1 = nn.GroupNorm(num_groups=32, num_channels=int(embed_dim / 2))\n self.IA11 = IA_gate(IA_in_dim, int(embed_dim / 2))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:308-332"
+ },
+ "7811": {
+ "file_id": 569,
+ "content": "This code is defining various components of a model for feature extraction and fusion. It includes IA_gate, Bottleneck, GCT, ASPP, nn.Conv2D, GroupNorm, ReLU layers and their configurations. The model has separate modules for encoding and decoding stages to process low-level and high-level features respectively.",
+ "type": "comment"
+ },
+ "7812": {
+ "file_id": 569,
+ "content": " self.conv2 = nn.Conv2D(int(embed_dim / 2),\n int(embed_dim / 2),\n kernel_size=3,\n padding=1,\n bias_attr=False)\n self.bn2 = nn.GroupNorm(num_groups=32, num_channels=int(embed_dim / 2))\n # Output\n self.IA_final_fg = nn.Linear(IA_in_dim, int(embed_dim / 2) + 1)\n self.IA_final_bg = nn.Linear(IA_in_dim, int(embed_dim / 2) + 1)\n self.conv_sc.weight.data = nn.initializer.KaimingNormal()\n self.conv1.weight.data = nn.initializer.KaimingNormal()\n self.conv2.weight.data = nn.initializer.KaimingNormal()\n def forward(self, all_x, all_IA_head=None, low_level_feat=None):\n x_4x, x_8x, x_16x = all_x\n IA_head = all_IA_head[0]\n # stage 1\n x = self.S1_IA1(x_4x, IA_head)\n x = self.S1_layer1(x)\n x = self.S1_IA2(x, IA_head)\n x = self.S1_layer2(x)\n low_level_feat = paddle.concat(\n [paddle.expand(low_level_feat, [x.shape[0], -1, -1, -1]), x],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:333-360"
+ },
+ "7813": {
+ "file_id": 569,
+ "content": "This code defines a neural network architecture for a computer vision task. It includes convolutional layers, batch normalization, and linear layers. The forward function applies these operations to input features at different scales (4x, 8x, 16x) and concatenates the results. The KaimingNormal initialization is used to set the weights of the convolution layers.",
+ "type": "comment"
+ },
+ "7814": {
+ "file_id": 569,
+ "content": " axis=1)\n # stage 2\n x = self.S2_IA1(x, IA_head)\n x = self.S2_layer1(x)\n x = paddle.concat([x, x_8x], axis=1)\n x = self.S2_IA2(x, IA_head)\n x = self.S2_layer2(x)\n x = self.S2_IA3(x, IA_head)\n x = self.S2_layer3(x)\n # stage 3\n x = self.S3_IA1(x, IA_head)\n x = self.S3_layer1(x)\n x = paddle.concat([x, x_16x], axis=1)\n x = self.S3_IA2(x, IA_head)\n x = self.S3_layer2(x)\n x = self.S3_IA3(x, IA_head)\n x = self.S3_layer3(x)\n # ASPP + Decoder\n x = self.ASPP_IA(x, IA_head)\n x = self.ASPP(x)\n x = self.decoder(x, low_level_feat, IA_head)\n fg_logit = self.IA_logit(x, IA_head, self.IA_final_fg)\n bg_logit = self.IA_logit(x, IA_head, self.IA_final_bg)\n pred = self.augment_background_logit(fg_logit, bg_logit)\n return pred\n def IA_logit(self, x, IA_head, IA_final):\n n, c, h, w = x.shape\n x = paddle.reshape(x, [1, n * c, h, w])\n IA_output = IA_final(IA_head)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:361-401"
+ },
+ "7815": {
+ "file_id": 569,
+ "content": "This code defines a neural network architecture for instance segmentation. It consists of multiple stages and an ASPP (Atrous Spatial Pyramid Pooling) module. The IA_logit function is used to output foreground and background logits. The final output, 'pred', is the instance segmentation prediction after applying background augmentation.",
+ "type": "comment"
+ },
+ "7816": {
+ "file_id": 569,
+ "content": " IA_weight = IA_output[:, :c]\n IA_bias = IA_output[:, -1]\n IA_weight = paddle.reshape(IA_weight, [n, c, 1, 1])\n IA_bias = paddle.reshape(IA_bias, [-1])\n logit = paddle.reshape(\n F.conv2d(x, weight=IA_weight, bias=IA_bias, groups=n), [n, 1, h, w])\n return logit\n def decoder(self, x, low_level_feat, IA_head):\n x = F.interpolate(x,\n size=low_level_feat.shape[2:],\n mode='bicubic',\n align_corners=True)\n low_level_feat = self.GCT_sc(low_level_feat)\n low_level_feat = self.conv_sc(low_level_feat)\n low_level_feat = self.bn_sc(low_level_feat)\n low_level_feat = self.relu(low_level_feat)\n x = paddle.concat([x, low_level_feat], axis=1)\n x = self.IA10(x, IA_head)\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n x = self.IA11(x, IA_head)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu(x)\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:402-433"
+ },
+ "7817": {
+ "file_id": 569,
+ "content": "The code defines two functions: `IA_head` and `decoder`. The `IA_head` function takes an input, applies a convolution with a weight and a bias, and reshapes the output. The `decoder` function combines an input image and a low-level feature, passes it through several convolutional layers with batch normalization and ReLU activation, then applies two IA heads.",
+ "type": "comment"
+ },
+ "7818": {
+ "file_id": 569,
+ "content": " def augment_background_logit(self, fg_logit, bg_logit):\n # We augment the logit of absolute background by using the relative background logit of all the\n # foreground objects.\n obj_num = fg_logit.shape[0]\n pred = fg_logit\n if obj_num > 1:\n bg_logit = bg_logit[1:obj_num, :, :, :]\n aug_bg_logit = paddle.min(bg_logit, axis=0, keepdim=True)\n pad = paddle.expand(paddle.zeros(aug_bg_logit.shape),\n [obj_num - 1, -1, -1, -1])\n aug_bg_logit = paddle.concat([aug_bg_logit, pad], axis=0)\n pred = pred + aug_bg_logit\n pred = paddle.transpose(pred, [1, 0, 2, 3])\n return pred",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/cfbi_head.py:435-448"
+ },
+ "7819": {
+ "file_id": 569,
+ "content": "This function takes two logits, fg_logit and bg_logit, and augments the absolute background logit by using relative background logits from all foreground objects. If there are more than one foreground object, it calculates the minimum of their relative background logits, pads with zeros to match the number of original background logits, concatenates, and adds this augmented background logit to the original fg_logit. The output is then transposed before being returned.",
+ "type": "comment"
+ },
+ "7820": {
+ "file_id": 570,
+ "content": "/paddlevideo/modeling/heads/ctrgcn_head.py",
+ "type": "filepath"
+ },
+ "7821": {
+ "file_id": 570,
+ "content": "The CTRGCNHead class is a neural network head for the CTR-GCN model in PaddleVideo library, containing layers initialization, weight initialization, and forward pass function definition. The ctrgcn_head class returns the result of passing input x through a fully connected layer (fc) for feature processing and prediction.",
+ "type": "summary"
+ },
+ "7822": {
+ "file_id": 570,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn as nn\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass CTRGCNHead(BaseHead):\n \"\"\"\n Head for CTR-GCN model.\n Args:\n in_channels: int, input feature channels. Default: 64.\n num_classes: int, output the number of classes.\n drop_out: float, dropout ratio of layer. Default: 0.\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ctrgcn_head.py:1-32"
+ },
+ "7823": {
+ "file_id": 570,
+ "content": "This code snippet is a part of the PaddleVideo library, specifically the CTRGCNHead class. It is a neural network head for the CTR-GCN model that takes in input feature channels and outputs the number of classes, with an optional dropout ratio. The code imports necessary libraries, registers the class under the HEADS registry, and defines the class itself as part of the BaseHead class.",
+ "type": "comment"
+ },
+ "7824": {
+ "file_id": 570,
+ "content": " def __init__(self, in_channels=64, num_classes=10, drop_out=0, **kwargs):\n super().__init__(num_classes, in_channels, **kwargs)\n self.in_channels = in_channels\n self.drop_out = drop_out\n self.fc = nn.Linear(self.in_channels * 4, self.num_classes)\n if drop_out:\n self.drop_out = nn.Dropout(self.drop_out)\n else:\n self.drop_out = lambda x: x\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer.weight,\n 'Normal',\n mean=0.0,\n std=math.sqrt(2. / self.num_classes))\n def forward(self, output_patch):\n \"\"\"Define how the head is going to run.\n \"\"\"\n x, N, M = output_patch\n # N*M,C,T,V\n _, c_new, T, V = x.shape\n x = paddle.reshape(x, shape=[N, M, c_new, T * V])\n x = x.mean(3).mean(1)\n x = self.drop_out(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ctrgcn_head.py:34-63"
+ },
+ "7825": {
+ "file_id": 570,
+ "content": "Class constructor for a neural network head with optional dropout. Initializes layers, applies weight initialization, and defines the forward pass function.",
+ "type": "comment"
+ },
+ "7826": {
+ "file_id": 570,
+ "content": " return self.fc(x)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ctrgcn_head.py:65-65"
+ },
+ "7827": {
+ "file_id": 570,
+ "content": "This code snippet is from the ctrgcn_head class, and it returns the result of passing the input x through a fully connected layer (fc). The purpose might be to process the features extracted by the previous layers in the model for making predictions or generating output.",
+ "type": "comment"
+ },
+ "7828": {
+ "file_id": 571,
+ "content": "/paddlevideo/modeling/heads/i3d_head.py",
+ "type": "filepath"
+ },
+ "7829": {
+ "file_id": 571,
+ "content": "This code defines an I3D classification head in PaddleVideo with options for loss, pooling type, dropout ratio and initialization standard deviation. It performs adaptive average pooling, dropout, linear layer, and has a learning rate of 10.0.",
+ "type": "summary"
+ },
+ "7830": {
+ "file_id": 571,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nfrom paddle import ParamAttr\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\nfrom .base import BaseHead\n@HEADS.register()\nclass I3DHead(BaseHead):\n \"\"\"Classification head for I3D.\n Args:\n num_classes (int): Number of classes to be classified.\n in_channels (int): Number of channels in input feature.\n loss_cls (dict): Config for building loss.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/i3d_head.py:1-31"
+ },
+ "7831": {
+ "file_id": 571,
+ "content": "This code snippet imports necessary libraries and defines a class called \"I3DHead\" which is a classification head for I3D models. It takes in arguments like the number of classes to be classified, the input channel size, and configuration for building loss. The code is part of PaddleVideo library and registered with HEADS registry.",
+ "type": "comment"
+ },
+ "7832": {
+ "file_id": 571,
+ "content": " Default: dict(name='CrossEntropyLoss')\n spatial_type (str): Pooling type in spatial dimension. Default: 'avg'.\n drop_ratio (float): Probability of dropout layer. Default: 0.5.\n std (float): Std value for Initiation. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to be used to initialize\n the head.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n spatial_type='avg',\n drop_ratio=0.5,\n std=0.01,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.spatial_type = spatial_type\n self.drop_ratio = drop_ratio\n self.stdv = std\n if self.drop_ratio != 0:\n self.dropout = nn.Dropout(p=self.drop_ratio)\n else:\n self.dropout = None\n self.fc = nn.Linear(\n self.in_channels,\n self.num_classes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/i3d_head.py:32-59"
+ },
+ "7833": {
+ "file_id": 571,
+ "content": "Class constructor for a head, with optional parameters for loss configuration, pooling type in spatial dimension, dropout ratio, and standard deviation for initialization. Initializes the base class, sets attributes, and optionally adds a Dropout layer if drop_ratio is non-zero.",
+ "type": "comment"
+ },
+ "7834": {
+ "file_id": 571,
+ "content": " weight_attr=ParamAttr(learning_rate=10.0),\n bias_attr=ParamAttr(learning_rate=10.0),\n )\n if self.spatial_type == 'avg':\n # use `nn.AdaptiveAvgPool3d` to adaptively match the in_channels.\n self.avg_pool = nn.AdaptiveAvgPool3D((1, 1, 1))\n else:\n self.avg_pool = None\n def init_weights(self):\n \"\"\"Initiate the parameters from scratch.\"\"\"\n weight_init_(self.fc, 'Normal', 'fc_0.w_0', 'fc_0.b_0', std=self.stdv)\n def forward(self, x):\n \"\"\"Defines the computation performed at every call.\n Args:\n x (torch.Tensor): The input data.\n Returns:\n torch.Tensor: The classification scores for input samples.\n \"\"\"\n # [N, in_channels, 4, 7, 7]\n if self.avg_pool is not None:\n x = self.avg_pool(x)\n # [N, in_channels, 1, 1, 1]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N, in_channels, 1, 1, 1]\n N = paddle.shape(x)[0]\n x = x.reshape([N, -1])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/i3d_head.py:60-91"
+ },
+ "7835": {
+ "file_id": 571,
+ "content": "Function: I3D Head\nPurpose: To process and classify the extracted features from an I3D network\nKey Operations: Adaptive average pooling, dropout, and a linear layer for classification\nLearning Rate: Set to 10.0 for weights and bias parameters",
+ "type": "comment"
+ },
+ "7836": {
+ "file_id": 571,
+ "content": " # [N, in_channels]\n cls_score = self.fc(x)\n # [N, num_classes]\n return cls_score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/i3d_head.py:92-95"
+ },
+ "7837": {
+ "file_id": 571,
+ "content": "This code snippet represents the output layer of a classification head in PaddleVideo. It takes input 'x' and passes it through 'self.fc', which is presumably a fully connected (FC) layer, producing 'cls_score'. The result is then returned as the final classification score for each sample. The shape of the output is [N, num_classes], where N is the batch size.",
+ "type": "comment"
+ },
+ "7838": {
+ "file_id": 572,
+ "content": "/paddlevideo/modeling/heads/movinet_head.py",
+ "type": "filepath"
+ },
+ "7839": {
+ "file_id": 572,
+ "content": "Class MoViNetHead extends BaseHead and registers itself with the HEADS registry. It initializes without any specific parameters and its forward function simply returns input 'x' without any modifications.",
+ "type": "summary"
+ },
+ "7840": {
+ "file_id": 572,
+ "content": "import collections.abc\ncontainer_abcs = collections.abc\nfrom ..registry import HEADS\nfrom .base import BaseHead\nfrom ..builder import build_loss\n@HEADS.register()\nclass MoViNetHead(BaseHead):\n def __init__(self):\n super().__init__()\n def forward(self, x, *args):\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/movinet_head.py:1-15"
+ },
+ "7841": {
+ "file_id": 572,
+ "content": "Class MoViNetHead extends BaseHead and registers itself with the HEADS registry. It initializes without any specific parameters and its forward function simply returns input 'x' without any modifications.",
+ "type": "comment"
+ },
+ "7842": {
+ "file_id": 573,
+ "content": "/paddlevideo/modeling/heads/ms_tcn_head.py",
+ "type": "filepath"
+ },
+ "7843": {
+ "file_id": 573,
+ "content": "The code defines a model and calculates loss, F1 score, and edit scores for recognition tasks. It retrieves label start/end times from recognized and ground truth sequences, then iterates through labels to calculate F-score for overlapping segments, updating tp, fp, fn counts and returning the F-score as a float value.",
+ "type": "summary"
+ },
+ "7844": {
+ "file_id": 573,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom paddle import ParamAttr\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass MSTCNHead(BaseHead):\n def __init__(self, num_classes, in_channels):\n super().__init__(num_classes, in_channels)\n self.ce = nn.CrossEntropyLoss(ignore_index=-100)\n self.mse = nn.MSELoss(reduction='none')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ms_tcn_head.py:1-33"
+ },
+ "7845": {
+ "file_id": 573,
+ "content": "This code defines the MSTCNHead class, a head for PaddleVideo's MS-TCN model. It inherits from BaseHead and initializes a CrossEntropyLoss and Mean Squared Error loss function.",
+ "type": "comment"
+ },
+ "7846": {
+ "file_id": 573,
+ "content": " self.num_classes = num_classes\n # cls score\n self.overlap = 0.5\n def forward(self, x):\n \"\"\"MS-TCN no head\n \"\"\"\n return x\n def loss(self, output, video_gt):\n \"\"\"calculate loss\n \"\"\"\n output_transpose = paddle.transpose(output, [2, 0, 1])\n ce_x = paddle.reshape(output_transpose,\n (output_transpose.shape[0] *\n output_transpose.shape[1], self.num_classes))\n ce_y = video_gt[0, :]\n ce_loss = self.ce(ce_x, ce_y)\n loss = ce_loss\n mse = self.mse(F.log_softmax(output[:, :, 1:], axis=1),\n F.log_softmax(output.detach()[:, :, :-1], axis=1))\n mse = paddle.clip(mse, min=0, max=16)\n mse_loss = 0.15 * paddle.mean(mse)\n loss += mse_loss\n return loss\n def get_F1_score(self, predicted, groundTruth):\n recog_content = list(predicted.numpy())\n gt_content = list(groundTruth[0].numpy())\n # cls score\n correct = 0",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ms_tcn_head.py:34-68"
+ },
+ "7847": {
+ "file_id": 573,
+ "content": "The code defines a class for the MS-TCN head, which calculates loss and F1 score. The forward function returns the input as is. The loss function transposes output tensor, computes cross-entropy (CE) loss, and adds mean squared error (MSE) loss with weight 0.15. The get_F1_score function converts predicted and ground truth to lists, counts correct classifications, and returns F1 score.",
+ "type": "comment"
+ },
+ "7848": {
+ "file_id": 573,
+ "content": " total = 0\n edit = 0\n for i in range(len(gt_content)):\n total += 1\n if gt_content[i] == recog_content[i]:\n correct += 1\n edit_num = self.edit_score(recog_content, gt_content)\n edit += edit_num\n tp, fp, fn = self.f_score(recog_content, gt_content, self.overlap)\n # cls metric\n precision = tp / float(tp + fp)\n recall = tp / float(fp + fn)\n if precision + recall > 0.0:\n f1 = 2.0 * (precision * recall) / (precision + recall)\n else:\n f1 = 0.0\n f1 = np.nan_to_num(f1)\n return f1\n def get_labels_start_end_time(self, frame_wise_labels):\n labels = []\n starts = []\n ends = []\n last_label = frame_wise_labels[0]\n labels.append(frame_wise_labels[0])\n starts.append(0)\n for i in range(len(frame_wise_labels)):\n if frame_wise_labels[i] != last_label:\n labels.append(frame_wise_labels[i])\n starts.append(i)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ms_tcn_head.py:69-105"
+ },
+ "7849": {
+ "file_id": 573,
+ "content": "This code calculates the F1 score based on a given sequence of content and then extracts labels, start times, and end times from frame-wise labels. It iterates through the sequence to determine correct and incorrect elements, as well as false positives and negatives for the F1 score calculation. The extracted labels, starts, and ends are stored in separate lists.",
+ "type": "comment"
+ },
+ "7850": {
+ "file_id": 573,
+ "content": " ends.append(i)\n last_label = frame_wise_labels[i]\n ends.append(i + 1)\n return labels, starts, ends\n def levenstein(self, p, y, norm=False):\n m_row = len(p)\n n_col = len(y)\n D = np.zeros([m_row + 1, n_col + 1], np.float)\n for i in range(m_row + 1):\n D[i, 0] = i\n for i in range(n_col + 1):\n D[0, i] = i\n for j in range(1, n_col + 1):\n for i in range(1, m_row + 1):\n if y[j - 1] == p[i - 1]:\n D[i, j] = D[i - 1, j - 1]\n else:\n D[i, j] = min(D[i - 1, j] + 1, D[i, j - 1] + 1,\n D[i - 1, j - 1] + 1)\n if norm:\n score = (1 - D[-1, -1] / max(m_row, n_col)) * 100\n else:\n score = D[-1, -1]\n return score\n def edit_score(self, recognized, ground_truth, norm=True):\n P, _, _ = self.get_labels_start_end_time(recognized)\n Y, _, _ = self.get_labels_start_end_time(ground_truth)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ms_tcn_head.py:106-137"
+ },
+ "7851": {
+ "file_id": 573,
+ "content": "This code defines two functions: \"labels_start_end\" and \"edit_score\". The first function takes in frame-wise labels, starts, and ends and returns the labels, starts, and ends. The second function calculates the edit score between recognized text and ground truth using a dynamic programming approach, specifically Levenshtein distance algorithm. It normalizes the scores if norm is True, and returns the unnormalized score otherwise.",
+ "type": "comment"
+ },
+ "7852": {
+ "file_id": 573,
+ "content": " return self.levenstein(P, Y, norm)\n def f_score(self, recognized, ground_truth, overlap):\n p_label, p_start, p_end = self.get_labels_start_end_time(recognized)\n y_label, y_start, y_end = self.get_labels_start_end_time(ground_truth)\n tp = 0\n fp = 0\n hits = np.zeros(len(y_label))\n for j in range(len(p_label)):\n intersection = np.minimum(p_end[j], y_end) - np.maximum(\n p_start[j], y_start)\n union = np.maximum(p_end[j], y_end) - np.minimum(\n p_start[j], y_start)\n IoU = (1.0 * intersection / union) * (\n [p_label[j] == y_label[x] for x in range(len(y_label))])\n # Get the best scoring segment\n idx = np.array(IoU).argmax()\n if IoU[idx] >= overlap and not hits[idx]:\n tp += 1\n hits[idx] = 1\n else:\n fp += 1\n fn = len(y_label) - sum(hits)\n return float(tp), float(fp), float(fn)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/ms_tcn_head.py:138-165"
+ },
+ "7853": {
+ "file_id": 573,
+ "content": "This code calculates the F-score for overlapping segments of labels in two sequences. It first retrieves the start and end times for each label in the recognized and ground truth sequences, then iterates through each label in the recognized sequence to calculate the intersection and union between the current recognized segment and each segment in the ground truth sequence. The code then determines if there is an overlap between the segments, updates the true positive (tp), false positive (fp), and false negative (fn) counts accordingly, and finally returns the F-score as a float value.",
+ "type": "comment"
+ },
+ "7854": {
+ "file_id": 574,
+ "content": "/paddlevideo/modeling/heads/pptimesformer_head.py",
+ "type": "filepath"
+ },
+ "7855": {
+ "file_id": 574,
+ "content": "The code defines a PaddlePaddle class \"ppTimeSformerHead\" as a head for the TimeSformer model, extending BaseHead and initializing fully connected layers with truncated normal distribution.",
+ "type": "summary"
+ },
+ "7856": {
+ "file_id": 574,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom paddle.nn import Linear\nfrom ..registry import HEADS\nfrom ..weight_init import trunc_normal_, weight_init_\nfrom .base import BaseHead\nfrom paddle import ParamAttr\nfrom paddle.regularizer import L2Decay\n@HEADS.register()\nclass ppTimeSformerHead(BaseHead):\n \"\"\"TimeSformerHead Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptimesformer_head.py:1-30"
+ },
+ "7857": {
+ "file_id": 574,
+ "content": "This code defines a class called \"ppTimeSformerHead\" which is a head for the TimeSformer model in PaddlePaddle framework. It extends the BaseHead class, and has attributes such as num_classes, in_channels. The class also registers itself in the HEADS registry of the PaddleVideo module. The code uses Linear and ParamAttr from paddle.nn and weight_init from .base, and imports trunc_normal_ and L2Decay from other modules.",
+ "type": "comment"
+ },
+ "7858": {
+ "file_id": 574,
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n std=0.02,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.std = std\n self.fc = Linear(self.in_channels,\n self.num_classes,\n bias_attr=ParamAttr(regularizer=L2Decay(0.0)))\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'TruncatedNormal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.0,\n std=self.std)\n # NOTE: Temporarily use trunc_normal_ instead of TruncatedNormal",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptimesformer_head.py:31-58"
+ },
+ "7859": {
+ "file_id": 574,
+ "content": "The code defines a class named \"PPTimesformerHead\" with an __init__ method that takes parameters such as num_classes, in_channels, loss_cfg (with default value), std (with default 0.02), and optional kwargs. It initializes superclass attributes, sets self.std, and initializes the FC layer parameters using weight_init_. The TruncatedNormal initialization method is used with specific attribute names.",
+ "type": "comment"
+ },
+ "7860": {
+ "file_id": 574,
+ "content": " trunc_normal_(self.fc.weight, std=self.std)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # XXX: check dropout location!\n # x.shape = [N, embed_dim]\n score = self.fc(x)\n # [N, num_class]\n # x = F.softmax(x) # NOTE remove\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptimesformer_head.py:59-74"
+ },
+ "7861": {
+ "file_id": 574,
+ "content": "The code defines a head for the PPTimesformer model. It initializes the fully connected layer (fc) with truncated normal distribution and defines the forward pass, which involves passing input through fc to generate scores for classification tasks.",
+ "type": "comment"
+ },
+ "7862": {
+ "file_id": 575,
+ "content": "/paddlevideo/modeling/heads/pptsm_head.py",
+ "type": "filepath"
+ },
+ "7863": {
+ "file_id": 575,
+ "content": "The code defines a ppTSMHead class, a subclass of TSNHead with L2Decay regularizer. It initializes the PPTSM model head with average pooling and dropout, defining an 'init_weights' function for FC layer parameters. This is part of the PaddlePaddle Video library.",
+ "type": "summary"
+ },
+ "7864": {
+ "file_id": 575,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn import Linear\nfrom paddle.regularizer import L2Decay\nfrom .tsn_head import TSNHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass ppTSMHead(TSNHead):\n \"\"\" ppTSM Head\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.\n loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsm_head.py:1-31"
+ },
+ "7865": {
+ "file_id": 575,
+ "content": "This code defines a ppTSMHead class, which is a subclass of TSNHead. It has arguments such as num_classes, in_channels, and loss_cfg. The class is registered under the HEADS registry for future use. The L2Decay regularizer is used, and weight initialization is performed using the weight_init function.",
+ "type": "comment"
+ },
+ "7866": {
+ "file_id": 575,
+ "content": " drop_ratio(float): drop ratio. Default: 0.8.\n std(float): Std(Scale) value in normal initilizar. Default: 0.001.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(\n self,\n num_classes,\n in_channels, # NOTE: 2048 for >= R50, 512 for <= R34\n drop_ratio=0.8,\n std=0.01,\n data_format=\"NCHW\",\n num_seg=8,\n **kwargs):\n super().__init__(num_classes,\n in_channels,\n drop_ratio=drop_ratio,\n std=std,\n data_format=data_format,\n **kwargs)\n self.fc = Linear(self.in_channels,\n self.num_classes,\n weight_attr=ParamAttr(learning_rate=5.0,\n regularizer=L2Decay(1e-4)),\n bias_attr=ParamAttr(learning_rate=10.0,\n regularizer=L2Decay(0.0)))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsm_head.py:32-58"
+ },
+ "7867": {
+ "file_id": 575,
+ "content": "This code defines a class with an __init__ method that takes arguments for number of classes, input channels, dropout ratio, std value, data format, and optional keyword arguments. It initializes the base class and sets up a linear layer (self.fc) with specified learning rates and regularizers.",
+ "type": "comment"
+ },
+ "7868": {
+ "file_id": 575,
+ "content": " self.stdv = std\n self.num_seg = num_seg\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc, 'Normal', 'fc_0.w_0', 'fc_0.b_0', std=self.stdv)\n def forward(self, x, num_seg=None):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n #XXX: check dropout location!\n # [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x)\n # [N * num_segs, in_channels, 1, 1]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N * num_seg, in_channels, 1, 1]\n num_seg = num_seg if num_seg is not None else self.num_seg\n x = paddle.reshape(x, [-1, num_seg, x.shape[1]])\n # [N, num_seg, in_channels]\n x = paddle.mean(x, axis=1)\n # [N, in_channels]\n x = paddle.reshape(x, shape=[-1, self.in_channels])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsm_head.py:59-87"
+ },
+ "7869": {
+ "file_id": 575,
+ "content": "The code initializes a head for the PPTSM model, which includes an average pooling layer, dropout if specified, and reshaping operations. It then returns the classification scores for input samples. The 'init_weights' function initializes the FC layer parameters with normal distribution using the given standard deviation (stdv).",
+ "type": "comment"
+ },
+ "7870": {
+ "file_id": 575,
+ "content": " # [N, in_channels]\n score = self.fc(x)\n # [N, num_class]\n #x = F.softmax(x) #NOTE remove\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsm_head.py:88-92"
+ },
+ "7871": {
+ "file_id": 575,
+ "content": "This code snippet is part of the PaddlePaddle Video (PaddleVideo) library. It defines a function within a class called \"pptsm_head\". The function takes input 'x' and performs a fully connected operation using \"self.fc\", returning the scores in the form of \"score\" with dimensions [N, in_channels]. The line \"#x = F.softmax(x) #NOTE remove\" was likely removed from the code, but its original purpose would have been to apply softmax function on 'x' and return the normalized probabilities.",
+ "type": "comment"
+ },
+ "7872": {
+ "file_id": 576,
+ "content": "/paddlevideo/modeling/heads/pptsn_head.py",
+ "type": "filepath"
+ },
+ "7873": {
+ "file_id": 576,
+ "content": "This Python code implements a PaddlePaddle neural network head for classification tasks using ppTSN Head, initializing the base class and applying dropout regularization with an FC layer. The init_weights function sets the FC layer's initial weights.",
+ "type": "summary"
+ },
+ "7874": {
+ "file_id": 576,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn import AdaptiveAvgPool2D, Linear, Dropout\nfrom paddle.regularizer import L2Decay\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass ppTSNHead(BaseHead):\n \"\"\"ppTSN Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsn_head.py:1-30"
+ },
+ "7875": {
+ "file_id": 576,
+ "content": "This code is the Python implementation of ppTSN Head, a classification model head used in PaddleVideo. The class has the number of classes and input channels as arguments. It inherits from BaseHead and is registered to the HEADS registry using @HEADS.register(). The code also imports necessary libraries and functions for its operations such as Linear, AdaptiveAvgPool2D, Dropout, ParamAttr, L2Decay, paddle.nn, and PaddleVideo's base and weight_init modules.",
+ "type": "comment"
+ },
+ "7876": {
+ "file_id": 576,
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n drop_ratio(float): drop ratio. Default: 0.4.\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n data_format(str): data format of input tensor in ['NCHW', 'NHWC']. Default: 'NCHW'.\n fclr5(bool): Whether to increase the learning rate of the fully connected layer. Default: True\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n drop_ratio=0.4,\n std=0.01,\n data_format=\"NCHW\",\n fclr5=True,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.drop_ratio = drop_ratio\n self.std = std\n # NOTE: global pool performance\n self.avgpool2d = AdaptiveAvgPool2D((1, 1), data_format=data_format)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsn_head.py:31-54"
+ },
+ "7877": {
+ "file_id": 576,
+ "content": "This code defines a class with an __init__ method, taking parameters such as num_classes, in_channels, loss_cfg, drop_ratio, std, data_format, and fclr5. It initializes the base class and sets the drop_ratio, std, and creates an AdaptiveAvgPool2D object for global pooling performance.",
+ "type": "comment"
+ },
+ "7878": {
+ "file_id": 576,
+ "content": " if self.drop_ratio != 0:\n self.dropout = Dropout(p=self.drop_ratio)\n else:\n self.dropout = None\n self.fc = Linear(\n self.in_channels,\n self.num_classes,\n weight_attr=ParamAttr(learning_rate=5.0 if fclr5 else 1.0,\n regularizer=L2Decay(1e-4)),\n bias_attr=ParamAttr(learning_rate=10.0 if fclr5 else 1.0,\n regularizer=L2Decay(0.0)))\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'Normal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.,\n std=self.std)\n def forward(self, x, num_seg=8):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsn_head.py:56-84"
+ },
+ "7879": {
+ "file_id": 576,
+ "content": "This code initializes and defines a PaddlePaddle neural network head for classification tasks. It includes optional dropout regularization, an FC layer with learnable parameters, and a forward function to process input data. The init_weights function is used to set the initial weights of the FC layer.",
+ "type": "comment"
+ },
+ "7880": {
+ "file_id": 576,
+ "content": " \"\"\"\n # XXX: check dropout location!\n # [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x)\n # [N * num_segs, in_channels, 1, 1]\n x = paddle.reshape(x, [-1, num_seg, x.shape[1]])\n # [N, num_seg, in_channels]\n x = paddle.mean(x, axis=1)\n # [N, in_channels]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N, in_channels]\n x = paddle.reshape(x, shape=[-1, self.in_channels])\n # [N, in_channels]\n score = self.fc(x)\n # [N, num_class]\n # x = F.softmax(x) # NOTE remove\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/pptsn_head.py:85-103"
+ },
+ "7881": {
+ "file_id": 576,
+ "content": "This code snippet is responsible for processing the input and output of a PPTSN head model. It performs average pooling, reshapes the tensor, calculates the mean along an axis, applies dropout if applicable, reshapes again, and finally passes the result through a fully connected layer to produce scores.",
+ "type": "comment"
+ },
+ "7882": {
+ "file_id": 577,
+ "content": "/paddlevideo/modeling/heads/roi_extractor.py",
+ "type": "filepath"
+ },
+ "7883": {
+ "file_id": 577,
+ "content": "RoIAlign is a class for region of interest alignment. It takes features, regions of interest (roi), and number of roi as inputs, and uses PaddlePaddle's roi_align operation to extract aligned features.",
+ "type": "summary"
+ },
+ "7884": {
+ "file_id": 577,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\n#@register\nclass RoIAlign(object):\n def __init__(self,\n resolution=14,\n spatial_scale=0.0625,\n sampling_ratio=0,\n aligned=False):\n super(RoIAlign, self).__init__()\n self.resolution = resolution\n self.spatial_scale = spatial_scale\n self.sampling_ratio = sampling_ratio\n self.aligned = aligned\n def __call__(self, feats, roi, rois_num):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_extractor.py:1-31"
+ },
+ "7885": {
+ "file_id": 577,
+ "content": "RoIAlign is a class used for region of interest alignment. It takes features, regions of interest (roi), and the number of roi as inputs. The aligned parameter specifies whether to return aligned features or not.",
+ "type": "comment"
+ },
+ "7886": {
+ "file_id": 577,
+ "content": " roi = paddle.concat(roi) if len(roi) > 1 else roi[0]\n rois_num = paddle.to_tensor(rois_num, dtype='int32')\n rois_num = paddle.cast(rois_num, dtype='int32')\n if len(feats) == 1:\n roi_feat = paddle.vision.ops.roi_align(feats,\n roi,\n rois_num,\n self.resolution,\n self.spatial_scale,\n self.sampling_ratio,\n self.aligned)\n else:\n rois_feat_list = []\n roi_feat = paddle.vision.ops.roi_align(feats,\n roi,\n rois_num,\n self.resolution,\n self.spatial_scale,\n self.sampling_ratio,\n self.aligned)\n return roi_feat",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_extractor.py:32-53"
+ },
+ "7887": {
+ "file_id": 577,
+ "content": "This code concatenates ROIs and ensures correct data type, then uses the PaddlePaddle library's roi_align operation to extract features from input features (feats) based on ROIs. If there is only one feature, it performs alignment for all ROIs. Otherwise, it creates a list of aligned feature tensors.",
+ "type": "comment"
+ },
+ "7888": {
+ "file_id": 578,
+ "content": "/paddlevideo/modeling/heads/roi_head.py",
+ "type": "filepath"
+ },
+ "7889": {
+ "file_id": 578,
+ "content": "The code introduces a Non-Maximum Suppression function for bounding boxes and defines the AVARoIHead class, an object detection layer performing ROI alignment with bbox loss calculation, image assignment & sampling, and result returning. The simple_test function tests detection without augmentation.",
+ "type": "summary"
+ },
+ "7890": {
+ "file_id": 578,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nfrom .. import builder\nfrom ..registry import HEADS\ndef bbox2result(bboxes, labels, num_classes, img_shape, thr=0.01):\n \"\"\"Convert detection results to a list of numpy arrays. \"\"\"\n if len(bboxes) == 0:\n return list(np.zeros((num_classes - 1, 0, 5), dtype=np.float32))\n else:\n bboxes = bboxes[0]\n labels = labels\n img_shape_np = img_shape",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:1-29"
+ },
+ "7891": {
+ "file_id": 578,
+ "content": "This code defines the function bbox2result, which takes in bounding box coordinates (bboxes), labels, number of classes (num_classes), image shape (img_shape) and a threshold value (thr). The function returns a list of numpy arrays representing the detection results. If there are no detections (i.e., bboxes is empty), it returns an empty list of zeros for each class.",
+ "type": "comment"
+ },
+ "7892": {
+ "file_id": 578,
+ "content": " img_h, img_w = img_shape_np[0][0], img_shape_np[0][1]\n img_w = paddle.cast(img_w, dtype='int32')\n img_h = paddle.cast(img_h, dtype='int32')\n bboxes[:, 0::2] /= img_w\n bboxes[:, 1::2] /= img_h\n # We only handle multilabel now\n assert labels.shape[-1] > 1\n scores = labels # rename\n thr = (thr, ) * num_classes if isinstance(thr, float) else thr\n assert scores.shape[1] == num_classes\n assert len(thr) == num_classes\n result = []\n for i in range(num_classes - 1):\n #step1. 对该类, 每个bbox的得分是否大于阈值\n where = scores[:, i + 1] > thr[i + 1]\n where = paddle.nonzero(where) # index\n bboxes_select = paddle.index_select(x=bboxes, index=where)\n bboxes_select = bboxes_select[:, :4]\n scores_select = paddle.index_select(x=scores, index=where)\n scores_select = scores_select[:, i + 1:i + 2]\n result.append(\n #对于step1中得分大于阈值的bbox(可能为空), 将bbox及在该类的score放入result列表.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:30-59"
+ },
+ "7893": {
+ "file_id": 578,
+ "content": "This code performs Non-Maximum Suppression (NMS) on bounding boxes and scores to filter out overlapping regions. It iterates through each class, selects bounding boxes and their corresponding scores that are above a certain threshold for each class, and appends them to the result list.",
+ "type": "comment"
+ },
+ "7894": {
+ "file_id": 578,
+ "content": " paddle.concat((bboxes_select, scores_select), axis=1).numpy())\n return result\n@HEADS.register()\nclass AVARoIHead(nn.Layer):\n def __init__(self,\n assigner,\n sampler,\n pos_weight=1.0,\n action_thr=0.0,\n bbox_roi_extractor=None,\n bbox_head=None,\n train_cfg=None,\n test_cfg=None):\n super().__init__()\n self.assigner = assigner\n self.sampler = sampler\n self.pos_weight = pos_weight\n self.action_thr = action_thr\n self.init_assigner_sampler()\n if bbox_head is not None:\n self.init_bbox_head(bbox_roi_extractor, bbox_head)\n def init_assigner_sampler(self):\n \"\"\"Initialize assigner and sampler.\"\"\"\n self.bbox_assigner = None\n self.bbox_sampler = None\n self.bbox_assigner = builder.build_assigner(self.assigner)\n self.bbox_sampler = builder.build_sampler(self.sampler, context=self)\n def init_bbox_head(self, bbox_roi_extractor, bbox_head):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:60-93"
+ },
+ "7895": {
+ "file_id": 578,
+ "content": "The code defines a class named AVARoIHead, which is a PaddlePaddle layer for object detection. It initializes the assigner and sampler, and optionally initializes the bbox_head (bounding box regression head). The method init_assigner_sampler initializes the bbox_assigner and bbox_sampler from the passed arguments. The method init_bbox_head initializes the bounding box regression head if the bbox_head is provided. This class registers with HEADS, which may be a registry or a list of defined classes.",
+ "type": "comment"
+ },
+ "7896": {
+ "file_id": 578,
+ "content": " \"\"\"Initialize ``bbox_head``\"\"\"\n self.bbox_roi_extractor = builder.build_roi_extractor(\n bbox_roi_extractor)\n self.bbox_head = builder.build_head(bbox_head)\n def _bbox_forward(self, x, rois, rois_num):\n bbox_feat = self.bbox_roi_extractor(x, rois, rois_num)\n cls_score, bbox_pred = self.bbox_head(\n bbox_feat, rois, rois_num\n ) #deal with: when roi's width or height = 0 , roi_align is wrong\n bbox_results = dict(cls_score=cls_score,\n bbox_pred=bbox_pred,\n bbox_feats=bbox_feat)\n return bbox_results\n def _bbox_forward_train(self, x, sampling_results, gt_bboxes, gt_labels):\n \"\"\"Run forward function and calculate loss for box head in training.\"\"\"\n rois = [res.bboxes for res in sampling_results]\n rois_num = [res.bboxes.shape[0] for res in sampling_results]\n bbox_results = self._bbox_forward(x, rois, rois_num)\n bbox_targets = self.bbox_head.get_targets(sampling_results, gt_bboxes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:94-114"
+ },
+ "7897": {
+ "file_id": 578,
+ "content": "This code initializes the bbox_head and defines the _bbox_forward function for feature extraction and prediction, as well as the _bbox_forward_train function for training purposes. It also handles situations where ROI's width or height equals 0 by correcting the roi_align operation.",
+ "type": "comment"
+ },
+ "7898": {
+ "file_id": 578,
+ "content": " gt_labels, self.pos_weight)\n loss_bbox = self.bbox_head.loss(bbox_results['cls_score'], bbox_targets)\n bbox_results.update(loss_bbox=loss_bbox)\n return bbox_results\n def train_step(self, x, img_metas, proposal_list, gt_bboxes, gt_labels):\n #1. assign gts and sample proposals\n num_imgs = len(img_metas[0])\n sampling_results = []\n for i in range(num_imgs):\n assign_result = self.bbox_assigner.assign(proposal_list[i],\n gt_bboxes[i],\n gt_labels[i])\n sampling_result = self.bbox_sampler.sample(assign_result,\n proposal_list[i],\n gt_bboxes[i],\n gt_labels[i])\n sampling_results.append(sampling_result)\n #2. forward and loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:115-134"
+ },
+ "7899": {
+ "file_id": 578,
+ "content": "The code defines a ROI head that calculates the bbox loss and performs assignment and sampling for each image in a batch. It takes input images, proposal list, ground truth bounding boxes, and labels as parameters and returns results containing loss_bbox.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/79.json b/docs/data/79.json
new file mode 100644
index 000000000..21071019c
--- /dev/null
+++ b/docs/data/79.json
@@ -0,0 +1,541 @@
+{
+ "7900": {
+ "file_id": 578,
+ "content": " bbox_results = self._bbox_forward_train(x, sampling_results, gt_bboxes,\n gt_labels)\n losses = dict()\n losses.update(bbox_results['loss_bbox'])\n return losses\n def simple_test(self, x, proposal_list, img_shape, rescale=False):\n x_shape = x[0].shape\n #assert x_shape[0] == 1, 'only accept 1 sample at test mode'\n det_bboxes, det_labels = self.simple_test_bboxes(x,\n img_shape,\n proposal_list,\n self.action_thr,\n rescale=rescale)\n bbox_results = bbox2result(det_bboxes, det_labels,\n self.bbox_head.num_classes, img_shape,\n self.action_thr)\n return [bbox_results]\n def simple_test_bboxes(self,\n x,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:135-158"
+ },
+ "7901": {
+ "file_id": 578,
+ "content": "The code contains two main functions, \"simple_test\" and \"_bbox_forward_train\". The former is for testing the model in simple test mode by taking input x, proposal list, img_shape, and rescale flag. It calculates det_bboxes and det_labels using the function \"simple_test_bboxes\". Then it uses bbox2result to convert det_bboxes and det_labels into bbox_results. The latter function takes input x, sampling results, gt_bboxes, and gt_labels to calculate bbox results and losses. It updates the losses dictionary with \"loss_bbox\" and returns the losses.",
+ "type": "comment"
+ },
+ "7902": {
+ "file_id": 578,
+ "content": " img_shape,\n proposals,\n action_thr,\n rescale=False):\n \"\"\"Test only det bboxes without augmentation.\"\"\"\n rois = [proposals]\n rois_num = [rois[0].shape[0]]\n bbox_results = self._bbox_forward(x, rois, rois_num)\n cls_score = bbox_results['cls_score']\n crop_quadruple = np.array([0, 0, 1, 1])\n flip = False\n det_bboxes, det_labels = self.bbox_head.get_det_bboxes(\n rois,\n cls_score,\n img_shape,\n flip=flip,\n crop_quadruple=crop_quadruple)\n return det_bboxes, det_labels",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/roi_head.py:159-177"
+ },
+ "7903": {
+ "file_id": 578,
+ "content": "This function tests only detection bboxes without augmentation. It takes input x, proposals, action_thr, and rescale as parameters. It creates rois and rois_num from the proposals. It then calls _bbox_forward to get cls_score. It sets crop_quadruple and flip to False. Finally, it calls bbox_head's get_det_bboxes to return det_bboxes and det_labels.",
+ "type": "comment"
+ },
+ "7904": {
+ "file_id": 579,
+ "content": "/paddlevideo/modeling/heads/single_straight3d.py",
+ "type": "filepath"
+ },
+ "7905": {
+ "file_id": 579,
+ "content": "This code defines a 3D ROI extractor class and head, performing feature extraction with optional temporal pooling. The forward method executes feature extraction based on input features, RoIs, and number of RoIs, and returns the final output after applying ROI layer and stacking features along axis 2.",
+ "type": "summary"
+ },
+ "7906": {
+ "file_id": 579,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport numpy as np\nfrom ..registry import ROI_EXTRACTORS\nfrom .roi_extractor import RoIAlign\n@ROI_EXTRACTORS.register()\nclass SingleRoIExtractor3D(nn.Layer):\n \"\"\"Extract RoI features from a single level feature map. \"\"\"\n def __init__(self,\n roi_layer_type='RoIAlign',\n featmap_stride=16,\n output_size=16,\n sampling_ratio=0,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/single_straight3d.py:1-28"
+ },
+ "7907": {
+ "file_id": 579,
+ "content": "This code imports necessary libraries and registers a new ROI (Region of Interest) extractor named \"SingleRoIExtractor3D\". This class inherits from nn.Layer and is designed to extract RoI features from a single level feature map with specific options such as roi_layer_type, featmap_stride, output_size, and sampling_ratio.",
+ "type": "comment"
+ },
+ "7908": {
+ "file_id": 579,
+ "content": " pool_mode='avg',\n aligned=True,\n with_temporal_pool=True,\n with_global=False):\n super().__init__()\n self.roi_layer_type = roi_layer_type\n assert self.roi_layer_type in ['RoIPool', 'RoIAlign']\n self.featmap_stride = featmap_stride\n self.spatial_scale = 1. / self.featmap_stride\n self.output_size = output_size\n self.sampling_ratio = sampling_ratio\n self.pool_mode = pool_mode\n self.aligned = aligned\n self.with_temporal_pool = with_temporal_pool\n self.with_global = with_global\n self.roi_layer = RoIAlign(resolution=self.output_size,\n spatial_scale=self.spatial_scale,\n sampling_ratio=self.sampling_ratio,\n aligned=self.aligned)\n def init_weights(self):\n pass\n # The shape of feat is N, C, T, H, W\n def forward(self, feat, rois, rois_num):\n if len(feat) >= 2:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/single_straight3d.py:29-55"
+ },
+ "7909": {
+ "file_id": 579,
+ "content": "This code defines a class for a 3D head that takes input features and regions of interest (ROIs) to extract features using the RoIAlign layer. It also includes an optional temporal pooling operation and an initialization function. The forward method performs feature extraction given the input features, ROIs, and number of ROIs.",
+ "type": "comment"
+ },
+ "7910": {
+ "file_id": 579,
+ "content": " assert self.with_temporal_pool\n if self.with_temporal_pool:\n xi = 0\n for x in feat:\n xi = xi + 1\n y = paddle.mean(x, 2, keepdim=True)\n feat = [paddle.mean(x, 2, keepdim=True) for x in feat]\n feat = paddle.concat(feat, axis=1) # merge slow and fast\n roi_feats = []\n for t in range(feat.shape[2]):\n if type(t) == paddle.static.Variable:\n index = paddle.to_tensor(t)\n else:\n data_index = np.array([t]).astype('int32')\n index = paddle.to_tensor(data_index)\n frame_feat = paddle.index_select(feat, index, axis=2)\n frame_feat = paddle.squeeze(frame_feat,\n axis=2) #axis=2,避免N=1时, 第一维度被删除.\n roi_feat = self.roi_layer(frame_feat, rois, rois_num)\n roi_feats.append(roi_feat)\n ret = paddle.stack(roi_feats, axis=2)\n return ret",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/single_straight3d.py:56-79"
+ },
+ "7911": {
+ "file_id": 579,
+ "content": "This code performs temporal pooling, concatenates slow and fast features, extracts frame-wise features using index selection, squeezes the dimensions to prevent deletion when N=1, applies a ROI layer on each frame, stacks the resulting features along axis 2, and returns the final output.",
+ "type": "comment"
+ },
+ "7912": {
+ "file_id": 580,
+ "content": "/paddlevideo/modeling/heads/slowfast_head.py",
+ "type": "filepath"
+ },
+ "7913": {
+ "file_id": 580,
+ "content": "This code defines 3D head projection classes for PaddleVideo library, initializes a SlowFast head model with dropout regularization and adaptive average pooling, performs convolutional inference, applies softmax activation, averages when not training, and reshapes before returning.",
+ "type": "summary"
+ },
+ "7914": {
+ "file_id": 580,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..registry import HEADS\nfrom .base import BaseHead\nimport paddle\nimport paddle.nn.functional as F\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass SlowFastHead(BaseHead):\n \"\"\"\n ResNe(X)t 3D head.\n This layer performs a fully-connected projection during training, when the\n input size is 1x1x1. It performs a convolutional projection during testing\n when the input size is larger than 1x1x1. If the inputs are from multiple",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/slowfast_head.py:1-30"
+ },
+ "7915": {
+ "file_id": 580,
+ "content": "This code is from the PaddleVideo library and defines a SlowFastHead class for ResNe(X)t 3D head. It performs a fully-connected projection during training and convolutional projection during testing, with different input sizes handled accordingly. The code includes import statements, registration using HEADS registry, and base class inheritance from BaseHead.",
+ "type": "comment"
+ },
+ "7916": {
+ "file_id": 580,
+ "content": " different pathways, the inputs will be concatenated after pooling.\n \"\"\"\n def __init__(self,\n width_per_group,\n alpha,\n beta,\n num_classes,\n num_frames,\n crop_size,\n dropout_rate,\n pool_size_ratio=[[1, 1, 1], [1, 1, 1]],\n loss_cfg=dict(name='CrossEntropyLoss'),\n multigrid_short=False,\n **kwargs):\n \"\"\"\n ResNetBasicHead takes p pathways as input where p in [1, infty].\n Args:\n dim_in (list): the list of channel dimensions of the p inputs to the\n ResNetHead.\n num_classes (int): the channel dimensions of the p outputs to the\n ResNetHead.\n pool_size (list): the list of kernel sizes of p spatial temporal\n poolings, temporal pool kernel size, spatial pool kernel size,\n spatial pool kernel size in order.\n dropout_rate (float): dropout rate. If equal to 0.0, perform no",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/slowfast_head.py:31-56"
+ },
+ "7917": {
+ "file_id": 580,
+ "content": "The code defines a class for SlowFast_Head, which takes different pathways as input and concatenates the inputs after pooling. It has various parameters such as width_per_group, alpha, beta, etc. The ResNetBasicHead takes p pathways as input where p can be in the range of 1 to infinity. It has arguments for dim_in (list), num_classes (int), pool_size (list), and dropout_rate (float).",
+ "type": "comment"
+ },
+ "7918": {
+ "file_id": 580,
+ "content": " dropout.\n \"\"\"\n super().__init__(num_classes, loss_cfg, **kwargs)\n self.multigrid_short = multigrid_short\n self.width_per_group = width_per_group\n self.alpha = alpha\n self.beta = beta\n self.num_classes = num_classes\n self.num_frames = num_frames\n self.crop_size = crop_size\n self.dropout_rate = dropout_rate\n self.pool_size_ratio = pool_size_ratio\n self.dim_in = [\n self.width_per_group * 32,\n self.width_per_group * 32 // self.beta,\n ]\n self.pool_size = [None, None] if self.multigrid_short else [\n [\n self.num_frames // self.alpha // self.pool_size_ratio[0][0],\n self.crop_size // 32 // self.pool_size_ratio[0][1],\n self.crop_size // 32 // self.pool_size_ratio[0][2],\n ],\n [\n self.num_frames // self.pool_size_ratio[1][0],\n self.crop_size // 32 // self.pool_size_ratio[1][1],\n self.crop_size // 32 // self.pool_size_ratio[1][2],",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/slowfast_head.py:57-83"
+ },
+ "7919": {
+ "file_id": 580,
+ "content": "This code is initializing a SlowFast head model with specified parameters such as multigrid_short, width_per_group, alpha, beta, num_classes, num_frames, crop_size, and dropout_rate. It also sets the dimension input (dim_in) based on these parameters, and determines the pool size accordingly based on whether multigrid_short is True or False.",
+ "type": "comment"
+ },
+ "7920": {
+ "file_id": 580,
+ "content": " ],\n ]\n assert (len({len(self.pool_size), len(self.dim_in)\n }) == 1), \"pathway dimensions are not consistent.\"\n self.num_pathways = len(self.pool_size)\n self.dropout = paddle.nn.Dropout(p=self.dropout_rate)\n self.projection = paddle.nn.Linear(\n in_features=sum(self.dim_in),\n out_features=self.num_classes,\n )\n def init_weights(self):\n weight_init_(self.projection,\n \"Normal\",\n bias_value=0.0,\n mean=0.0,\n std=0.01)\n def forward(self, inputs):\n assert (len(inputs) == self.num_pathways\n ), \"Input tensor does not contain {} pathway\".format(\n self.num_pathways)\n pool_out = []\n for pathway in range(self.num_pathways):\n if self.pool_size[pathway] is None:\n tmp_out = F.adaptive_avg_pool3d(x=inputs[pathway],\n output_size=(1, 1, 1),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/slowfast_head.py:84-113"
+ },
+ "7921": {
+ "file_id": 580,
+ "content": "This code initializes a SlowFast head model. It defines the number of pathways, applies dropout regularization, and initializes weights for linear projection. The forward method expects inputs with the same number of pathways as defined in the model. It then performs adaptive average pooling on each input pathway separately.",
+ "type": "comment"
+ },
+ "7922": {
+ "file_id": 580,
+ "content": " data_format=\"NCDHW\")\n else:\n tmp_out = F.avg_pool3d(x=inputs[pathway],\n kernel_size=self.pool_size[pathway],\n stride=1,\n data_format=\"NCDHW\")\n pool_out.append(tmp_out)\n x = paddle.concat(x=pool_out, axis=1)\n x = paddle.transpose(x=x, perm=(0, 2, 3, 4, 1))\n # Perform dropout.\n if self.dropout_rate > 0.0:\n x = self.dropout(x)\n x = self.projection(x)\n # Performs fully convlutional inference.\n if not self.training: # attr of base class\n x = F.softmax(x, axis=4)\n x = paddle.mean(x, axis=[1, 2, 3])\n x = paddle.reshape(x, shape=(x.shape[0], -1))\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/slowfast_head.py:114-137"
+ },
+ "7923": {
+ "file_id": 580,
+ "content": "This code performs pooling and dropout operations on input tensors, followed by projection and fully convolutional inference. It also applies softmax activation and averaging when not in training mode. The resulting tensor is reshaped before returning.",
+ "type": "comment"
+ },
+ "7924": {
+ "file_id": 581,
+ "content": "/paddlevideo/modeling/heads/stgcn_head.py",
+ "type": "filepath"
+ },
+ "7925": {
+ "file_id": 581,
+ "content": "This code creates a STGCNHead class in PaddlePaddle's video modeling library, initializing a convolutional layer and applying forward pass for input x to produce N, C shaped output.",
+ "type": "summary"
+ },
+ "7926": {
+ "file_id": 581,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass STGCNHead(BaseHead):\n \"\"\"\n Head for ST-GCN model.\n Args:\n in_channels: int, input feature channels. Default: 256.\n num_classes: int, number classes. Default: 10.\n \"\"\"\n def __init__(self, in_channels=256, num_classes=10, **kwargs):\n super().__init__(num_classes, in_channels, **kwargs)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/stgcn_head.py:1-32"
+ },
+ "7927": {
+ "file_id": 581,
+ "content": "This code snippet is for the STGCNHead class in PaddlePaddle's video modeling library. It's a subclass of BaseHead with 256 input feature channels and 10 number classes as default values, initialized using super(). This model can be customized further by passing additional keyword arguments.",
+ "type": "comment"
+ },
+ "7928": {
+ "file_id": 581,
+ "content": " self.fcn = nn.Conv2D(in_channels=in_channels,\n out_channels=num_classes,\n kernel_size=1)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer, 'Normal', std=0.02)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n \"\"\"\n x = self.fcn(x)\n x = paddle.reshape_(x, (x.shape[0], -1)) # N,C,1,1 --> N,C\n return x",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/stgcn_head.py:33-50"
+ },
+ "7929": {
+ "file_id": 581,
+ "content": "The code defines a head class with a convolutional layer, initializes its weights using a normal distribution with standard deviation 0.02, and applies the forward pass to input x by passing it through a convolutional layer and reshaping the output to shape N, C (N: number of samples, C: number of classes).",
+ "type": "comment"
+ },
+ "7930": {
+ "file_id": 582,
+ "content": "/paddlevideo/modeling/heads/timesformer_head.py",
+ "type": "filepath"
+ },
+ "7931": {
+ "file_id": 582,
+ "content": "The PaddleVideo's TimeSformerHead class is a model head in the TimeSformer architecture that initializes parameters and defines forward methods for computing output. It uses Linear layers from PaddlePaddle and allows customizing parameters with keyword arguments. The function applies an fc layer to input tensor x and returns classification scores without softmax, with unclear dropout location.",
+ "type": "summary"
+ },
+ "7932": {
+ "file_id": 582,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom paddle.nn import Linear\nfrom ..registry import HEADS\nfrom ..weight_init import trunc_normal_, weight_init_\nfrom .base import BaseHead\n@HEADS.register()\nclass TimeSformerHead(BaseHead):\n \"\"\"TimeSformerHead Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.\n loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/timesformer_head.py:1-29"
+ },
+ "7933": {
+ "file_id": 582,
+ "content": "This code is from PaddleVideo's TimeSformerHead class, which is a head in the modeling module. It is a subclass of BaseHead and has attributes such as num_classes, in_channels, and loss_cfg. The class is registered using HEADS registry, and it uses functions from paddle.nn, Linear, and BaseHead modules. Weight initialization is performed using trunc_normal_ and weight_init functions.",
+ "type": "comment"
+ },
+ "7934": {
+ "file_id": 582,
+ "content": " std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n std=0.02,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.std = std\n self.fc = Linear(self.in_channels, self.num_classes)\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'TruncatedNormal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.0,\n std=self.std)\n # NOTE: Temporarily use trunc_normal_ instead of TruncatedNormal\n trunc_normal_(self.fc.weight, std=self.std)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/timesformer_head.py:30-60"
+ },
+ "7935": {
+ "file_id": 582,
+ "content": "The code snippet defines a class for the TimeSformer head, initializes its parameters and provides a forward method to compute the output of the head. It uses PaddlePaddle's Linear layer and allows setting a specific std value in normal initialization, as well as customizing other parameters with additional keyword arguments. The forward function defines how the head operates on input data x.",
+ "type": "comment"
+ },
+ "7936": {
+ "file_id": 582,
+ "content": " Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # XXX: check dropout location!\n # x.shape = [N, embed_dim]\n score = self.fc(x)\n # [N, num_class]\n # x = F.softmax(x) # NOTE remove\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/timesformer_head.py:61-70"
+ },
+ "7937": {
+ "file_id": 582,
+ "content": "This function applies a fully connected layer (fc) to the input tensor x and returns the classification scores for input samples without applying softmax. The dropout location needs further clarification as indicated by XXX.",
+ "type": "comment"
+ },
+ "7938": {
+ "file_id": 583,
+ "content": "/paddlevideo/modeling/heads/token_shift_head.py",
+ "type": "filepath"
+ },
+ "7939": {
+ "file_id": 583,
+ "content": "The code defines a TokenShiftHead class for classification tasks in Paddle. It inherits from BaseHead, uses Linear module, and returns classification scores after passing input tensor x through fully connected layer self.fc.",
+ "type": "summary"
+ },
+ "7940": {
+ "file_id": 583,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom paddle.nn import Linear\nimport paddle\nfrom ..registry import HEADS\nfrom ..weight_init import trunc_normal_, weight_init_\nfrom .base import BaseHead\n@HEADS.register()\nclass TokenShiftHead(BaseHead):\n \"\"\"TokenShift Transformer Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.\n num_seg(int): The number of segments. Default: 8. ",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/token_shift_head.py:1-30"
+ },
+ "7941": {
+ "file_id": 583,
+ "content": "The code is defining a class called TokenShiftHead, which is a Transformer head for classification tasks. It has attributes such as num_classes, in_channels, and num_seg (defaulted to 8). The class inherits from BaseHead and is registered under the HEADS registry. The code imports necessary modules and functions, and uses Paddle's Linear module for the layer implementation.",
+ "type": "comment"
+ },
+ "7942": {
+ "file_id": 583,
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n ls_eps (float): Label smoothing epsilon. Default: 0.01.\n std (float): Std(Scale) Value in normal initilizar. Default: 0.02.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n num_seg=8,\n loss_cfg=dict(name='CrossEntropyLoss'),\n ls_eps=0.01,\n std=0.02,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, ls_eps)\n self.num_seg = num_seg\n self.std = std\n self.fc = Linear(self.in_channels, self.num_classes)\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'TruncatedNormal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.0,\n std=self.std)\n # NOTE: Temporarily use trunc_normal_ instead of TruncatedNormal",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/token_shift_head.py:31-60"
+ },
+ "7943": {
+ "file_id": 583,
+ "content": "__init__ function initializes the class with specified parameters, and init_weights is used to initialize the FC layer's parameters using truncated normal distribution.",
+ "type": "comment"
+ },
+ "7944": {
+ "file_id": 583,
+ "content": " trunc_normal_(self.fc.weight, std=self.std)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # XXX: check dropout location!\n # x.shape = [N, embed_dim]\n score = self.fc(x)\n # [N*T, num_class]\n _, _m = score.shape\n _t = self.num_seg\n score = score.reshape([-1, _t, _m])\n score = paddle.mean(score, 1) # averaging predictions for every frame\n score = paddle.squeeze(score, axis=1)\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/token_shift_head.py:61-79"
+ },
+ "7945": {
+ "file_id": 583,
+ "content": "This code defines a TokenShiftHead, which performs classification tasks. The forward function takes input tensor x and passes it through fully connected layer self.fc, resulting in classification scores for each sample. It then reshapes the score to average predictions for every frame, finally squeezing the axis to return the final score.",
+ "type": "comment"
+ },
+ "7946": {
+ "file_id": 584,
+ "content": "/paddlevideo/modeling/heads/transnetv2_head.py",
+ "type": "filepath"
+ },
+ "7947": {
+ "file_id": 584,
+ "content": "This code defines the TransNetV2Head class, a type of head used in computer vision models, inheriting from BaseHead with arguments for number of classes, input channels, and loss configuration. It also includes TransNetV2Loss class registered as HEADS registry, and two methods (loss and get_score) for calculating loss and F1 score between predictions and ground truth.",
+ "type": "summary"
+ },
+ "7948": {
+ "file_id": 584,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..losses import TransNetV2Loss\nfrom ...metrics.transnetv2_metric import create_scene_based_summaries\n@HEADS.register()\nclass TransNetV2Head(BaseHead):\n \"\"\"TransNetV2 Head.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name=\"TransNetV2Loss\")\n ):\n super().__init__(num_classes,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/transnetv2_head.py:1-29"
+ },
+ "7949": {
+ "file_id": 584,
+ "content": "This code defines the TransNetV2Head class, which is a type of head used in computer vision models. It inherits from BaseHead and takes arguments for number of classes, input channels, and loss configuration. The TransNetV2Loss class is registered with the HEADS registry to be used by this head. Additionally, the code provides comments about licensing and copyright information.",
+ "type": "comment"
+ },
+ "7950": {
+ "file_id": 584,
+ "content": " in_channels,\n loss_cfg)\n def loss(self, one_hot_pred, one_hot_gt,\n many_hot_pred=None, many_hot_gt=None, reg_losses=None):\n losses = dict()\n loss = self.loss_func(scores, labels, **kwargs)\n f1 = self.get_score(one_hot_pred, one_hot_gt)\n losses['f1'] = f1\n losses['loss'] = loss\n return losses\n def get_score(self, one_hot_pred, one_hot_gt):\n f1 = create_scene_based_summaries(one_hot_pred, one_hot_gt)\n return f1",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/transnetv2_head.py:30-45"
+ },
+ "7951": {
+ "file_id": 584,
+ "content": "This code defines a class with two methods, `loss` and `get_score`. The `loss` method calculates the loss between predictions and ground truth, while `get_score` method calculates an F1 score based on one-hot predictions and ground truth. The calculated losses are stored in a dictionary for further use.",
+ "type": "comment"
+ },
+ "7952": {
+ "file_id": 585,
+ "content": "/paddlevideo/modeling/heads/tsm_head.py",
+ "type": "filepath"
+ },
+ "7953": {
+ "file_id": 585,
+ "content": "TSMHead, a classification task-oriented class extending TSNHead, initializes weights and registers in the HEADS registry. It is part of PaddleVideo's temporal segment network head, with parameters for weights and data format, forward function with average pooling and optional dropout, and possible tensor reshaping.",
+ "type": "summary"
+ },
+ "7954": {
+ "file_id": 585,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn import Linear\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom .tsn_head import TSNHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass TSMHead(TSNHead):\n \"\"\" TSM Head\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsm_head.py:1-33"
+ },
+ "7955": {
+ "file_id": 585,
+ "content": "This code defines the TSMHead class, which extends the TSNHead class. It is used for classification tasks with a specific number of classes and input feature channels. The class is registered in the HEADS registry and follows a certain weight initialization method.",
+ "type": "comment"
+ },
+ "7956": {
+ "file_id": 585,
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n drop_ratio(float): drop ratio. Default: 0.5.\n std(float): Std(Scale) value in normal initilizar. Default: 0.001.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n drop_ratio=0.5,\n std=0.001,\n data_format=\"NCHW\",\n **kwargs):\n super().__init__(num_classes,\n in_channels,\n drop_ratio=drop_ratio,\n std=std,\n data_format=data_format,\n **kwargs)\n self.fc = Linear(self.in_channels,\n self.num_classes,\n weight_attr=ParamAttr(learning_rate=5.0,\n regularizer=L2Decay(1e-4)),\n bias_attr=ParamAttr(learning_rate=10.0,",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsm_head.py:34-57"
+ },
+ "7957": {
+ "file_id": 585,
+ "content": "The code defines a class with an __init__ method that initializes the TsmHead object. It takes arguments like num_classes, in_channels, drop_ratio, std, and data_format to set up the internal structure of the class. The Linear layer is also initialized with specific learning rates for weights and biases.",
+ "type": "comment"
+ },
+ "7958": {
+ "file_id": 585,
+ "content": " regularizer=L2Decay(0.0)))\n assert (data_format in [\n 'NCHW', 'NHWC'\n ]), f\"data_format must be 'NCHW' or 'NHWC', but got {data_format}\"\n self.data_format = data_format\n self.stdv = std\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc, 'Normal', 'fc_0.w_0', 'fc_0.b_0', std=self.stdv)\n def forward(self, x, num_seg):\n \"\"\"Define how the tsm-head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # x.shape = [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x) # [N * num_segs, in_channels, 1, 1]\n if self.dropout is not None:\n x = self.dropout(x) # [N * num_seg, in_channels, 1, 1]\n if self.data_format == 'NCHW':\n x = paddle.reshape(x, x.shape[:2])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsm_head.py:58-89"
+ },
+ "7959": {
+ "file_id": 585,
+ "content": "The code initializes a TSM head, sets the data format and standard deviation for weights, initializes FC layer parameters, defines the forward function to perform average pooling, dropout if applicable, and reshapes the tensor based on the data format.",
+ "type": "comment"
+ },
+ "7960": {
+ "file_id": 585,
+ "content": " else:\n x = paddle.reshape(x, x.shape[::3])\n score = self.fc(x) # [N * num_seg, num_class]\n score = paddle.reshape(\n score, [-1, num_seg, score.shape[1]]) # [N, num_seg, num_class]\n score = paddle.mean(score, axis=1) # [N, num_class]\n score = paddle.reshape(score,\n shape=[-1, self.num_classes]) # [N, num_class]\n # score = F.softmax(score) #NOTE remove\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsm_head.py:90-99"
+ },
+ "7961": {
+ "file_id": 585,
+ "content": "This code is part of a temporal segment network head in PaddleVideo. If the input is not 3-channel, it reshapes the feature map to have only the first third of channels. Then, it passes the reshaped feature through a fully connected layer and averages across segments to get a score for each class. The shape of the scores is then modified accordingly, and softmax could be applied (note: comment indicates that softmax might be removed).",
+ "type": "comment"
+ },
+ "7962": {
+ "file_id": 586,
+ "content": "/paddlevideo/modeling/heads/tsn_head.py",
+ "type": "filepath"
+ },
+ "7963": {
+ "file_id": 586,
+ "content": "TSNHead class is defined for image classification, inheriting from BaseHead with num_classes, in_channels, and loss_cfg parameters. It uses AdaptiveAvgPool2D, Linear, Dropout layers and weight_init function for initialization. The function defines a head for TSN model that performs average pooling, reshapes, takes mean, applies dropout if enabled, and passes through fully connected layer for classification scores.",
+ "type": "summary"
+ },
+ "7964": {
+ "file_id": 586,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle.nn import AdaptiveAvgPool2D, Linear, Dropout\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass TSNHead(BaseHead):\n \"\"\"TSN Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.\n loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsn_head.py:1-30"
+ },
+ "7965": {
+ "file_id": 586,
+ "content": "The code is defining a TSNHead class for image classification, which inherits from the BaseHead class. It has parameters for num_classes, in_channels, and loss_cfg, and uses AdaptiveAvgPool2D, Linear, Dropout layers. The weight_init function is also imported for weight initialization.",
+ "type": "comment"
+ },
+ "7966": {
+ "file_id": 586,
+ "content": " drop_ratio(float): drop ratio. Default: 0.4.\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n drop_ratio=0.4,\n std=0.01,\n data_format=\"NCHW\",\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.drop_ratio = drop_ratio\n self.std = std\n #NOTE: global pool performance\n self.avgpool2d = AdaptiveAvgPool2D((1, 1), data_format=data_format)\n if self.drop_ratio != 0:\n self.dropout = Dropout(p=self.drop_ratio)\n else:\n self.dropout = None\n self.fc = Linear(self.in_channels, self.num_classes)\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'Normal',",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsn_head.py:31-63"
+ },
+ "7967": {
+ "file_id": 586,
+ "content": "Initializes a TSN head with specified parameters, including num_classes, in_channels, loss_cfg, drop_ratio, std, and data_format. It creates an adaptive average pooling layer, a dropout layer if drop_ratio is non-zero, and a fully connected linear layer (fc). The fc layer weights are then initialized with normal distribution.",
+ "type": "comment"
+ },
+ "7968": {
+ "file_id": 586,
+ "content": " 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.,\n std=self.std)\n def forward(self, x, num_seg):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n #XXX: check dropout location!\n # [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x)\n # [N * num_segs, in_channels, 1, 1]\n x = paddle.reshape(x, [-1, num_seg, x.shape[1]])\n # [N, num_seg, in_channels]\n x = paddle.mean(x, axis=1)\n # [N, in_channels]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N, in_channels]\n score = self.fc(x)\n # [N, num_class]\n #x = F.softmax(x) #NOTE remove\n return score",
+ "type": "code",
+ "location": "/paddlevideo/modeling/heads/tsn_head.py:64-93"
+ },
+ "7969": {
+ "file_id": 586,
+ "content": "The function defines a head for the TSN model. It performs average pooling, reshapes the input, takes the mean along an axis, applies dropout if enabled, and passes the result through a fully connected layer to output scores for classification.",
+ "type": "comment"
+ },
+ "7970": {
+ "file_id": 587,
+ "content": "/paddlevideo/modeling/losses/__init__.py",
+ "type": "filepath"
+ },
+ "7971": {
+ "file_id": 587,
+ "content": "This code imports diverse loss functions from different modules, such as CrossEntropyLoss, BMNLoss, and TransNetV2Loss for PaddleVideo's video recognition, segmentation tasks, providing a comprehensive list of usable losses. The PaddleVideo model uses BaseWeightedLoss, ASRFLoss, DistillationCELoss, and DistillationDMLLoss for audio-visual speech recognition, distillation-based learning, and region-specific loss computation.",
+ "type": "summary"
+ },
+ "7972": {
+ "file_id": 587,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .base import BaseWeightedLoss\nfrom .bmn_loss import BMNLoss\nfrom .cross_entropy_loss import CrossEntropyLoss\nfrom .depth_loss import ADDSLoss\nfrom .transnetv2_loss import TransNetV2Loss\nfrom .actbert_loss import ActBertLoss\nfrom .asrf_loss import ASRFLoss\nfrom .distillation_loss import DistillationCELoss, DistillationDMLLoss\nfrom .yowo_loss import RegionLoss\n__all__ = [\n 'CrossEntropyLoss', 'BMNLoss', 'TransNetV2Loss', 'ActBertLoss', 'ADDSLoss',",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/__init__.py:1-26"
+ },
+ "7973": {
+ "file_id": 587,
+ "content": "This code imports different types of loss functions from various modules, such as CrossEntropyLoss, BMNLoss, and TransNetV2Loss. These losses are used in PaddleVideo for various applications like video recognition, segmentation, and more. The code provides a comprehensive list of loss functions that can be used depending on the specific task.",
+ "type": "comment"
+ },
+ "7974": {
+ "file_id": 587,
+ "content": " 'BaseWeightedLoss', 'ASRFLoss', 'DistillationCELoss', 'DistillationDMLLoss',\n 'RegionLoss'\n]",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/__init__.py:27-29"
+ },
+ "7975": {
+ "file_id": 587,
+ "content": "The code defines a list of loss functions used in the PaddleVideo model. These losses include BaseWeightedLoss, ASRFLoss, DistillationCELoss, and DistillationDMLLoss for various tasks like audio-visual speech recognition, distillation-based learning, and region-specific loss computation.",
+ "type": "comment"
+ },
+ "7976": {
+ "file_id": 588,
+ "content": "/paddlevideo/modeling/losses/actbert_loss.py",
+ "type": "filepath"
+ },
+ "7977": {
+ "file_id": 588,
+ "content": "This code defines ActBertLoss and actBertLoss classes as loss functions for ActBert model, using CrossEntropyLoss and nn.KLDivLoss. The total loss is calculated by summing masked text, masked image, masked action, and next sentence losses, based on predictions and labels from various sources.",
+ "type": "summary"
+ },
+ "7978": {
+ "file_id": 588,
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass ActBertLoss(BaseWeightedLoss):\n \"\"\"Loss for ActBert model\n \"\"\"\n def __init__(self, vocab_size=30522, a_target_size=700):\n super().__init__()\n self.vocab_size = vocab_size\n self.a_target_size = a_target_size\n self.loss_fct = nn.CrossEntropyLoss(ignore_index=-1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/actbert_loss.py:1-32"
+ },
+ "7979": {
+ "file_id": 588,
+ "content": "This code defines the ActBertLoss class, which is a loss function for the ActBert model. It uses the CrossEntropyLoss from PaddlePaddle's nn library and takes two arguments: vocab_size and a_target_size. The class inherits from BaseWeightedLoss.",
+ "type": "comment"
+ },
+ "7980": {
+ "file_id": 588,
+ "content": " self.vis_criterion = nn.KLDivLoss(reduction=\"none\")\n def forward(self, prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score, \\\n text_labels, image_label, image_target, action_label, next_sentence_label):\n \"\"\"\n Args:\n text_label: text label(with mask). Shape: [batch_size, seqence_length]\n image_label: image label(with mask). Shape: [batch_size, region_length]\n image_target: label of image feature distribution,\n Shape: [batch_size, region_length-1, num_image_class](minus 1 for xxx).\n action label: action label(with mask), Shape: [batch_size, action_length]\n next_sentence_label: is next sentence or not. Shape: [batch_size]\n \"\"\"\n prediction_scores_v = prediction_scores_v[:,\n 1:] #8,37,1601 --> 8,36,1601\n img_loss = self.vis_criterion(\n F.log_softmax(prediction_scores_v, axis=2),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/actbert_loss.py:33-50"
+ },
+ "7981": {
+ "file_id": 588,
+ "content": "This code defines a class for an actBert loss function. It uses the nn.KLDivLoss function as a criterion and takes in prediction scores, sequence labels, image labels, image targets, action labels, and next sentence labels as input to compute the visual loss (img_loss). The prediction_scores_v variable is modified by removing the first element from each sequence, likely for consistency purposes.",
+ "type": "comment"
+ },
+ "7982": {
+ "file_id": 588,
+ "content": " image_target #8,36,1601\n )\n masked_img_loss = paddle.sum(\n img_loss * (image_label == 1).unsqueeze(2).astype('float32')) / max(\n paddle.sum((image_label == 1).astype('float32')), 1e-6)\n masked_text_loss = self.loss_fct(\n prediction_scores_t.reshape([-1, self.vocab_size]), #8,36,30522\n text_labels.reshape([-1]), #8,36 # label -1 will be ignored\n )\n masked_action_loss = self.loss_fct(\n prediction_scores_a.reshape([-1, self.a_target_size]), #8,5,700\n action_label.reshape([-1]), #8,5\n )\n next_sentence_loss = self.loss_fct(\n seq_relationship_score.reshape([-1, 2]),\n next_sentence_label.reshape([-1]) #8,2\n )\n total_loss = masked_text_loss.unsqueeze(0) + masked_img_loss.unsqueeze(\n 0) + masked_action_loss.unsqueeze(0) + next_sentence_loss.unsqueeze(\n 0)\n return total_loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/actbert_loss.py:51-75"
+ },
+ "7983": {
+ "file_id": 588,
+ "content": "This code calculates a total loss by summing the masked text loss, masked image loss, masked action loss, and next sentence loss. The losses are calculated based on predictions and labels from various sources. The `loss_fct` function is used to compute these losses, and they are reshaped before being added together for the final total loss.",
+ "type": "comment"
+ },
+ "7984": {
+ "file_id": 589,
+ "content": "/paddlevideo/modeling/losses/asrf_loss.py",
+ "type": "filepath"
+ },
+ "7985": {
+ "file_id": 589,
+ "content": "This code defines custom loss functions for video modeling, including TMSE and GSTMSE, with the ActionSegmentationLoss class applying various criteria like regression, classification, and temporal segmentation losses.",
+ "type": "summary"
+ },
+ "7986": {
+ "file_id": 589,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# https://github.com/yiskw713/asrf/libs/loss_fn/__init__.py\nimport numpy as np\nimport pandas as pd\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport sys\nimport os\nfrom ..registry import LOSSES\nclass TMSE(nn.Layer):\n \"\"\"\n Temporal MSE Loss Function\n Proposed in Y. A. Farha et al. MS-TCN: Multi-Stage Temporal Convolutional Network for ActionSegmentation in CVPR2019\n arXiv: https://arxiv.org/pdf/1903.01945.pdf",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:1-32"
+ },
+ "7987": {
+ "file_id": 589,
+ "content": "The code defines a class TMSE, which is a temporal MSE loss function. It's inspired by the MS-TCN method proposed in CVPR2019 for action segmentation tasks.",
+ "type": "comment"
+ },
+ "7988": {
+ "file_id": 589,
+ "content": " \"\"\"\n def __init__(self, threshold=4, ignore_index=255):\n super().__init__()\n self.threshold = threshold\n self.ignore_index = ignore_index\n self.mse = nn.MSELoss(reduction=\"none\")\n def forward(self, preds, gts):\n total_loss = 0.0\n batch_size = preds.shape[0]\n for pred, gt in zip(preds, gts):\n pred = paddle.gather(pred,\n paddle.nonzero(gt != self.ignore_index)[:, 0])\n loss = self.mse(F.log_softmax(pred[:, 1:], axis=1),\n F.log_softmax(pred[:, :-1], axis=1))\n loss = paddle.clip(loss, min=0, max=self.threshold**2)\n total_loss += paddle.mean(loss)\n return total_loss / batch_size\nclass GaussianSimilarityTMSE(nn.Layer):\n \"\"\"\n Temporal MSE Loss Function with Gaussian Similarity Weighting\n \"\"\"\n def __init__(self, threshold=4, sigma=1.0, ignore_index=255):\n super().__init__()\n self.threshold = threshold\n self.ignore_index = ignore_index",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:33-66"
+ },
+ "7989": {
+ "file_id": 589,
+ "content": "The code defines two classes: ASRF_Loss and GaussianSimilarityTMSE. The first class represents the Average Symmetric Ranking Forest Loss, while the second class is a Temporal MSE Loss Function with Gaussian Similarity Weighting. Both classes inherit from nn.Layer and have an __init__ method for initialization, as well as a forward method for calculating losses. The ASRF_Loss class uses an MSELoss function to calculate loss between predicted and ground truth frames, while the GaussianSimilarityTMSE class calculates temporal MSE with Gaussian similarity weighting.",
+ "type": "comment"
+ },
+ "7990": {
+ "file_id": 589,
+ "content": " self.mse = nn.MSELoss(reduction=\"none\")\n self.sigma = sigma\n def forward(self, preds, gts, sim_index):\n \"\"\"\n Args:\n preds: the output of model before softmax. (N, C, T)\n gts: Ground Truth. (N, T)\n sim_index: similarity index. (N, C, T)\n Return:\n the value of Temporal MSE weighted by Gaussian Similarity.\n \"\"\"\n total_loss = 0.0\n batch_size = preds.shape[0]\n for pred, gt, sim in zip(preds, gts, sim_index):\n pred = paddle.gather(pred,\n paddle.nonzero(gt != self.ignore_index)[:, 0],\n axis=1)\n sim = paddle.gather(sim,\n paddle.nonzero(gt != self.ignore_index)[:, 0],\n axis=1)\n # calculate gaussian similarity\n diff = sim[:, 1:] - sim[:, :-1]\n similarity = paddle.exp(\n (-1 * paddle.norm(diff, axis=0)) / (2 * self.sigma**2))",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:67-92"
+ },
+ "7991": {
+ "file_id": 589,
+ "content": "This code calculates a temporal MSE loss weighted by Gaussian similarity. It uses Paddle's nn.MSELoss function, with reduction set to \"none\". The forward method takes in the model's predictions (before softmax), ground truth and similarity index as inputs. It loops through each input, performs non-zero checks for gt != ignore_index, then gathers the relevant rows from the predicted values. It calculates gaussian similarity using the gathered data and the given sigma value. The calculated loss is returned.",
+ "type": "comment"
+ },
+ "7992": {
+ "file_id": 589,
+ "content": " # calculate temporal mse\n loss = self.mse(F.log_softmax(pred[:, 1:], axis=1),\n F.log_softmax(pred[:, :-1], axis=1))\n loss = paddle.clip(loss, min=0, max=self.threshold**2)\n # gaussian similarity weighting\n loss = similarity * loss\n total_loss += paddle.mean(loss)\n return total_loss / batch_size\nclass FocalLoss(nn.Layer):\n def __init__(self,\n weight=None,\n size_average=True,\n batch_average=True,\n ignore_index=255,\n gamma=2.0,\n alpha=0.25):\n super().__init__()\n self.gamma = gamma\n self.alpha = alpha\n self.batch_average = batch_average\n self.criterion = nn.CrossEntropyLoss(weight=weight,\n ignore_index=ignore_index,\n size_average=size_average)\n def forward(self, logit, target):\n n, _, _ = logit.size()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:94-126"
+ },
+ "7993": {
+ "file_id": 589,
+ "content": "This code defines a class \"ASRF_Loss\" for calculating ASRF loss using temporal MSE and Gaussian similarity weighting. It also defines a class \"FocalLoss\" for focal loss calculation using CrossEntropyLoss with custom gamma and alpha parameters.",
+ "type": "comment"
+ },
+ "7994": {
+ "file_id": 589,
+ "content": " logpt = -self.criterion(logit, target.long())\n pt = paddle.exp(logpt)\n if self.alpha is not None:\n logpt *= self.alpha\n loss = -((1 - pt)**self.gamma) * logpt\n if self.batch_average:\n loss /= n\n return loss\nclass ActionSegmentationLoss(nn.Layer):\n \"\"\"\n Loss Function for Action Segmentation\n You can choose the below loss functions and combine them.\n - Cross Entropy Loss (CE)\n - Focal Loss\n - Temporal MSE (TMSE)\n - Gaussian Similarity TMSE (GSTMSE)\n \"\"\"\n def __init__(self,\n num_classes,\n file_path,\n label_path,\n ce=True,\n focal=True,\n tmse=False,\n gstmse=False,\n weight=None,\n threshold=4.,\n ignore_index=255,\n ce_weight=1.0,\n focal_weight=1.0,\n tmse_weight=0.15,\n gstmse_weight=0.15):\n super().__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:128-167"
+ },
+ "7995": {
+ "file_id": 589,
+ "content": "This code defines an ActionSegmentationLoss class, which is a loss function for action segmentation tasks. It allows the user to choose from various loss functions including Cross Entropy Loss (CE), Focal Loss, Temporal MSE (TMSE), and Gaussian Similarity TMSE (GSTMSE). The user can specify parameters such as num_classes, file_path, label_path, ce, focal, tmse, gstmse, weight, threshold, ignore_index, ce_weight, focal_weight, and tmse_weight. The class initializes the chosen loss functions and calculates the overall loss based on user inputs.",
+ "type": "comment"
+ },
+ "7996": {
+ "file_id": 589,
+ "content": " self.criterions = []\n self.weights = []\n self.num_classes = num_classes\n self.file_path = file_path\n self.label_path = label_path\n if weight:\n class_weight = self.get_class_weight()\n else:\n class_weight = None\n if ce:\n self.criterions.append(\n nn.CrossEntropyLoss(weight=class_weight,\n ignore_index=ignore_index))\n self.weights.append(ce_weight)\n if focal:\n self.criterions.append(FocalLoss(ignore_index=ignore_index))\n self.weights.append(focal_weight)\n if tmse:\n self.criterions.append(\n TMSE(threshold=threshold, ignore_index=ignore_index))\n self.weights.append(tmse_weight)\n if gstmse:\n self.criterions.append(\n GaussianSimilarityTMSE(threshold=threshold,\n ignore_index=ignore_index))\n self.weights.append(gstmse_weight)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:168-198"
+ },
+ "7997": {
+ "file_id": 589,
+ "content": "The code initializes criterions and weights for different loss functions based on the provided parameters. It adds CrossEntropyLoss, FocalLoss, TMSE, and GaussianSimilarityTMSE to self.criterions list, and their corresponding weights to self.weights list. The weight parameter determines whether class weights are used in CrossEntropyLoss. Ignore_index is added for all loss functions.",
+ "type": "comment"
+ },
+ "7998": {
+ "file_id": 589,
+ "content": " if len(self.criterions) == 0:\n print(\"You have to choose at least one loss function.\")\n sys.exit(1)\n def get_class_weight(self):\n \"\"\"\n Class weight for CrossEntropy\n Class weight is calculated in the way described in:\n D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” in ICCV,\n openaccess: https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Eigen_Predicting_Depth_Surface_ICCV_2015_paper.pdf\n \"\"\"\n # load file list\n file_ptr = open(self.file_path, 'r')\n info = file_ptr.read().split('\\n')[:-1]\n file_ptr.close()\n nums = [0 for i in range(self.num_classes)]\n for i in range(len(info)):\n video_name = info[i]\n file_name = video_name.split('.')[0] + \".npy\"\n label_file_path = os.path.join(self.label_path, file_name)\n label = np.load(label_file_path).astype(np.int64)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:200-221"
+ },
+ "7999": {
+ "file_id": 589,
+ "content": "The code snippet loads file information from a given path and calculates class weights for CrossEntropy loss function, based on the method described in the Eigen and Fergus paper. It reads file names and their corresponding labels, and stores them as lists for later use.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/8.json b/docs/data/8.json
new file mode 100644
index 000000000..8a0ad0920
--- /dev/null
+++ b/docs/data/8.json
@@ -0,0 +1,539 @@
+{
+ "800": {
+ "file_id": 69,
+ "content": " self.fmt = fmt\n self.need_avg = need_avg\n self.reset()\n def reset(self):\n \"\"\" reset \"\"\"\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def update(self, val, n=1):\n \"\"\" update \"\"\"\n if isinstance(val, paddle.Tensor):\n val = float(val)\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n @property\n def total(self):\n return '{self.name}_sum: {self.sum:{self.fmt}}'.format(self=self)\n @property\n def total_minute(self):\n return '{self.name}_sum: {s:{self.fmt}} min'.format(s=self.sum / 60,\n self=self)\n @property\n def mean(self):\n return '{self.name}_avg: {self.avg:{self.fmt}}'.format(\n self=self) if self.need_avg else ''\n @property\n def value(self):\n return '{self.name}: {self.val:{self.fmt}}'.format(self=self)\ndef log_batch(metric_list,\n batch_id,",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py:73-113"
+ },
+ "801": {
+ "file_id": 69,
+ "content": "This code defines a class for tracking metrics such as sum, count, average, and total values. The `update` method allows updating the metric with a new value, while the `total`, `total_minute`, `mean`, and `value` properties retrieve the current metric value in different formats. The `log_batch` function logs batch metrics for a list of metrics at a specified batch ID.",
+ "type": "comment"
+ },
+ "802": {
+ "file_id": 69,
+ "content": " epoch_id,\n total_epoch,\n mode,\n ips,\n tot_step=None,\n max_iters=None):\n batch_cost = str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = str(metric_list['reader_time'].value) + ' sec,'\n metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].value)\n metric_str = ' '.join([str(v) for v in metric_values])\n if max_iters:\n epoch_str = \"iter:[{:>3d}/{:<3d}]\".format(tot_step, max_iters)\n else:\n epoch_str = \"epoch:[{:>3d}/{:<3d}]\".format(epoch_id, total_epoch)\n step_str = \"{:s} step:{:<4d}\".format(mode, batch_id)\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {}\".format(\n coloring(epoch_str, \"HEADER\") if batch_id == 0 else epoch_str,\n coloring(step_str, \"PURPLE\"), coloring(metric_str, 'OKGREEN'),\n coloring(batch_cost, \"OKGREEN\"), coloring(reader_cost, 'OKGREEN'),\n ips))\ndef log_epoch(metric_list, epoch, mode, ips):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py:114-141"
+ },
+ "803": {
+ "file_id": 69,
+ "content": "This function logs epoch metrics and step information for a video processing task. It formats the log string with different colors for each section: epoch or iteration, step number, metric values, batch time, reader time, and ips (images per second). The logger outputs this formatted string to provide an informative summary of the task's progress.",
+ "type": "comment"
+ },
+ "804": {
+ "file_id": 69,
+ "content": " batch_cost = 'avg_' + str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = 'avg_' + str(metric_list['reader_time'].value) + ' sec,'\n batch_sum = str(metric_list['batch_time'].total) + ' sec,'\n metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].mean)\n metric_str = ' '.join([str(v) for v in metric_values])\n end_epoch_str = \"END epoch:{:<3d}\".format(epoch)\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {:s} {}\".format(\n coloring(end_epoch_str, \"RED\"), coloring(mode, \"PURPLE\"),\n coloring(metric_str, \"OKGREEN\"), coloring(batch_cost, \"OKGREEN\"),\n coloring(reader_cost, \"OKGREEN\"), coloring(batch_sum, \"OKGREEN\"), ips))",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/record.py:142-157"
+ },
+ "805": {
+ "file_id": 69,
+ "content": "This code is formatting and logging information at the end of an epoch. It calculates various metric values, constructs a formatted string with different colors, and then logs this information using logger.info(). The metrics include batch time, reader time, total cost, mode, and inference per second. The strings are color-coded for visual clarity.",
+ "type": "comment"
+ },
+ "806": {
+ "file_id": 70,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py",
+ "type": "filepath"
+ },
+ "807": {
+ "file_id": 70,
+ "content": "The code defines a Registry class for mapping names to objects and provides methods for registering, getting, and unregistering objects. It utilizes the @BACKBONES.register() decorator or BACKBONES.register(ResNet) function for registration, and also verifies if an object with a given name exists in the registry using the `get` method.",
+ "type": "summary"
+ },
+ "808": {
+ "file_id": 70,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nclass Registry(object):\n \"\"\"\n The registry that provides name -> object mapping, to support third-party users' custom modules.\n To register an object:\n .. code-block:: python\n BACKBONES = Registry('backbone')\n @BACKBONES.register()\n class ResNet:\n pass\n Or:\n .. code-block:: python\n BACKBONES = Registry('backbone')\n class ResNet:\n pass\n BACKBONES.register(ResNet)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py:1-34"
+ },
+ "809": {
+ "file_id": 70,
+ "content": "This code defines a Registry class that provides name to object mapping, allowing third-party users to register their custom modules. Users can register their objects by using the @BACKBONES.register() decorator or by calling BACKBONES.register(ResNet).",
+ "type": "comment"
+ },
+ "810": {
+ "file_id": 70,
+ "content": " Usage: To build a module.\n .. code-block:: python\n backbone_name = \"ResNet\"\n b = BACKBONES.get(backbone_name)()\n \"\"\"\n def __init__(self, name):\n \"\"\"\n Args:\n name (str): the name of this registry\n \"\"\"\n self._name = name\n self._obj_map = {}\n def __contains__(self, key):\n return self._obj_map.get(key) is not None\n def _do_register(self, name, obj):\n assert (\n name not in self._obj_map\n ), \"An object named '{}' was already registered in '{}' registry!\".format(\n name, self._name)\n self._obj_map[name] = obj\n def register(self, obj=None, name=None):\n \"\"\"\n Register the given object under the the name `obj.__name__`.\n Can be used as either a decorator or not. See docstring of this class for usage.\n \"\"\"\n if obj is None:\n # used as a decorator\n def deco(func_or_class, name=name):\n if name is None:\n name = func_or_class.__name__",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py:36-70"
+ },
+ "811": {
+ "file_id": 70,
+ "content": "This code is a registry class for storing and managing objects. It allows registering objects under their names or using decorators, and provides functions to check if an object with a given name exists in the registry.",
+ "type": "comment"
+ },
+ "812": {
+ "file_id": 70,
+ "content": " self._do_register(name, func_or_class)\n return func_or_class\n return deco\n # used as a function call\n if name is None:\n name = obj.__name__\n self._do_register(name, obj)\n def get(self, name):\n \"\"\"Get the registry record.\n Args:\n name (str): The class name.\n Returns:\n ret: The class.\n \"\"\"\n ret = self._obj_map.get(name)\n if ret is None:\n raise KeyError(\n \"No object named '{}' found in '{}' registry!\".format(\n name, self._name))\n return ret",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py:71-96"
+ },
+ "813": {
+ "file_id": 70,
+ "content": "The code defines a class with methods for registering, getting and unregistering objects in a registry. The `_do_register` method is used to store the object's name and function or class into a dictionary. If no name is provided when calling the function, it defaults to the object's name. The `get` method retrieves an object from the registry using its name. If the object is not found, it raises a KeyError with an error message.",
+ "type": "comment"
+ },
+ "814": {
+ "file_id": 71,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py",
+ "type": "filepath"
+ },
+ "815": {
+ "file_id": 71,
+ "content": "This code adapts ViT model parameters, modifies pos_embed and time_embed for compatibility, and includes functions for loading/saving PaddlePaddle models with parallel/non-parallel handling and progress bar.",
+ "type": "summary"
+ },
+ "816": {
+ "file_id": 71,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport os.path as osp\nimport time\nfrom tqdm import tqdm\nimport paddle\nimport paddle.nn.functional as F\nfrom EIVideo.paddlevideo.utils import get_logger\nfrom EIVideo.paddlevideo.utils import main_only\ndef pretrain_vit_param_trans(model, state_dicts, num_patches, seg_num,\n attention_type):\n \"\"\"\n Convert ViT's pre-trained model parameters to a parameter dictionary that matches the existing model",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:1-28"
+ },
+ "817": {
+ "file_id": 71,
+ "content": "The code is a function that converts pre-trained ViT model parameters to match the existing model. It takes in the model, state_dicts, num_patches, seg_num, and attention_type as arguments. The function adapts the ViT's pre-trained model parameters for better compatibility with the existing model structure.",
+ "type": "comment"
+ },
+ "818": {
+ "file_id": 71,
+ "content": " \"\"\"\n if 'head' + '.weight' in state_dicts:\n del state_dicts['head' + '.weight']\n if 'head' + '.bias' in state_dicts:\n del state_dicts['head' + '.bias']\n total_len = len(model.state_dict())\n if num_patches + 1 != state_dicts['pos_embed'].shape[1]:\n pos_embed = state_dicts['pos_embed']\n cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)\n other_pos_embed = pos_embed[0,\n 1:, :].unsqueeze(0).unsqueeze(1).transpose(\n (0, 1, 3, 2))\n new_pos_embed = F.interpolate(other_pos_embed,\n size=(other_pos_embed.shape[-2],\n num_patches),\n mode='nearest')\n new_pos_embed = new_pos_embed.squeeze(0).transpose((0, 2, 1))\n new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed), axis=1)\n state_dicts['pos_embed'] = new_pos_embed\n time.sleep(0.01)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:29-49"
+ },
+ "819": {
+ "file_id": 71,
+ "content": "This code modifies the 'pos_embed' tensor in state_dicts if its shape doesn't match the expected shape. It interpolates the other_pos_embed to fit the desired size, then concatenates the cls_pos_embed and new_pos_embed and updates the state_dicts['pos_embed']. This allows the code to maintain consistency in the 'pos_embed' tensor.",
+ "type": "comment"
+ },
+ "820": {
+ "file_id": 71,
+ "content": " if 'time_embed' in state_dicts and seg_num != state_dicts[\n 'time_embed'].shape[1]:\n time_embed = state_dicts['time_embed'].transpose((0, 2, 1)).unsqueeze(0)\n new_time_embed = F.interpolate(time_embed,\n size=(time_embed.shape[-2], seg_num),\n mode='nearest')\n state_dicts['time_embed'] = new_time_embed.squeeze(0).transpose(\n (0, 2, 1))\n time.sleep(0.01)\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n if attention_type == 'divided_space_time':\n new_state_dicts = state_dicts.copy()\n for key in tqdm(state_dicts):\n if 'blocks' in key and 'attn' in key:\n desc.set_description(\"Loading %s\" % key)\n new_key = key.replace('attn', 'temporal_attn')\n if not new_key in state_dicts:\n new_state_dicts[new_key] = state_dicts[key]",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:51-71"
+ },
+ "821": {
+ "file_id": 71,
+ "content": "The code checks if a specific key 'time_embed' exists in the state_dicts and adjusts its shape accordingly. It then interpolates the time_embed using nearest mode and transposes it to fit into the new shape. After that, it creates a progress bar \"Loading weights\" using tqdm for the total length of data and sets the description as the current key being processed. If 'attn' is present in the key and 'blocks', it replaces 'attn' with 'temporal_attn' if not already present in state_dicts and adds it to new_state_dicts.",
+ "type": "comment"
+ },
+ "822": {
+ "file_id": 71,
+ "content": " else:\n new_state_dicts[new_key] = state_dicts[new_key]\n if 'blocks' in key and 'norm1' in key:\n desc.set_description(\"Loading %s\" % key)\n new_key = key.replace('norm1', 'temporal_norm1')\n if not new_key in state_dicts:\n new_state_dicts[new_key] = state_dicts[key]\n else:\n new_state_dicts[new_key] = state_dicts[new_key]\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n return new_state_dicts\ndef pretrain_resnet18_param_trans(model, loaded_dict):\n encoder_dict = model.encoder.state_dict()\n pose_encoder_dict = model.pose_encoder.state_dict()\n names = ['encoder.', 'encoder_day.', 'encoder_night.']\n for name in names:\n for key, value in loaded_dict.items():\n key = str(name + key)\n if key in encoder_dict:",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:72-96"
+ },
+ "823": {
+ "file_id": 71,
+ "content": "This code is checking if a certain key exists in the state_dict and creating a new key with 'temporal' added to it. It is also updating the description for loading weights and returning the updated state_dicts. The function pretrain_resnet18_param_trans compares loaded dict with encoder and pose_encoder dictionaries in the model, possibly for parameter transfer learning.",
+ "type": "comment"
+ },
+ "824": {
+ "file_id": 71,
+ "content": " encoder_dict[key] = value\n num_input_images = 2\n loaded_dict['conv1.weight'] = paddle.concat(\n [loaded_dict['conv1.weight']] * num_input_images, 1) / num_input_images\n for name, value in loaded_dict.items():\n name = str('encoder.' + name)\n if name in pose_encoder_dict:\n pose_encoder_dict[name] = value\n return encoder_dict, pose_encoder_dict\n#XXX(shipping): maybe need load N times because of different cards have different params.\n@main_only\ndef load_ckpt(model, weight_path, **kargs):\n \"\"\"\n 1. Load pre-trained model parameters\n 2. Extract and convert from the pre-trained model to the parameters\n required by the existing model\n 3. Load the converted parameters of the existing model\n \"\"\"\n #model.set_state_dict(state_dict)\n if not osp.isfile(weight_path):\n raise IOError(f'{weight_path} is not a checkpoint file')\n #state_dicts = load(weight_path)\n logger = get_logger(\"paddlevideo\")\n state_dicts = paddle.load(weight_path)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:97-127"
+ },
+ "825": {
+ "file_id": 71,
+ "content": "The code is defining a function to load pre-trained model parameters, which requires converting the parameters of the pre-trained model into the parameters needed for the current model. The function first checks if the weight path exists and raises an IOError if it does not. Then, it loads the state_dicts from the given weight_path using paddle.load().",
+ "type": "comment"
+ },
+ "826": {
+ "file_id": 71,
+ "content": " if 'ResnetEncoder' in str(model):\n encoder_dict, pose_encoder_dict = pretrain_resnet18_param_trans(\n model, state_dicts)\n tmp = model.state_dict()\n tmp.update(\n {'backbone.encoder.' + k: v\n for (k, v) in encoder_dict.items()})\n tmp.update({\n 'backbone.pose_encoder.' + k: v\n for (k, v) in pose_encoder_dict.items()\n })\n elif \"VisionTransformer\" in str(model): # For TimeSformer case\n tmp = pretrain_vit_param_trans(model, state_dicts, kargs['num_patches'],\n kargs['seg_num'],\n kargs['attention_type'])\n else:\n tmp = {}\n total_len = len(model.state_dict())\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n for item in tqdm(model.state_dict(), total=total_len, position=0):\n name = item\n desc.set_description('Loading %s' % name)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:128-152"
+ },
+ "827": {
+ "file_id": 71,
+ "content": "This code is used to load weights for a model, specifically handling Resnet Encoder and Vision Transformer cases. For Resnet Encoder, it updates the state dictionary with separate dictionaries for encoder and pose_encoder. For Vision Transformer (TimeSformer), it uses pretrain_vit_param_trans function. If the model is neither of these types, it initializes an empty dictionary. The code also includes a tqdm progress bar to show the loading progress.",
+ "type": "comment"
+ },
+ "828": {
+ "file_id": 71,
+ "content": " if name not in state_dicts: # Convert from non-parallel model\n if str('backbone.' + name) in state_dicts:\n tmp[name] = state_dicts['backbone.' + name]\n else: # Convert from parallel model\n tmp[name] = state_dicts[name]\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n model.set_state_dict(tmp)\ndef mkdir(dir):\n if not os.path.exists(dir):\n # avoid error when train with multiple gpus\n try:\n os.makedirs(dir)\n except:\n pass\n@main_only\ndef save(obj, path):\n paddle.save(obj, path)\ndef load(file_name):\n if not osp.isfile(file_name):\n raise IOError(f'{file_name} not exist')\n return paddle.load(file_name)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py:153-182"
+ },
+ "829": {
+ "file_id": 71,
+ "content": "This code snippet defines functions for loading and saving PaddlePaddle models. It checks if the model is parallel or non-parallel, converts the state dictionaries accordingly, and updates the model's state dictionary. The `mkdir` function creates a directory if it doesn't exist already, and there are separate save and load functions defined for ease of use.",
+ "type": "comment"
+ },
+ "830": {
+ "file_id": 72,
+ "content": "/applications/EIVideo/EIVideo/paddlevideo/version.py",
+ "type": "filepath"
+ },
+ "831": {
+ "file_id": 72,
+ "content": "This code contains the version information for PaddleVideo, licensed under the Apache License 2.0, and defines the current version as \"0.0.1\".",
+ "type": "summary"
+ },
+ "832": {
+ "file_id": 72,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n__all__ = [\"paddlevideo_version\"]\npaddlevideo_version = \"0.0.1\"",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/version.py:1-16"
+ },
+ "833": {
+ "file_id": 72,
+ "content": "This code contains the version information for PaddleVideo, licensed under the Apache License 2.0, and defines the current version as \"0.0.1\".",
+ "type": "comment"
+ },
+ "834": {
+ "file_id": 73,
+ "content": "/applications/EIVideo/EIVideo/setup.py",
+ "type": "filepath"
+ },
+ "835": {
+ "file_id": 73,
+ "content": "These comments provide author, date, and copyright information, indicating the source should be cited if the code is reprinted.",
+ "type": "summary"
+ },
+ "836": {
+ "file_id": 73,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/11\n# Copyright belongs to the author.\n# Please indicate the source for reprinting.",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/setup.py:1-4"
+ },
+ "837": {
+ "file_id": 73,
+ "content": "These comments provide author, date, and copyright information, indicating the source should be cited if the code is reprinted.",
+ "type": "comment"
+ },
+ "838": {
+ "file_id": 74,
+ "content": "/applications/EIVideo/EIVideo/version.py",
+ "type": "filepath"
+ },
+ "839": {
+ "file_id": 74,
+ "content": "This code snippet is the version information for the EIVideo application, created by Acer Zhang on January 11th, 2022. It has a version number \"0.1a\" and the author requests proper attribution if reusing the code.",
+ "type": "summary"
+ },
+ "840": {
+ "file_id": 74,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/11 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\n__version__ = \"0.1a\"",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/version.py:1-6"
+ },
+ "841": {
+ "file_id": 74,
+ "content": "This code snippet is the version information for the EIVideo application, created by Acer Zhang on January 11th, 2022. It has a version number \"0.1a\" and the author requests proper attribution if reusing the code.",
+ "type": "comment"
+ },
+ "842": {
+ "file_id": 75,
+ "content": "/applications/EIVideo/QEIVideo/__init__.py",
+ "type": "filepath"
+ },
+ "843": {
+ "file_id": 75,
+ "content": "Code sets the QEI_VIDEO_ROOT variable to the absolute path of the directory containing the current file. It also imports the version module from QEIVideo and assigns its __version__ attribute to a variable. This may be used for identifying the version of the QEIVideo application.",
+ "type": "summary"
+ },
+ "844": {
+ "file_id": 75,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport os\nQEI_VIDEO_ROOT = os.path.abspath(os.path.dirname(__file__))\nimport os\nfrom QEIVideo.version import __version__\nQEI_VIDEO_ROOT = os.path.abspath(os.path.dirname(__file__))",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/__init__.py:1-13"
+ },
+ "845": {
+ "file_id": 75,
+ "content": "Code sets the QEI_VIDEO_ROOT variable to the absolute path of the directory containing the current file. It also imports the version module from QEIVideo and assigns its __version__ attribute to a variable. This may be used for identifying the version of the QEIVideo application.",
+ "type": "comment"
+ },
+ "846": {
+ "file_id": 76,
+ "content": "/applications/EIVideo/QEIVideo/build_gui.py",
+ "type": "filepath"
+ },
+ "847": {
+ "file_id": 76,
+ "content": "The script uses PyQt5 to create a video processing GUI with functions for initializing variables, opening file dialogs, handling combo box indexing, and pen color changes. It also includes a `open_frame` function that updates the progress slider and stops the video at the last frame.",
+ "type": "summary"
+ },
+ "848": {
+ "file_id": 76,
+ "content": "# Author: Acer Zhang\n# Datetime:2022/1/11 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport json\nimport os\nimport numpy as np\nfrom PIL import Image\nfrom PyQt5 import QtCore, QtWidgets\nfrom PyQt5.QtGui import *\nfrom PyQt5.QtWidgets import *\nfrom PyQt5.QtCore import *\nimport cv2\nfrom EIVideo.api import json2frame, png2json, load_video\nfrom EIVideo.main import main\n# ToDo To AP-kai: 这是定义前端临时保存用于推理的json的地点之类的，因为是固定的，所以声明为全局常量是最好的\nfrom EIVideo import TEMP_JSON_SAVE_PATH, TEMP_IMG_SAVE_PATH, TEMP_JSON_FINAL_PATH\nfrom QEIVideo.gui.ui_main_window import Ui_MainWindow\nclass BuildGUI(QMainWindow, Ui_MainWindow):\n def __init__(self):\n super(BuildGUI, self).__init__()\n # ToDo To AP-kai: 这里定义当前选择的视频路径的占位符，相当于全局变量\n self.select_video_path = None\n # ToDo To AP-kai: 未来为用户提供个保存路径的入口哈，这里先随意定义了个路径\n self.save_path = \"./result\"\n os.makedirs(self.save_path, exist_ok=True)\n self.setupUi(self)\n def infer(self):\n self.label.setText(\"Start infer\")",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/build_gui.py:1-36"
+ },
+ "849": {
+ "file_id": 76,
+ "content": "This code is the initial part of a Python script for building a GUI (Graphical User Interface) application using PyQt5 library. It defines a class called BuildGUI that inherits from QMainWindow and Ui_MainWindow, which likely contains the layout and design elements of the GUI. The __init__ method sets up some initial variables such as the selected video path and save path for results. The infer method is meant to start an inference process.",
+ "type": "comment"
+ },
+ "850": {
+ "file_id": 76,
+ "content": " self.progressBar.setProperty(\"value\", 0)\n image = self.paintBoard.get_content_as_q_image()\n image.save(TEMP_IMG_SAVE_PATH)\n print(self.slider_frame_num)\n self.progressBar.setProperty(\"value\", 25)\n # ToDo To AP-kai:相同的文件路径，直接定义一个常量就好\n png2json(TEMP_IMG_SAVE_PATH, self.slider_frame_num, TEMP_JSON_SAVE_PATH)\n self.progressBar.setProperty(\"value\", 50)\n # ToDo To AP-kai:打印的信息，需要注意首字母大写\n # ToDo To AP-kai: 此处传入保存路径以及当前选择的视频路径，最后会在manet_stage1.py里通过cfg来传入\n out = main(video_path=self.select_video_path, save_path=self.save_path)\n print('Infer ok')\n self.progressBar.setProperty(\"value\", 75)\n self.all_frames = json2frame(TEMP_JSON_FINAL_PATH)\n print(\"Success get submit_masks\")\n self.open_frame()\n self.progressBar.setProperty(\"value\", 100)\n self.label.setText(\"Infer succeed\")\n def btn_func(self, btn):\n if btn == self.playbtn:\n self.label.setText(\"Play video\")\n if self.progress_slider.value() == self.cap.get(7) - 1:",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/build_gui.py:37-59"
+ },
+ "851": {
+ "file_id": 76,
+ "content": "The code snippet sets the progress bar value, saves an image, prints frame numbers, calls a main function to perform inference, loads JSON frames, opens a frame, and updates a label when play button is clicked.",
+ "type": "comment"
+ },
+ "852": {
+ "file_id": 76,
+ "content": " self.slider_frame_num = 0\n self.progress_slider.setValue(self.slider_frame_num)\n self.time_label.setText('{}/{}'.format(self.slider_frame_num, self.cap.get(7)))\n self.timer_camera = QTimer() # 定义定时器\n self.timer_camera.start(1000 / self.cap.get(cv2.CAP_PROP_FPS))\n self.slider_frame_num = self.progress_slider.value()\n self.timer_camera.timeout.connect(self.open_frame)\n elif btn == self.pushButton_2:\n self.label.setText(\"Stop video\")\n self.slot_stop()\n elif btn == self.pushButton_4:\n self.label.setText(\"Choose video\")\n self.select_video_path, _ = QFileDialog.getOpenFileName(self.frame, \"Open\", \"\", \"*.mp4;;All Files(*)\")\n print(\"-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-\")\n print(\"Select video file path:\\t\" + self.select_video_path)\n # ToDo To AP-kai:下断点来看一下，如果不选择的时候返回值是什么样的，然后再做判断，目前这个if没有生效\n if self.select_video_path != \"\":",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/build_gui.py:60-78"
+ },
+ "853": {
+ "file_id": 76,
+ "content": "The code above contains three elif conditions for button press events. If the self.pushButton_1 is pressed, it initializes variables and starts a timer to update the video frame. If self.pushButton_2 is pressed, it stops the video and sets the label text to \"Stop video\". If self.pushButton_4 is pressed, it opens a file dialog for choosing a video file, and if a file is chosen, it prints the selected video file path. The current code is checking if there is a non-empty selected video file path after the file dialog is closed.",
+ "type": "comment"
+ },
+ "854": {
+ "file_id": 76,
+ "content": " self.cap = cv2.VideoCapture(self.select_video_path)\n # 存所有frame\n self.save_temp_frame()\n print(\"save temp frame done\")\n self.progress_slider.setRange(0, self.cap.get(cv2.CAP_PROP_FRAME_COUNT))\n self.slider_frame_num = 0\n self.open_frame()\n # ToDo To AP-kai: 未来这个地方增加提示框，告诉他没有选择文件\n def on_cbtn_eraser_clicked(self):\n self.label.setText(\"Eraser On\")\n if self.cbtn_Eraser.isChecked():\n self.paintBoard.EraserMode = True # 进入橡皮擦模式\n else:\n self.paintBoard.EraserMode = False # 退出橡皮擦模式\n def fill_color_list(self, combo_box):\n index_black = 0\n index = 0\n for color in self.colorList:\n if color == \"black\":\n index_black = index\n index += 1\n pix = QPixmap(70, 20)\n pix.fill(QColor(color))\n combo_box.addItem(QIcon(pix), None)\n combo_box.setIconSize(QSize(70, 20))\n combo_box.setSizeAdjustPolicy(QComboBox.AdjustToContents)",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/build_gui.py:79-107"
+ },
+ "855": {
+ "file_id": 76,
+ "content": "The code snippet creates a GUI for video processing. It sets up a VideoCapture object, stores all frames, and initializes a progress slider with the total number of frames. The Eraser button toggles between EraserMode on/off in the paintBoard. The fill_color_list function populates a color combo box with predefined colors, including black at a specific index.",
+ "type": "comment"
+ },
+ "856": {
+ "file_id": 76,
+ "content": " combo_box.setCurrentIndex(index_black)\n def on_pen_color_change(self):\n self.label.setText(\"Change pen color\")\n color_index = self.comboBox_penColor.currentIndex()\n color_str = self.colorList[color_index]\n self.paintBoard.change_pen_color(color_str)\n # 拖拽进度条\n def update_video_position_func(self):\n self.label.setText(\"Change slider position\")\n self.slider_frame_num = self.progress_slider.value()\n self.slot_stop()\n self.open_frame()\n self.progress_slider.setValue(self.slider_frame_num)\n self.time_label.setText('{}/{}'.format(self.slider_frame_num, self.cap.get(7)))\n def save_temp_frame(self):\n _, self.all_frames = load_video(self.select_video_path, 480)\n def slot_stop(self):\n if self.cap != []:\n self.timer_camera.stop() # 停止计时器\n else:\n # ToDo To AP-kai: QMessageBox.warning没有返回值，这里我把Warming = QMessageBox.warning的Warming删去了\n QMessageBox.warning(self, \"Warming\", \"Push the left upper corner button to Quit.\",",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/build_gui.py:109-135"
+ },
+ "857": {
+ "file_id": 76,
+ "content": "This code defines several functions for a GUI application. It sets the current index of a combo box, handles pen color changes by updating the paintBoard's color, updates the video position based on progress slider input, saves a temporary frame from a video file, and stops the timer if it is running.",
+ "type": "comment"
+ },
+ "858": {
+ "file_id": 76,
+ "content": " QMessageBox.Yes)\n def open_frame(self):\n self.progress_slider.setValue(self.slider_frame_num)\n self.slider_frame_num = self.progress_slider.value()\n self.frame = self.all_frames[self.slider_frame_num]\n frame = self.frame\n height, width, bytes_per_component = frame.shape\n bytes_per_line = bytes_per_component * width\n q_image = QImage(frame.data, width, height, bytes_per_line,\n QImage.Format_RGB888).scaled(self.picturelabel.width(), self.picturelabel.height())\n self.picturelabel.setPixmap(QPixmap.fromImage(q_image))\n self.slider_frame_num = self.slider_frame_num + 1\n self.time_label.setText('{}/{}'.format(self.slider_frame_num, self.cap.get(7)))\n if self.progress_slider.value() == self.cap.get(7) - 1:\n self.slot_stop()",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/build_gui.py:136-151"
+ },
+ "859": {
+ "file_id": 76,
+ "content": "This code defines a function `open_frame` which updates the progress slider, displays the current frame using QImage and QPixmap, increments the slider value, sets the time label, and stops the video if at the last frame.",
+ "type": "comment"
+ },
+ "860": {
+ "file_id": 77,
+ "content": "/applications/EIVideo/QEIVideo/gui/__init__.py",
+ "type": "filepath"
+ },
+ "861": {
+ "file_id": 77,
+ "content": "This code block appears to be a comment section at the beginning of the file, indicating the author's name, date of creation, copyright information, and request for proper citation in case of reuse. The code seems to belong to PaddleVideo's EIVideo application, specifically within QEIVideo's gui module.",
+ "type": "summary"
+ },
+ "862": {
+ "file_id": 77,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/__init__.py:1-4"
+ },
+ "863": {
+ "file_id": 77,
+ "content": "This code block appears to be a comment section at the beginning of the file, indicating the author's name, date of creation, copyright information, and request for proper citation in case of reuse. The code seems to belong to PaddleVideo's EIVideo application, specifically within QEIVideo's gui module.",
+ "type": "comment"
+ },
+ "864": {
+ "file_id": 78,
+ "content": "/applications/EIVideo/QEIVideo/gui/demo.py",
+ "type": "filepath"
+ },
+ "865": {
+ "file_id": 78,
+ "content": "The code defines a `DrawFrame` class, which is a QWidget for drawing paths and responding to mouse events. It is used in conjunction with other classes such as `DemoUI`. The 'export' function converts painter's polygon to fill polygon and is triggered by the 'start_btn'. The code initializes the application and starts the event loop.",
+ "type": "summary"
+ },
+ "866": {
+ "file_id": 78,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport sys\nfrom PyQt5.QtWidgets import QApplication, QMainWindow, QFrame, QWidget\nfrom PyQt5.QtGui import QPainter, QPixmap, QPen, QColor, QPainterPath\nfrom PyQt5.QtCore import Qt, QPoint\nfrom PyQt5 import QtCore, QtGui, QtWidgets\nfrom QEIVideo.ui.demo import Ui_MainWindow as DemoUIRoot\nclass DrawFrame(QWidget):\n def __init__(self, painter, *args, **kwargs):\n super(DrawFrame, self).__init__(*args, **kwargs)\n self.painter = painter\n def paintEvent(self, event):\n painter = QPainter(self)\n pen = QPen(QColor(\"orange\"))\n pen.setWidth(5)\n pen.setCapStyle(Qt.RoundCap)\n pen.setJoinStyle(Qt.RoundJoin)\n painter.setPen(pen)\n painter.drawPath(self.painter)\n def mousePressEvent(self, event):\n self.painter.moveTo(event.pos())\n self.update()\n def mouseMoveEvent(self, event):\n self.painter.lineTo(event.pos())\n self.update()",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/demo.py:1-36"
+ },
+ "867": {
+ "file_id": 78,
+ "content": "This code defines a class called DrawFrame, which is a QWidget that can be drawn on. It overrides the paintEvent method to draw paths using a QPainter, and responds to mouse events for line drawing. The class takes a painter object in its constructor, suggesting it could be used in conjunction with other classes or methods.",
+ "type": "comment"
+ },
+ "868": {
+ "file_id": 78,
+ "content": "class DemoUI(QMainWindow, DemoUIRoot):\n def __init__(self):\n super(DemoUI, self).__init__()\n self.setupUi(self)\n self.painter = QPainterPath()\n self.draw_frame = DrawFrame(self.painter, self.video_frame)\n self.draw_frame.setGeometry(QtCore.QRect(0, 10, 751, 301))\n self.draw_frame.setObjectName(\"draw_frame\")\n self.draw_frame.raise_()\n self.draw_frame.setAttribute(QtCore.Qt.WA_TranslucentBackground)\n self.start_btn.clicked.connect(self.export)\n def export(self):\n a = self.painter.toFillPolygon()\n pass\nif __name__ == '__main__':\n app = QApplication(sys.argv)\n gui_class = DemoUI()\n gui_class.show()\n sys.exit(app.exec_())",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/demo.py:39-62"
+ },
+ "869": {
+ "file_id": 78,
+ "content": "This code initializes a `DemoUI` class with a `DrawFrame` object that draws video frames. The `export` function converts the painter's polygon to a fill polygon and is triggered by the 'start_btn'. The code also sets up the application, creates an instance of the `DemoUI` class, and starts the event loop.",
+ "type": "comment"
+ },
+ "870": {
+ "file_id": 79,
+ "content": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py",
+ "type": "filepath"
+ },
+ "871": {
+ "file_id": 79,
+ "content": "This code initializes a video/painting app with UI elements, connects events to functions for smooth operation, and displays \"Hi, This is EIVideo\" on stop.",
+ "type": "summary"
+ },
+ "872": {
+ "file_id": 79,
+ "content": "# -*- coding: utf-8 -*-\n# Form implementation generated from reading ui file 'GUI.ui'\n#\n# Created by: PyQt5 UI code generator 5.15.2\n#\n# WARNING: Any manual changes made to this file will be lost when pyuic5 is\n# run again. Do not edit this file unless you know what you are doing.\nfrom PyQt5 import QtCore, QtWidgets\nfrom PyQt5.QtGui import *\nfrom PyQt5.QtWidgets import *\nfrom PyQt5.QtCore import *\nfrom QEIVideo.widget.PaintBoard import PaintBoard\nclass Ui_MainWindow(object):\n def setupUi(self, MainWindow):\n MainWindow.setObjectName(\"EIVideo\")\n MainWindow.resize(1101, 751)\n self.centralwidget = QtWidgets.QWidget(MainWindow)\n self.centralwidget.setObjectName(\"centralwidget\")\n self.frame = QtWidgets.QFrame(self.centralwidget)\n self.frame.setGeometry(QtCore.QRect(20, 20, 1271, 771))\n self.frame.setFrameShadow(QtWidgets.QFrame.Raised)\n self.frame.setObjectName(\"frame\")\n self.cap = []\n self.all_frames = []\n self.fps = None\n self.timer = QTimer(self.frame)",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:1-32"
+ },
+ "873": {
+ "file_id": 79,
+ "content": "This code initializes the main window of a video application, sets its size and geometry, creates an empty list for capturing frames, and starts a timer.",
+ "type": "comment"
+ },
+ "874": {
+ "file_id": 79,
+ "content": " self.time_label = QLabel('--/--', self.frame)\n self.progress_slider = QSlider(self.frame)\n self.progress_slider.setEnabled(True)\n self.progress_slider.setOrientation(Qt.Horizontal)\n self.progress_slider.setFixedWidth(710)\n self.progress_slider.setFixedHeight(20)\n self.progress_slider.setSingleStep(1) # 设置变化步长\n self.progress_slider.setValue(0)\n self.progress_slider.sliderReleased.connect(self.update_video_position_func) # 拖拽进度条\n self.picturelabel = QtWidgets.QLabel(self.frame)\n self.picturelabel.setGeometry(30, 30, 810, 458)\n self.picturelabel.setText(\"\")\n self.picturelabel.setObjectName(\"picturelabel\")\n self.paintBoard = PaintBoard(self.frame)\n self.paintBoard.setGeometry(30, 30, 810, 458)\n self.cbtn_Eraser = QCheckBox(\"橡皮擦\")\n self.cbtn_Eraser.setParent(self.frame)\n self.cbtn_Eraser.move(950, 40)\n self.cbtn_Eraser.clicked.connect(self.on_cbtn_eraser_clicked)\n self.btn_Clear = QPushButton(\"清空画板\")",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:33-56"
+ },
+ "875": {
+ "file_id": 79,
+ "content": "This code is initializing UI elements in a window, setting up progress slider for video position tracking, adding a picture label and paint board for drawing, and enabling interaction with eraser checkbox and clear button.",
+ "type": "comment"
+ },
+ "876": {
+ "file_id": 79,
+ "content": " self.btn_Clear.setParent(self.frame) # 设置父对象为本界面\n self.btn_Clear.move(950, 60)\n self.btn_Clear.clicked.connect(self.paintBoard.clear)\n self.label_penColor = QLabel(self.frame)\n self.label_penColor.setText(\"画笔颜色\")\n self.label_penColor.move(990, 100)\n # 获取颜色列表(字符串类型)\n self.colorList = QColor.colorNames()\n self.comboBox_penColor = QComboBox(self.frame)\n self.fill_color_list(self.comboBox_penColor) # 用各种颜色填充下拉列表\n self.comboBox_penColor.move(1080, 80)\n self.comboBox_penColor.currentIndexChanged.connect(\n self.on_pen_color_change) # 关联下拉列表的当前索引变更信号与函数on_PenColorChange\n self.helplabel = QLabel()\n self.helplabel.setText(\"Hi,Welcome to use EIVideo\\n\"\n \"This is a guide for EIVideo,\\n\"\n \"please check\\n\"\n \"1. Choose 'Add' for a video\\n\"\n \"2. Click 'Play' to start playing\\n\"\n \"3. At this point, all functions \\n\"",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:57-77"
+ },
+ "877": {
+ "file_id": 79,
+ "content": "The code sets the parent object of btn_Clear to self.frame, moves the btn_Clear and label_penColor widgets to specific positions, connects a button click event to clear the paintBoard, fills a comboBox_penColor with color options, places it at a particular location, links its currentIndexChanged signal to on_pen_color_change function, and provides guidance for using EIVideo.",
+ "type": "comment"
+ },
+ "878": {
+ "file_id": 79,
+ "content": " \"are unlocked\\n\"\n \"4. Paint and enjoy it!\\n\")\n self.widget2 = QtWidgets.QWidget(self.frame)\n self.widget2.setGeometry(860, 60, 200, 300)\n self.widget2.setObjectName(\"widget2\")\n self.rightLayout = QtWidgets.QVBoxLayout(self.widget2)\n self.rightLayout.setContentsMargins(0, 0, 0, 0)\n self.rightLayout.setObjectName(\"rightLayout\")\n self.rightLayout.addWidget(self.helplabel)\n self.rightLayout.addSpacing(50)\n self.rightLayout.addWidget(self.cbtn_Eraser)\n self.rightLayout.addWidget(self.btn_Clear)\n self.colorLayout = QtWidgets.QHBoxLayout(self.widget2)\n self.colorLayout.setContentsMargins(0, 0, 0, 0)\n self.colorLayout.setObjectName('colorLayout')\n self.colorLayout.addWidget(self.label_penColor)\n self.colorLayout.addWidget(self.comboBox_penColor)\n self.rightLayout.addLayout(self.colorLayout)\n # pushButton_6 -> GO\n self.pushButton_6 = QtWidgets.QPushButton(self.frame)",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:78-101"
+ },
+ "879": {
+ "file_id": 79,
+ "content": "This code sets up a user interface layout for a painting application. It includes a help label, buttons to clear and switch the eraser, a color picker, and a 'GO' push button. The layout is organized in a vertical box and horizontal box arrangement.",
+ "type": "comment"
+ },
+ "880": {
+ "file_id": 79,
+ "content": " self.pushButton_6.setGeometry(870, 600, 150, 90)\n self.pushButton_6.setObjectName(\"pushButton_6\")\n self.pushButton_6.clicked.connect(self.infer)\n self.widget1 = QtWidgets.QWidget(self.frame)\n self.widget1.move(60, 520)\n self.widget1.setObjectName(\"widget1\")\n self.barLayout = QtWidgets.QVBoxLayout(self.widget1)\n self.barLayout.setContentsMargins(0, 0, 0, 0)\n self.barLayout.setObjectName(\"barLayout\")\n self.horizontalLayout = QtWidgets.QHBoxLayout(self.widget1)\n self.horizontalLayout.setContentsMargins(0, 0, 0, 0)\n self.horizontalLayout.setObjectName(\"horizontalLayout\")\n self.timeLayout = QtWidgets.QHBoxLayout(self.widget1)\n self.timeLayout.setContentsMargins(0, 0, 0, 0)\n self.timeLayout.setObjectName(\"horizontalLayout\")\n self.playbtn = QtWidgets.QPushButton(self.widget1)\n self.playbtn.setObjectName(\"playbtn\")\n self.playbtn.clicked.connect(lambda: self.btn_func(self.playbtn))\n self.horizontalLayout.addWidget(self.playbtn)",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:102-122"
+ },
+ "881": {
+ "file_id": 79,
+ "content": "This code is creating a UI for a video player. It sets the position and function of a play button, as well as defining layouts for other UI elements such as time display. The play button's click event is connected to a method called \"btn_func\" which takes the play button as an argument.",
+ "type": "comment"
+ },
+ "882": {
+ "file_id": 79,
+ "content": " self.pushButton_2 = QtWidgets.QPushButton(self.widget1)\n self.pushButton_2.setObjectName(\"pushButton_2\")\n self.pushButton_2.clicked.connect(lambda: self.btn_func(self.pushButton_2))\n self.horizontalLayout.addWidget(self.pushButton_2)\n self.pushButton_4 = QtWidgets.QPushButton(self.widget1)\n self.pushButton_4.setObjectName(\"pushButton_4\")\n self.pushButton_4.clicked.connect(lambda: self.btn_func(self.pushButton_4))\n self.horizontalLayout.addWidget(self.pushButton_4)\n self.timeLayout.addWidget(self.progress_slider)\n self.timeLayout.addWidget(self.time_label)\n self.barLayout.addSpacing(20)\n self.barLayout.addLayout(self.timeLayout)\n self.barLayout.addSpacing(30)\n self.barLayout.addLayout(self.horizontalLayout)\n self.splitter = QtWidgets.QSplitter(self.frame)\n self.splitter.setGeometry(QtCore.QRect(71, 670, 750, 20))\n self.splitter.setOrientation(QtCore.Qt.Horizontal)\n self.splitter.setObjectName(\"splitter\")",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:123-142"
+ },
+ "883": {
+ "file_id": 79,
+ "content": "This code initializes two push buttons, connects their click events to a function, adds them to a horizontal layout, and adds the layout to a splitter. It also adds time-related widgets to another layout, and adds both layouts to the splitter.",
+ "type": "comment"
+ },
+ "884": {
+ "file_id": 79,
+ "content": " self.label = QtWidgets.QLabel(self.splitter)\n self.label.setObjectName(\"label\")\n self.progressBar = QtWidgets.QProgressBar(self.splitter)\n self.progressBar.setProperty(\"value\", 0)\n self.progressBar.setObjectName(\"progressBar\")\n MainWindow.setCentralWidget(self.centralwidget)\n self.menubar = QtWidgets.QMenuBar(MainWindow)\n self.menubar.setGeometry(QtCore.QRect(0, 0, 1327, 23))\n self.menubar.setObjectName(\"menubar\")\n MainWindow.setMenuBar(self.menubar)\n self.statusbar = QtWidgets.QStatusBar(MainWindow)\n self.statusbar.setObjectName(\"statusbar\")\n MainWindow.setStatusBar(self.statusbar)\n self.retranslateUi(MainWindow)\n QtCore.QMetaObject.connectSlotsByName(MainWindow)\n def retranslateUi(self, MainWindow):\n _translate = QtCore.QCoreApplication.translate\n MainWindow.setWindowTitle(_translate(\"MainWindow\", \"MainWindow\"))\n self.pushButton_6.setText(_translate(\"MainWindow\", \"GO\"))\n self.playbtn.setText(_translate(\"MainWindow\", \"Play\"))",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:143-164"
+ },
+ "885": {
+ "file_id": 79,
+ "content": "This code is creating a user interface for a main window of an application. It includes a label, progress bar, menu bar, and status bar. The window has a title and two buttons: \"GO\" and \"Play\". The `retranslateUi` function is used to set the window title and button labels.",
+ "type": "comment"
+ },
+ "886": {
+ "file_id": 79,
+ "content": " self.pushButton_2.setText(_translate(\"MainWindow\", \"Stop\"))\n self.pushButton_4.setText(_translate(\"MainWindow\", \"Add\"))\n self.label.setText(_translate(\"MainWindow\", \"Hi, This is EIVideo\"))",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/ui_main_window.py:165-167"
+ },
+ "887": {
+ "file_id": 79,
+ "content": "This code updates the text on three GUI elements in the \"MainWindow\" class. The first button is labeled \"Stop,\" the second button is labeled \"Add,\" and a label displays \"Hi, This is EIVideo.\"",
+ "type": "comment"
+ },
+ "888": {
+ "file_id": 80,
+ "content": "/applications/EIVideo/QEIVideo/start.py",
+ "type": "filepath"
+ },
+ "889": {
+ "file_id": 80,
+ "content": "The code is a Python script that initializes a QApplication object, creates an instance of the BuildGUI class from the QEIVideo module, displays it, and executes the application's event loop. It is likely used to launch a graphical user interface (GUI) for a video processing or analysis application. The author is credited, and the code includes a copyright notice requesting proper attribution if reused.",
+ "type": "summary"
+ },
+ "890": {
+ "file_id": 80,
+ "content": "# Author: AP-Kai\n# Datetime: 2022/1/7\n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport sys\nfrom QEIVideo.build_gui import BuildGUI\nfrom PyQt5.QtWidgets import QApplication\ndef run():\n app = QApplication(sys.argv)\n demo = BuildGUI()\n demo.show()\n sys.exit(app.exec())\nif __name__ == '__main__':\n run()",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/start.py:1-20"
+ },
+ "891": {
+ "file_id": 80,
+ "content": "The code is a Python script that initializes a QApplication object, creates an instance of the BuildGUI class from the QEIVideo module, displays it, and executes the application's event loop. It is likely used to launch a graphical user interface (GUI) for a video processing or analysis application. The author is credited, and the code includes a copyright notice requesting proper attribution if reused.",
+ "type": "comment"
+ },
+ "892": {
+ "file_id": 81,
+ "content": "/applications/EIVideo/QEIVideo/tools/__init__.py",
+ "type": "filepath"
+ },
+ "893": {
+ "file_id": 81,
+ "content": "This code block appears to be a comment section at the beginning of the file, indicating the author's name, date of creation, copyright information, and request for proper citation in case of reuse. The code seems to belong to PaddleVideo's EIVideo application, specifically within QEIVideo's gui module.",
+ "type": "summary"
+ },
+ "894": {
+ "file_id": 81,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/__init__.py:1-4"
+ },
+ "895": {
+ "file_id": 81,
+ "content": "This code block appears to be a comment section at the beginning of the file, indicating the author's name, date of creation, copyright information, and request for proper citation in case of reuse. The code seems to belong to PaddleVideo's EIVideo application, specifically within QEIVideo's gui module.",
+ "type": "comment"
+ },
+ "896": {
+ "file_id": 82,
+ "content": "/applications/EIVideo/QEIVideo/ui/__init__.py",
+ "type": "filepath"
+ },
+ "897": {
+ "file_id": 82,
+ "content": "This code block appears to be a comment section at the beginning of the file, indicating the author's name, date of creation, copyright information, and request for proper citation in case of reuse. The code seems to belong to PaddleVideo's EIVideo application, specifically within QEIVideo's gui module.",
+ "type": "summary"
+ },
+ "898": {
+ "file_id": 82,
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.",
+ "type": "code",
+ "location": "/applications/EIVideo/QEIVideo/gui/__init__.py:1-4"
+ },
+ "899": {
+ "file_id": 82,
+ "content": "This code block appears to be a comment section at the beginning of the file, indicating the author's name, date of creation, copyright information, and request for proper citation in case of reuse. The code seems to belong to PaddleVideo's EIVideo application, specifically within QEIVideo's gui module.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/80.json b/docs/data/80.json
new file mode 100644
index 000000000..250492bba
--- /dev/null
+++ b/docs/data/80.json
@@ -0,0 +1,545 @@
+{
+ "8000": {
+ "file_id": 589,
+ "content": " num, cnt = np.unique(label, return_counts=True)\n for n, c in zip(num, cnt):\n nums[n] += c\n class_num = paddle.to_tensor(nums, dtype=\"float32\")\n total = class_num.sum().item()\n frequency = class_num / total\n median = paddle.median(frequency)\n class_weight = median / frequency\n return class_weight\n def forward(self, preds, gts, sim_index):\n \"\"\"\n Args:\n preds: paddle.float (N, C, T).\n gts: paddle.int64 (N, T).\n sim_index: paddle.float (N, C', T).\n \"\"\"\n loss = 0.0\n for criterion, weight in zip(self.criterions, self.weights):\n if isinstance(criterion, GaussianSimilarityTMSE):\n loss += weight * criterion(preds, gts, sim_index)\n elif isinstance(criterion, nn.CrossEntropyLoss):\n preds_t = paddle.transpose(preds, perm=[0, 2, 1])\n loss += weight * criterion(preds_t, gts)\n else:\n loss += weight * criterion(preds, gts)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:222-248"
+ },
+ "8001": {
+ "file_id": 589,
+ "content": "This code defines a class that calculates class weights based on the frequency of occurrence in labels. It also includes a forward function for applying different loss functions to predictions and ground truths, with associated weights. The criterion types include GaussianSimilarityTMSE and nn.CrossEntropyLoss.",
+ "type": "comment"
+ },
+ "8002": {
+ "file_id": 589,
+ "content": " return loss\nclass BoundaryRegressionLoss(nn.Layer):\n \"\"\"\n Boundary Regression Loss\n bce: Binary Cross Entropy Loss for Boundary Prediction\n mse: Mean Squared Error\n \"\"\"\n def __init__(self,\n file_path,\n label_path,\n bce=True,\n focal=False,\n mse=False,\n weight=None,\n pos_weight=None):\n super().__init__()\n self.criterions = []\n self.file_path = file_path\n self.label_path = label_path\n pos_weight = self.get_pos_weight()\n if bce:\n self.criterions.append(\n nn.BCEWithLogitsLoss(weight=weight, pos_weight=pos_weight))\n if focal:\n self.criterions.append(FocalLoss())\n if mse:\n self.criterions.append(nn.MSELoss())\n if len(self.criterions) == 0:\n print(\"You have to choose at least one loss function.\")\n sys.exit(1)\n def get_pos_weight(self, norm=None):\n \"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:250-291"
+ },
+ "8003": {
+ "file_id": 589,
+ "content": "This class defines a boundary regression loss function, which combines different loss types such as Binary Cross Entropy Loss (bce), Mean Squared Error (mse) and Focal Loss (focal). It initializes with file_path, label_path, bce, focal, mse, weight and pos_weight parameters. The get_pos_weight method retrieves a position weight depending on the norm parameter. If at least one loss function is chosen, the criterions list is created. If not, it prints an error message and exits the program.",
+ "type": "comment"
+ },
+ "8004": {
+ "file_id": 589,
+ "content": " pos_weight for binary cross entropy with logits loss\n pos_weight is defined as reciprocal of ratio of positive samples in the dataset\n \"\"\"\n # load file list\n file_ptr = open(self.file_path, 'r')\n info = file_ptr.read().split('\\n')[:-1]\n file_ptr.close()\n n_classes = 2 # boundary or not\n nums = [0 for i in range(n_classes)]\n for i in range(len(info)):\n video_name = info[i]\n file_name = video_name.split('.')[0] + \".npy\"\n label_file_path = os.path.join(self.label_path, file_name)\n label = np.load(label_file_path).astype(np.int64)\n num, cnt = np.unique(label, return_counts=True)\n for n, c in zip(num, cnt):\n nums[n] += c\n pos_ratio = nums[1] / sum(nums)\n pos_weight = 1 / pos_ratio\n if norm is not None:\n pos_weight /= norm\n return paddle.to_tensor(pos_weight, dtype=\"float32\")\n def forward(self, preds, gts):\n \"\"\"\n Args:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:292-321"
+ },
+ "8005": {
+ "file_id": 589,
+ "content": "This code calculates the positive weight for binary cross-entropy with logits loss. It loads file information from a given path, counts the number of positive and negative samples, then calculates the ratio of positive samples to total samples. The positive weight is set as the reciprocal of this ratio. If a normalization factor is provided, it divides the positive weight by this factor before returning it in float32 tensor format.",
+ "type": "comment"
+ },
+ "8006": {
+ "file_id": 589,
+ "content": " preds: paddle.float (N, 1, T).\n gts: paddle.float (N, 1, T).\n \"\"\"\n loss = 0.0\n batch_size = float(preds.shape[0])\n for criterion in self.criterions:\n for pred, gt in zip(preds, gts):\n loss += criterion(pred, gt)\n return loss / batch_size\n@LOSSES.register()\nclass ASRFLoss(nn.Layer):\n def __init__(self,\n lambda_bound_loss,\n num_classes,\n file_path,\n label_path,\n boundary_path,\n ce=True,\n asl_focal=True,\n tmse=False,\n gstmse=False,\n asl_weight=None,\n threshold=4.,\n ignore_index=255,\n ce_weight=1.0,\n focal_weight=1.0,\n tmse_weight=0.15,\n gstmse_weight=0.15,\n bce=True,\n brl_focal=False,\n mse=False,\n brl_weight=None):\n super().__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:322-359"
+ },
+ "8007": {
+ "file_id": 589,
+ "content": "The ASRFLoss class is an ASR (Automatic Speech Recognition) loss function implemented with various criteria for prediction and ground truth. It uses different weights for CE, Focal, TMSE, GST MSE, BCE, and BR LFocal losses depending on the input parameters. The function returns the average loss across all criterions and samples.",
+ "type": "comment"
+ },
+ "8008": {
+ "file_id": 589,
+ "content": " self.criterion_cls = ActionSegmentationLoss(ce=ce,\n focal=asl_focal,\n tmse=tmse,\n gstmse=gstmse,\n weight=asl_weight,\n threshold=threshold,\n ignore_index=ignore_index,\n ce_weight=ce_weight,\n focal_weight=focal_weight,\n tmse_weight=tmse_weight,\n gstmse_weight=gstmse_weight,\n file_path=file_path,\n label_path=label_path,\n num_classes=num_classes)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:360-373"
+ },
+ "8009": {
+ "file_id": 589,
+ "content": "This code initializes an ActionSegmentationLoss object with specified parameters for classification loss, focal loss, and temporal segmentation losses. It also takes weights and file paths as inputs to optimize the model's performance.",
+ "type": "comment"
+ },
+ "8010": {
+ "file_id": 589,
+ "content": " self.criterion_boundary = BoundaryRegressionLoss(\n bce=bce,\n focal=brl_focal,\n mse=mse,\n weight=brl_weight,\n file_path=file_path,\n label_path=boundary_path)\n self.lambda_bound_loss = lambda_bound_loss\n def forward(self, x, output_cls, label, outputs_boundary, boundary):\n loss = 0.0\n if isinstance(output_cls, list):\n n = len(output_cls)\n for out in output_cls:\n loss += self.criterion_cls(out, label, x) / n\n else:\n loss += self.criterion_cls(output_cls, label, x)\n if isinstance(outputs_boundary, list):\n n = len(outputs_boundary)\n for out in outputs_boundary:\n loss += self.lambda_bound_loss * self.criterion_boundary(\n out, boundary) / n\n else:\n loss += self.lambda_bound_loss * self.criterion_boundary(\n outputs_boundary, boundary)\n return loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/asrf_loss.py:374-401"
+ },
+ "8011": {
+ "file_id": 589,
+ "content": "This code defines a custom loss function for a video modeling framework. It initializes a boundary regression loss criterion and takes a weighted average of classification and boundary losses. The forward method calculates the total loss by summing weighted classification and boundary losses, and returns the final loss value.",
+ "type": "comment"
+ },
+ "8012": {
+ "file_id": 590,
+ "content": "/paddlevideo/modeling/losses/base.py",
+ "type": "filepath"
+ },
+ "8013": {
+ "file_id": 590,
+ "content": "This code is a PaddlePaddle base class for loss functions, requiring subclasses to override `_forward()` and supports optional weight scaling. It initializes the loss class and defines forward pass computation.",
+ "type": "summary"
+ },
+ "8014": {
+ "file_id": 590,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\n#XXX use _forward?? or forward??\nclass BaseWeightedLoss(nn.Layer):\n \"\"\"Base class for loss.\n All subclass should overwrite the ``_forward()`` method which returns the\n normal loss without loss weights.\n Args:\n loss_weight (float): Factor scalar multiplied on the loss.\n Default: 1.0.\n \"\"\"\n def __init__(self, loss_weight=1.0):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/base.py:1-31"
+ },
+ "8015": {
+ "file_id": 590,
+ "content": "This code is the base class for a loss function in PaddlePaddle. It requires subclasses to override the `_forward()` method, which returns the normal loss without weights. The `loss_weight` parameter is optional and defaults to 1.0, which can be used to scale the final loss value.",
+ "type": "comment"
+ },
+ "8016": {
+ "file_id": 590,
+ "content": " super().__init__()\n self.loss_weight = loss_weight\n @abstractmethod\n def _forward(self, *args, **kwargs):\n pass\n def forward(self, *args, **kwargs):\n \"\"\"Defines the computation performed at every call.\n Args:\n *args: The positional arguments for the corresponding\n loss.\n **kwargs: The keyword arguments for the corresponding\n loss.\n Returns:\n paddle.Tensor: The calculated loss.\n \"\"\"\n return self._forward(*args, **kwargs) * self.loss_weight",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/base.py:32-49"
+ },
+ "8017": {
+ "file_id": 590,
+ "content": "Initializes the loss class with a weight and defines forward pass for computation.",
+ "type": "comment"
+ },
+ "8018": {
+ "file_id": 591,
+ "content": "/paddlevideo/modeling/losses/bmn_loss.py",
+ "type": "filepath"
+ },
+ "8019": {
+ "file_id": 591,
+ "content": "This code defines a BMN loss function for PaddleVideo, considering time-scale attributes and ratio of positive entries. It also includes a loss function for object detection models with weighted samples, position losses, and ground truth IoU masks. The code further defines a loss function for PEM and TEAM tasks by combining predicted and ground truth values using three loss functions.",
+ "type": "summary"
+ },
+ "8020": {
+ "file_id": 591,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass BMNLoss(BaseWeightedLoss):\n \"\"\"Loss for BMN model\n Args:\n tscale (int): sequence length, default 100.\n dscale (int): max duration length, default 100.\n \"\"\"\n def __init__(self, dscale, tscale):\n super().__init__()\n self.dscale = dscale",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:1-32"
+ },
+ "8021": {
+ "file_id": 591,
+ "content": "This code defines a BMN loss function for the PaddleVideo library. It is registered in the LOSSES registry and takes two arguments: dscale and tscale, which represent max duration length and sequence length respectively. The class extends BaseWeightedLoss, suggesting it combines multiple weighted losses.",
+ "type": "comment"
+ },
+ "8022": {
+ "file_id": 591,
+ "content": " self.tscale = tscale\n def _get_mask(self, dscale, tscale):\n bm_mask = []\n for idx in range(dscale):\n mask_vector = [1 for i in range(tscale - idx)\n ] + [0 for i in range(idx)]\n bm_mask.append(mask_vector)\n bm_mask = np.array(bm_mask, dtype='float32')\n bm_mask = paddle.to_tensor(bm_mask)\n bm_mask.stop_gradient = True\n return bm_mask\n def tem_loss_func(self, pred_start, pred_end, gt_start, gt_end):\n def bi_loss(pred_score, gt_label, datatype):\n pred_score = paddle.reshape(x=pred_score, shape=[-1])\n gt_label = paddle.reshape(x=gt_label, shape=[-1])\n gt_label.stop_gradient = True\n pmask = paddle.cast(x=(gt_label > 0.5), dtype=datatype)\n num_entries = paddle.cast(paddle.shape(pmask), dtype=datatype)\n num_positive = paddle.cast(paddle.sum(pmask), dtype=datatype)\n ratio = num_entries / num_positive\n coef_0 = 0.5 * ratio / (ratio - 1)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:33-55"
+ },
+ "8023": {
+ "file_id": 591,
+ "content": "This code defines a class with a time-scale attribute, a method to create binary mask arrays, and a loss function for a specific task. The loss function takes in predicted start and end positions along with ground truth values and calculates a ratio between the number of entries and positive values. This ratio is then used to calculate a coefficient for the loss function.",
+ "type": "comment"
+ },
+ "8024": {
+ "file_id": 591,
+ "content": " coef_1 = 0.5 * ratio\n epsilon = 0.000001\n loss_pos = paddle.multiply(paddle.log(pred_score + epsilon), pmask)\n loss_pos = coef_1 * paddle.mean(loss_pos)\n loss_neg = paddle.multiply(paddle.log(1.0 - pred_score + epsilon),\n (1.0 - pmask))\n loss_neg = coef_0 * paddle.mean(loss_neg)\n loss = -1 * (loss_pos + loss_neg)\n return loss\n loss_start = bi_loss(pred_start, gt_start, pred_start.dtype)\n loss_end = bi_loss(pred_end, gt_end, pred_start.dtype)\n loss = loss_start + loss_end\n return loss\n def pem_reg_loss_func(self, pred_score, gt_iou_map, mask):\n gt_iou_map = paddle.multiply(gt_iou_map, mask)\n u_hmask = paddle.cast(x=gt_iou_map > 0.7, dtype=pred_score.dtype)\n u_mmask = paddle.logical_and(gt_iou_map <= 0.7, gt_iou_map > 0.3)\n u_mmask = paddle.cast(x=u_mmask, dtype=pred_score.dtype)\n u_lmask = paddle.logical_and(gt_iou_map <= 0.3, gt_iou_map >= 0.)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:56-77"
+ },
+ "8025": {
+ "file_id": 591,
+ "content": "The code defines a loss function for object detection models. It calculates the loss by considering positive and negative samples, applying weights to each sample based on their ratio, and then combines them. The bi_loss function is used to calculate losses for start and end positions. In another function, Pem_reg_loss_func, it separates ground truth IoU map into three masks: high (>0.7), medium (<=0.7 & >0.3), and low (<=0.3 & >=0). It then applies these masks to calculate the loss.",
+ "type": "comment"
+ },
+ "8026": {
+ "file_id": 591,
+ "content": " u_lmask = paddle.cast(x=u_lmask, dtype=pred_score.dtype)\n u_lmask = paddle.multiply(u_lmask, mask)\n num_h = paddle.cast(paddle.sum(u_hmask), dtype=pred_score.dtype)\n num_m = paddle.cast(paddle.sum(u_mmask), dtype=pred_score.dtype)\n num_l = paddle.cast(paddle.sum(u_lmask), dtype=pred_score.dtype)\n r_m = num_h / num_m\n u_smmask = paddle.uniform(shape=[\n gt_iou_map.shape[1], gt_iou_map.shape[2]\n ],\n min=0.0,\n max=1.0).astype(pred_score.dtype)\n u_smmask = paddle.multiply(u_mmask, u_smmask)\n u_smmask = paddle.cast(x=(u_smmask > (1. - r_m)),\n dtype=pred_score.dtype)\n r_l = num_h / num_l\n u_slmask = paddle.uniform(shape=[\n gt_iou_map.shape[1], gt_iou_map.shape[2]\n ],\n min=0.0,\n max=1.0).astype(pred_score.dtype)\n u_slmask = paddle.multiply(u_lmask, u_slmask)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:78-101"
+ },
+ "8027": {
+ "file_id": 591,
+ "content": "Calculating the number of elements in different masks and using them to calculate ratios for later mask operations. Creating uniform masks and multiplying them with corresponding existing masks, then casting the results.",
+ "type": "comment"
+ },
+ "8028": {
+ "file_id": 591,
+ "content": " u_slmask = paddle.cast(x=(u_slmask > (1. - r_l)),\n dtype=pred_score.dtype)\n weights = u_hmask + u_smmask + u_slmask\n weights.stop_gradient = True\n loss = F.square_error_cost(pred_score, gt_iou_map)\n loss = paddle.multiply(loss, weights)\n loss = 0.5 * paddle.sum(loss) / paddle.sum(weights)\n return loss\n def pem_cls_loss_func(self, pred_score, gt_iou_map, mask):\n gt_iou_map = paddle.multiply(gt_iou_map, mask)\n gt_iou_map.stop_gradient = True\n pmask = paddle.cast(x=(gt_iou_map > 0.9), dtype=pred_score.dtype)\n nmask = paddle.cast(x=(gt_iou_map <= 0.9), dtype=pred_score.dtype)\n nmask = paddle.multiply(nmask, mask)\n num_positive = paddle.sum(pmask)\n num_entries = num_positive + paddle.sum(nmask)\n ratio = num_entries / num_positive\n coef_0 = 0.5 * ratio / (ratio - 1)\n coef_1 = 0.5 * ratio\n epsilon = 0.000001\n loss_pos = paddle.multiply(paddle.log(pred_score + epsilon), pmask)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:102-126"
+ },
+ "8029": {
+ "file_id": 591,
+ "content": "In this code, u_slmask is created by comparing r_l with 1 and casting the result to the dtype of pred_score. Then, weights are calculated by adding u_hmask, u_smmask, and u_slmask. The stop_gradient attribute of weights is set to True. Loss is calculated using square error cost between pred_score and gt_iou_map, multiplied by weights, averaged, and returned.\nIn the pem_cls_loss_func, gt_iou_map is multiplied by mask and marked as non-trainable (stop_gradient = True). Pmask and nmask are created based on conditions with gt_iou_map and mask. Num_positive and num_entries are calculated. Ratios are used to determine coef_0 and coef_1. Loss_pos is log(pred_score + epsilon) multiplied by pmask.",
+ "type": "comment"
+ },
+ "8030": {
+ "file_id": 591,
+ "content": " loss_pos = coef_1 * paddle.sum(loss_pos)\n loss_neg = paddle.multiply(paddle.log(1.0 - pred_score + epsilon),\n nmask)\n loss_neg = coef_0 * paddle.sum(loss_neg)\n loss = -1 * (loss_pos + loss_neg) / num_entries\n return loss\n def forward(self, pred_bm, pred_start, pred_end, gt_iou_map, gt_start,\n gt_end):\n pred_bm_reg = paddle.squeeze(paddle.slice(pred_bm,\n axes=[1],\n starts=[0],\n ends=[1]),\n axis=[1])\n pred_bm_cls = paddle.squeeze(paddle.slice(pred_bm,\n axes=[1],\n starts=[1],\n ends=[2]),\n axis=[1])\n bm_mask = self._get_mask(self.dscale, self.tscale)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:127-147"
+ },
+ "8031": {
+ "file_id": 591,
+ "content": "Function `forward` takes in `pred_bm`, `pred_start`, `pred_end`, `gt_iou_map`, `gt_start`, and `gt_end`. It first extracts `pred_bm_reg` and `pred_bm_cls` by slicing `pred_bm` along the specified axes. Then, it calculates the `bm_mask` using `_get_mask` with given scales. The function returns the calculated loss from the input parameters.",
+ "type": "comment"
+ },
+ "8032": {
+ "file_id": 591,
+ "content": " pem_reg_loss = self.pem_reg_loss_func(pred_bm_reg, gt_iou_map, bm_mask)\n pem_cls_loss = self.pem_cls_loss_func(pred_bm_cls, gt_iou_map, bm_mask)\n tem_loss = self.tem_loss_func(pred_start, pred_end, gt_start, gt_end)\n loss = tem_loss + 10 * pem_reg_loss + pem_cls_loss\n return loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/bmn_loss.py:149-155"
+ },
+ "8033": {
+ "file_id": 591,
+ "content": "This code calculates the loss for PEM and TEAM detection tasks by combining the predicted and ground truth values. It uses three loss functions: `pem_reg_loss_func`, `pem_cls_loss_func`, and `tem_loss_func`. The final loss is the sum of the temporal (TEM) loss, 10 times the PEM regression loss, and the PEM classification loss.",
+ "type": "comment"
+ },
+ "8034": {
+ "file_id": 592,
+ "content": "/paddlevideo/modeling/losses/cross_entropy_loss.py",
+ "type": "filepath"
+ },
+ "8035": {
+ "file_id": 592,
+ "content": "CrossEntropyLoss is a custom loss function in PaddlePaddle, inheriting from BaseWeightedLoss, for classification tasks. It calculates CrossEntropy loss between scores and labels using F.cross_entropy method and returns the result as a tensor.",
+ "type": "summary"
+ },
+ "8036": {
+ "file_id": 592,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass CrossEntropyLoss(BaseWeightedLoss):\n \"\"\"Cross Entropy Loss.\"\"\"\n def _forward(self, score, labels, **kwargs):\n \"\"\"Forward function.\n Args:\n score (paddle.Tensor): The class score.\n labels (paddle.Tensor): The ground truth labels.\n kwargs: Any keyword argument to be used to calculate",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/cross_entropy_loss.py:1-30"
+ },
+ "8037": {
+ "file_id": 592,
+ "content": "CrossEntropyLoss is a custom loss function in PaddlePaddle for classification tasks. It inherits from BaseWeightedLoss and has a forward method that takes class scores and labels as input.",
+ "type": "comment"
+ },
+ "8038": {
+ "file_id": 592,
+ "content": " CrossEntropy loss.\n Returns:\n loss (paddle.Tensor): The returned CrossEntropy loss.\n \"\"\"\n loss = F.cross_entropy(score, labels, **kwargs)\n return loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/cross_entropy_loss.py:31-36"
+ },
+ "8039": {
+ "file_id": 592,
+ "content": "This function calculates the CrossEntropy loss between score and labels, using Paddle's F.cross_entropy method, and returns the result as a tensor.",
+ "type": "comment"
+ },
+ "8040": {
+ "file_id": 593,
+ "content": "/paddlevideo/modeling/losses/depth_loss.py",
+ "type": "filepath"
+ },
+ "8041": {
+ "file_id": 593,
+ "content": "This code calculates smoothness and reprojection losses for depth estimation tasks, combining identity and reprojection losses to compute disparity loss. It handles day and night scenarios while saving images if necessary. The total loss is stored in the losses dictionary.",
+ "type": "summary"
+ },
+ "8042": {
+ "file_id": 593,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\ndef get_smooth_loss(disp, img):\n \"\"\"Computes the smoothness loss for a disparity image\n The color image is used for edge-aware smoothness\n \"\"\"\n grad_disp_x = paddle.abs(disp[:, :, :, :-1] - disp[:, :, :, 1:])\n grad_disp_y = paddle.abs(disp[:, :, :-1, :] - disp[:, :, 1:, :])\n grad_img_x = paddle.mean(paddle.abs(img[:, :, :, :-1] - img[:, :, :, 1:]),",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:1-29"
+ },
+ "8043": {
+ "file_id": 593,
+ "content": "This code defines a function \"get_smooth_loss\" that calculates the smoothness loss for disparity images using color image gradients and disparity image gradients. It uses PaddlePaddle library functions like paddle.abs() and paddle.mean(). The function is part of the BaseWeightedLoss class in the LOSSES registry.",
+ "type": "comment"
+ },
+ "8044": {
+ "file_id": 593,
+ "content": " 1,\n keepdim=True)\n grad_img_y = paddle.mean(paddle.abs(img[:, :, :-1, :] - img[:, :, 1:, :]),\n 1,\n keepdim=True)\n grad_disp_x *= paddle.exp(-grad_img_x)\n grad_disp_y *= paddle.exp(-grad_img_y)\n return grad_disp_x.mean() + grad_disp_y.mean()\nclass DiffLoss(nn.Layer):\n def __init__(self):\n super(DiffLoss, self).__init__()\n def forward(self, input1, input2):\n batch_size = input1.shape[0]\n input1 = input1.reshape([batch_size, -1])\n input2 = input2.reshape([batch_size, -1])\n input1_l2 = input1\n input2_l2 = input2\n diff_loss = 0\n dim = input1.shape[1]\n for i in range(input1.shape[0]):\n diff_loss = diff_loss + paddle.mean(\n ((input1_l2[i:i + 1, :].mm(input2_l2[i:i + 1, :].T)).pow(2)) /\n dim)\n diff_loss = diff_loss / input1.shape[0]\n return diff_loss\nclass MSE(nn.Layer):\n def __init__(self):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:30-67"
+ },
+ "8045": {
+ "file_id": 593,
+ "content": "The code defines two classes: DiffLoss and MSE. DiffLoss calculates the loss between two inputs using L2 norm, while MSE calculates mean squared error loss for a single input. The function on lines 29-66 calculates gradients of disparity maps using image differences, applies exponential decay based on gradient values, and returns their average. This seems to be related to depth estimation or disparity prediction in computer vision tasks.",
+ "type": "comment"
+ },
+ "8046": {
+ "file_id": 593,
+ "content": " super(MSE, self).__init__()\n def forward(self, pred, real):\n diffs = paddle.add(real, -pred)\n n = paddle.numel(diffs)\n mse = paddle.sum(diffs.pow(2)) / n\n return mse\nclass SIMSE(nn.Layer):\n def __init__(self):\n super(SIMSE, self).__init__()\n def forward(self, pred, real):\n diffs = paddle.add(real, -pred)\n n = paddle.numel(diffs)\n simse = paddle.sum(diffs).pow(2) / (n**2)\n return simse\nclass SSIM(nn.Layer):\n \"\"\"Layer to compute the SSIM loss between a pair of images\n \"\"\"\n def __init__(self):\n super(SSIM, self).__init__()\n self.mu_x_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.mu_y_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.sig_x_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.sig_y_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.sig_xy_pool = nn.AvgPool2D(3, 1, exclusive=False)\n self.refl = nn.Pad2D(1, mode='reflect')\n self.C1 = 0.01**2\n self.C2 = 0.03**2",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:68-104"
+ },
+ "8047": {
+ "file_id": 593,
+ "content": "MSE class is a mean squared error loss function for PaddlePaddle, SIMSE class calculates the structured iterative mean squared error loss, and SSIM class computes the structural similarity index (SSIM) loss between a pair of images using various pooling operations and constants.",
+ "type": "comment"
+ },
+ "8048": {
+ "file_id": 593,
+ "content": " def forward(self, x, y):\n x = self.refl(x)\n y = self.refl(y)\n mu_x = self.mu_x_pool(x)\n mu_y = self.mu_y_pool(y)\n sigma_x = self.sig_x_pool(x**2) - mu_x**2\n sigma_y = self.sig_y_pool(y**2) - mu_y**2\n sigma_xy = self.sig_xy_pool(x * y) - mu_x * mu_y\n SSIM_n = (2 * mu_x * mu_y + self.C1) * (2 * sigma_xy + self.C2)\n SSIM_d = (mu_x**2 + mu_y**2 + self.C1) * (sigma_x + sigma_y + self.C2)\n return paddle.clip((1 - SSIM_n / SSIM_d) / 2, 0, 1)\n@LOSSES.register()\nclass ADDSLoss(BaseWeightedLoss):\n def __init__(self, avg_reprojection, disparity_smoothness, no_ssim):\n super(ADDSLoss, self).__init__()\n self.avg_reprojection = avg_reprojection\n self.disparity_smoothness = disparity_smoothness\n self.no_ssim = no_ssim\n self.loss_diff = DiffLoss()\n self.loss_recon1 = MSE()\n self.loss_recon2 = SIMSE()\n self.loss_similarity = MSE()\n def compute_reprojection_loss(self, pred, target):\n \"\"\"Computes reprojection loss between a batch of predicted and target images",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:106-137"
+ },
+ "8049": {
+ "file_id": 593,
+ "content": "This code defines a forward function for calculating the SSIM loss, which is used in the ADDSLoss class. The SSIM loss measures the structural similarity between two images and takes into account luminance (mu_x and mu_y) and contrast (sigma_x and sigma_y) for each image. It also considers the covariance of the two images (sigma_xy). The SSIM loss is then used in the ADDSLoss class to compute the reprojection loss between predicted and target images.",
+ "type": "comment"
+ },
+ "8050": {
+ "file_id": 593,
+ "content": " \"\"\"\n abs_diff = paddle.abs(target - pred)\n l1_loss = abs_diff.mean(1, True)\n if not self.no_ssim:\n self.ssim = SSIM()\n if self.no_ssim:\n reprojection_loss = l1_loss\n else:\n ssim_loss = self.ssim(pred, target).mean(1, True)\n reprojection_loss = 0.85 * ssim_loss + 0.15 * l1_loss\n return reprojection_loss\n def compute_losses(self, inputs, outputs, is_night):\n \"\"\"Compute the reprojection and smoothness losses for a minibatch\n \"\"\"\n losses = {}\n total_loss = 0\n for scale in outputs['scales']:\n loss = 0\n reprojection_losses = []\n source_scale = 0\n disp = outputs[(\"disp\", scale)]\n if is_night:\n color = inputs[(\"color_n\", 0, scale)]\n target = inputs[(\"color_n\", 0, source_scale)]\n else:\n color = inputs[(\"color\", 0, scale)]\n target = inputs[(\"color\", 0, source_scale)]\n for frame_id in outputs['frame_ids'][1:]:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:138-173"
+ },
+ "8051": {
+ "file_id": 593,
+ "content": "This code computes the reprojection and smoothness losses for a minibatch by iterating over different scales. It calculates the L1 loss between the predicted depth and the target depth, and optionally computes the SSIM (Structural Similarity Index) loss as well. The reprojection loss is determined based on these two values, with 85% weighted towards the SSIM loss and 15% towards the L1 loss. The total loss for the minibatch is accumulated in the \"total_loss\" variable.",
+ "type": "comment"
+ },
+ "8052": {
+ "file_id": 593,
+ "content": " pred = outputs[(\"color\", frame_id, scale)]\n reprojection_losses.append(\n self.compute_reprojection_loss(pred, target))\n reprojection_losses = paddle.concat(reprojection_losses, 1)\n identity_reprojection_losses = []\n for frame_id in outputs['frame_ids'][1:]:\n if is_night:\n pred = inputs[(\"color_n\", frame_id, source_scale)]\n else:\n pred = inputs[(\"color\", frame_id, source_scale)]\n identity_reprojection_losses.append(\n self.compute_reprojection_loss(pred, target))\n identity_reprojection_losses = paddle.concat(\n identity_reprojection_losses, 1)\n if self.avg_reprojection:\n identity_reprojection_loss = identity_reprojection_losses.mean(\n 1, keepdim=True)\n else:\n # save both images, and do min all at once below\n identity_reprojection_loss = identity_reprojection_losses",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:174-197"
+ },
+ "8053": {
+ "file_id": 593,
+ "content": "This code computes reprojection losses for day and night scenarios, concatenates them into a single tensor, and then checks if average reprojection loss should be computed. If not, it saves both images and performs minimum operation all at once.",
+ "type": "comment"
+ },
+ "8054": {
+ "file_id": 593,
+ "content": " if self.avg_reprojection:\n reprojection_loss = reprojection_losses.mean(1, keepdim=True)\n else:\n reprojection_loss = reprojection_losses\n # add random numbers to break ties\n identity_reprojection_loss = identity_reprojection_loss + paddle.randn(\n identity_reprojection_loss.shape) * 0.00001\n combined = paddle.concat(\n (identity_reprojection_loss, reprojection_loss), axis=1)\n if combined.shape[1] == 1:\n to_optimise = combined\n else:\n to_optimise = paddle.min(combined, axis=1)\n loss = loss + to_optimise.mean()\n mean_disp = disp.mean(2, True).mean(3, True)\n norm_disp = disp / (mean_disp + 1e-7)\n smooth_loss = get_smooth_loss(norm_disp, color)\n loss = loss + self.disparity_smoothness * smooth_loss / (2**scale)\n total_loss = total_loss + loss\n losses[\"loss/{}\".format(scale)] = loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:199-223"
+ },
+ "8055": {
+ "file_id": 593,
+ "content": "This code calculates the depth loss by combining identity and reprojection losses, adds random numbers to break ties, concatenates them, selects minimum values for optimization, and calculates disparity smoothness loss. It then updates the total loss and stores it in the losses dictionary.",
+ "type": "comment"
+ },
+ "8056": {
+ "file_id": 593,
+ "content": " total_loss /= len(outputs['scales'])\n losses[\"loss\"] = total_loss\n return losses\n def forward(self, inputs, outputs):\n losses_day = self.compute_losses(inputs, outputs, 'day')\n losses_night = self.compute_losses(inputs, outputs['outputs_night'],\n 'night')\n loss = 0\n losses = []\n # diff\n target_diff1 = 0.5 * self.loss_diff(\n outputs['result'][0], outputs['result'][2]) # 10 when batchsize=1\n target_diff2 = 0.5 * self.loss_diff(outputs['result_night'][0],\n outputs['result_night'][2])\n losses.append(target_diff1)\n losses.append(target_diff2)\n loss = loss + target_diff1\n loss = loss + target_diff2\n target_diff3 = 1 * self.loss_diff(\n outputs['result'][1], outputs['result'][3]) # 10 when batchsize=1\n target_diff4 = 1 * self.loss_diff(outputs['result_night'][1],\n outputs['result_night'][3])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:225-250"
+ },
+ "8057": {
+ "file_id": 593,
+ "content": "This code computes losses for both day and night scenes in a video, using the compute_losses function. It appends two target differences to the 'loss' list and adds them to the total loss. The target_diff1 and target_diff2 are calculated by the loss_diff function, comparing specific elements from the outputs. Target_diff3 and target_diff4 are also computed in a similar manner. The final total loss is divided by the number of scales and stored in the losses dictionary before returning.",
+ "type": "comment"
+ },
+ "8058": {
+ "file_id": 593,
+ "content": " losses.append(target_diff3)\n losses.append(target_diff4)\n loss = loss + target_diff3\n loss = loss + target_diff4\n # recon\n target_mse = 1 * self.loss_recon1(outputs['result'][5],\n inputs[\"color_aug\", 0, 0])\n loss = loss + target_mse\n target_simse = 1 * self.loss_recon2(outputs['result'][5],\n inputs[\"color_aug\", 0, 0])\n loss = loss + target_simse\n losses.append(target_mse)\n losses.append(target_simse)\n target_mse_night = 1 * self.loss_recon1(outputs['result_night'][5],\n inputs[\"color_n_aug\", 0, 0])\n loss = loss + target_mse_night\n target_simse_night = 1 * self.loss_recon2(outputs['result_night'][5],\n inputs[\"color_n_aug\", 0, 0])\n loss = loss + target_simse_night\n losses.append(target_mse_night)\n losses.append(target_simse_night)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:251-276"
+ },
+ "8059": {
+ "file_id": 593,
+ "content": "The code calculates multiple losses, including depth and reconstruction, for both daytime and night-time scenes. It then adds these losses to the total loss and appends them to the 'losses' list.",
+ "type": "comment"
+ },
+ "8060": {
+ "file_id": 593,
+ "content": " # depth loss\n pseudo_label = outputs[(\"disp\", 0)].detach()\n depth_loss = 1 * self.loss_similarity(\n outputs['outputs_night'][(\"disp\", 0)], pseudo_label)\n loss = loss + depth_loss\n losses.append(depth_loss)\n outputs['loss'] = loss + losses_day['loss'] + losses_night['loss']\n outputs['losses_day'] = losses_day['loss']\n outputs['losses_night'] = losses_night['loss']\n return outputs",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/depth_loss.py:278-290"
+ },
+ "8061": {
+ "file_id": 593,
+ "content": "This code calculates a depth loss by comparing predicted depths with detached pseudo-labels, then adds it to the overall loss and appends it to the losses list. Finally, it updates the output dictionary with the total loss and separate day/night losses before returning the updated outputs.",
+ "type": "comment"
+ },
+ "8062": {
+ "file_id": 594,
+ "content": "/paddlevideo/modeling/losses/distillation_loss.py",
+ "type": "filepath"
+ },
+ "8063": {
+ "file_id": 594,
+ "content": "This code defines Distillation Entropy Loss and KL divergence loss classes, implementing CrossEntropy loss for single/triple labels and KL divergence respectively, with optional weighted average and activation functions.",
+ "type": "summary"
+ },
+ "8064": {
+ "file_id": 594,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nimport paddle.nn as nn\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass DistillationCELoss(BaseWeightedLoss):\n \"\"\"Distillation Entropy Loss.\"\"\"\n def _forward(self, score, labels, **kwargs):\n \"\"\"Forward function.\n Args:\n score (paddle.Tensor): The class score.\n labels (paddle.Tensor): The ground truth labels.",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/distillation_loss.py:1-30"
+ },
+ "8065": {
+ "file_id": 594,
+ "content": "Defines a Distillation Entropy Loss class, which inherits from BaseWeightedLoss and takes score and labels as input for its forward function.",
+ "type": "comment"
+ },
+ "8066": {
+ "file_id": 594,
+ "content": " kwargs: Any keyword argument to be used to calculate\n CrossEntropy loss.\n Returns:\n loss (paddle.Tensor): The returned CrossEntropy loss.\n \"\"\"\n if len(labels) == 1:\n label = labels[0]\n loss = F.cross_entropy(score, label, **kwargs)\n # Deal with VideoMix\n elif len(labels) == 3:\n label_a, label_b, lam = labels\n loss_a = F.cross_entropy(score, label_a, **kwargs)\n loss_b = F.cross_entropy(score, label_b, **kwargs)\n loss = lam * loss_a + (1 - lam) * loss_b\n loss = paddle.mean(loss) #lam shape is bs\n return loss\n@LOSSES.register()\nclass DistillationDMLLoss(BaseWeightedLoss):\n \"\"\"\n DistillationDMLLoss\n \"\"\"\n def __init__(self, act=\"softmax\", eps=1e-12, **kargs):\n super().__init__(**kargs)\n if act is not None:\n assert act in [\"softmax\", \"sigmoid\"]\n if act == \"softmax\":\n self.act = nn.Softmax(axis=-1)\n elif act == \"sigmoid\":",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/distillation_loss.py:31-60"
+ },
+ "8067": {
+ "file_id": 594,
+ "content": "The code defines a loss function that calculates CrossEntropy loss and supports both single and triple labels. For single label, it directly calculates the CrossEntropy loss. For triple labels, it first calculates two separate CrossEntropy losses, then combines them with a weighted average based on a given lambda value (lam). The DistillationDMLLoss class implements this behavior and also handles the act parameter for specifying different activation functions.",
+ "type": "comment"
+ },
+ "8068": {
+ "file_id": 594,
+ "content": " self.act = nn.Sigmoid()\n else:\n self.act = None\n self.eps = eps\n def _kldiv(self, x, target):\n class_num = x.shape[-1]\n cost = target * paddle.log(\n (target + self.eps) / (x + self.eps)) * class_num\n return cost\n def _forward(self, x, target):\n if self.act is not None:\n x = self.act(x)\n target = self.act(target)\n loss = self._kldiv(x, target) + self._kldiv(target, x)\n loss = loss / 2\n loss = paddle.mean(loss)\n return loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/distillation_loss.py:61-79"
+ },
+ "8069": {
+ "file_id": 594,
+ "content": "This code defines a class for implementing the Kullback-Leibler (KL) divergence loss. The constructor takes an optional activation function and epsilon for numerical stability. The _kldiv method calculates the KL divergence between two vectors, while the _forward method applies the activation function if provided and computes the final loss by averaging the KL divergences in both directions.",
+ "type": "comment"
+ },
+ "8070": {
+ "file_id": 595,
+ "content": "/paddlevideo/modeling/losses/transnetv2_loss.py",
+ "type": "filepath"
+ },
+ "8071": {
+ "file_id": 595,
+ "content": "This code defines a class \"TransNetV2Loss\" for calculating TransNetV2 model loss with transition_weight and many-hot_loss_weight parameters, using weighted binary cross-entropy loss for one-hot and many-hot predictions. The snippet returns the total loss from TransNetV2 components.",
+ "type": "summary"
+ },
+ "8072": {
+ "file_id": 595,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass TransNetV2Loss(BaseWeightedLoss):\n \"\"\"Loss for TransNetV2 model\n \"\"\"\n def __init__(self, transition_weight=5.0, many_hot_loss_weight=0.1):\n self.transition_weight = transition_weight\n self.many_hot_loss_weight = many_hot_loss_weight\n super().__init__()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/transnetv2_loss.py:1-28"
+ },
+ "8073": {
+ "file_id": 595,
+ "content": "This code defines a class called \"TransNetV2Loss\" for calculating the loss in TransNetV2 model. It inherits from BaseWeightedLoss and takes transition_weight and many_hot_loss_weight as parameters for customizing the loss calculation.",
+ "type": "comment"
+ },
+ "8074": {
+ "file_id": 595,
+ "content": " def _forward(self, one_hot_pred, one_hot_gt,\n many_hot_pred=None, many_hot_gt=None, reg_losses=None):\n assert transition_weight != 1\n one_hot_pred = one_hot_pred[:, :, 0]\n one_hot_gt = one_hot_gt.astype('float32')\n one_hot_loss = F.binary_cross_entropy_with_logits(logit=one_hot_pred, label=one_hot_gt, reduction='none')\n one_hot_loss *= 1 + one_hot_gt * (transition_weight - 1)\n one_hot_loss = paddle.mean(one_hot_loss)\n many_hot_loss = 0.\n if many_hot_loss_weight != 0. and many_hot_pred is not None:\n many_hot_loss = many_hot_loss_weight * paddle.mean(\n F.binary_cross_entropy_with_logits(logit=many_hot_pred[:, :, 0],\n label=many_hot_gt.astype('float32'), reduction='none'))\n total_loss = one_hot_loss + many_hot_loss\n if reg_losses is not None:\n for name, value in reg_losses.items():\n if value is not None:\n total_loss += value",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/transnetv2_loss.py:30-54"
+ },
+ "8075": {
+ "file_id": 595,
+ "content": "This code defines a loss function for the TransNetV2 model, taking in one-hot and many-hot predictions and ground truth labels. It calculates the binary cross-entropy loss for both types of predictions, applies a weighted factor based on transition weight, and averages the losses before summing them together.",
+ "type": "comment"
+ },
+ "8076": {
+ "file_id": 595,
+ "content": " return total_loss",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/transnetv2_loss.py:56-56"
+ },
+ "8077": {
+ "file_id": 595,
+ "content": "This code snippet is returning the total loss computed from various components of the TransNetV2 model.",
+ "type": "comment"
+ },
+ "8078": {
+ "file_id": 596,
+ "content": "/paddlevideo/modeling/losses/yowo_loss.py",
+ "type": "filepath"
+ },
+ "8079": {
+ "file_id": 596,
+ "content": "FocalLoss optimizes hard examples in object detection, while YowoLoss and RegionLoss use softmax encoding. Code prepares input with reshaping, sigmoid activation, and anchor parameters. The code calculates YOLOv3-style losses for bounding box location, confidence, and classification on GPU.",
+ "type": "summary"
+ },
+ "8080": {
+ "file_id": 596,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy\nimport paddle\nimport paddle.nn.functional as F\nimport paddle.nn as nn\nfrom paddle.static import Variable\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\nfrom ..framework.localizers.yowo_utils import build_targets\nclass FocalLoss(nn.Layer):\n \"\"\"\n This criterion is a implemenation of Focal Loss, which is proposed in\n Focal Loss for Dense Object Detection.\n Loss(x, class) = - \\alpha (1-softmax(x)[class])^gamma \\log(softmax(x)[class])",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:1-31"
+ },
+ "8081": {
+ "file_id": 596,
+ "content": "This code snippet defines a FocalLoss class that implements the Focal Loss criterion. It is used for dense object detection and aims to reduce the classification loss for well-classified samples, focusing more on hard examples. The formula for the loss is given as -α(1-softmax(x)[class])^γ * log(softmax(x)[class]).",
+ "type": "comment"
+ },
+ "8082": {
+ "file_id": 596,
+ "content": " The losses are averaged across observations for each minibatch.\n Args:\n alpha(1D Tensor, Variable) : the scalar factor for this criterion\n gamma(float, double) : gamma > 0; reduces the relative loss for well-classified examples (p > .5),\n putting more focus on hard, misclassified examples\n size_average(bool): size_average(bool): By default, the losses are averaged over observations for each minibatch.\n However, if the field size_average is set to False, the losses are\n instead summed for each minibatch.\n \"\"\"\n def __init__(self, class_num, alpha=None, gamma=2, size_average=True):\n super(FocalLoss, self).__init__()\n if alpha is None:\n self.alpha = paddle.ones(\n [class_num, 1])\n self.alpha.stop_gradient = False\n else:\n if isinstance(alpha, Variable):\n self.alpha = alpha\n else:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:33-55"
+ },
+ "8083": {
+ "file_id": 596,
+ "content": "FocalLoss is a criterion that takes in alpha, gamma, and size_average as arguments. It averages losses across observations for each minibatch by default but can sum the losses if size_average is set to False. Alpha is either a tensor or variable, and gamma should be greater than 0, reducing relative loss for well-classified examples.",
+ "type": "comment"
+ },
+ "8084": {
+ "file_id": 596,
+ "content": " self.alpha = (alpha)\n self.alpha.stop_gradient = False\n self.gamma = gamma\n self.class_num = class_num\n self.size_average = size_average\n def forward(self, inputs, targets):\n N = inputs.shape[0]\n C = inputs.shape[1]\n P = F.softmax(inputs, axis=1)\n tmp = numpy.zeros((N, C))\n class_mask = paddle.to_tensor(tmp, place=inputs.place)\n class_mask.stop_gradient = False\n ids = paddle.reshape(targets, [-1, 1])\n class_mask = F.one_hot(ids.squeeze(-1), class_mask.shape[1])\n if \"Place\" not in str(inputs.place) and \"Place\" not in str(self.alpha.place):\n self.alpha = self.alpha.cuda()\n alpha = self.alpha[paddle.reshape(ids.detach(), [-1])]\n probs = paddle.reshape((P * class_mask).sum(1), [-1, 1])\n log_p = probs.log()\n batch_loss = -alpha * (paddle.pow((1 - probs), self.gamma)) * log_p\n if self.size_average:\n loss = batch_loss.mean()\n else:\n loss = batch_loss.sum()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:56-87"
+ },
+ "8085": {
+ "file_id": 596,
+ "content": "This code defines a class for the Yowo loss function. The constructor sets various attributes like alpha, gamma, class_num, size_average, and stop_gradient. The forward method calculates the loss using softmax, one-hot encoding, and other operations. If inputs or self.alpha are not in GPU, it transfers them to the GPU. It then computes the batch_loss and finally returns either average or sum depending on size_average attribute.",
+ "type": "comment"
+ },
+ "8086": {
+ "file_id": 596,
+ "content": " return loss\n@LOSSES.register()\nclass RegionLoss(BaseWeightedLoss):\n # for our model anchors has 10 values and number of anchors is 5\n # parameters: 24, 10 float values, 24, 5\n def __init__(self, num_classes, anchors, num_anchors, object_scale, noobject_scale, class_scale, coord_scale):\n super().__init__()\n self.num_classes = num_classes\n self.anchors = [float(x) for x in anchors]\n self.num_anchors = num_anchors\n self.anchor_step = len(self.anchors) // self.num_anchors # each anchor has 2 parameters\n self.object_scale = object_scale\n self.noobject_scale = noobject_scale\n self.class_scale = class_scale\n self.coord_scale = coord_scale\n self.focalloss = FocalLoss(class_num=self.num_classes, gamma=2, size_average=False)\n self.thresh = 0.6\n def convert2cpu(self, gpu_matrix):\n # return paddle.to_tensor((gpu_matrix.shape), dtype=\"float32\").copy_(gpu_matrix)\n return gpu_matrix.cpu()\n def forward(self, output, target):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:88-112"
+ },
+ "8087": {
+ "file_id": 596,
+ "content": "This code defines a RegionLoss class that inherits from BaseWeightedLoss. It takes parameters such as num_classes, anchors, num_anchors, object_scale, noobject_scale, class_scale, and coord_scale. The class initializes an instance of FocalLoss and sets a threshold. The forward method computes the loss between output and target tensors.",
+ "type": "comment"
+ },
+ "8088": {
+ "file_id": 596,
+ "content": " # output : B*A*(4+1+num_classes)*H*W 8*5*29*24*24\n # B: number of batches\n # A: number of anchors\n # 4: 4 parameters for each bounding box\n # 1: confidence score\n # num_classes\n # H: height of the image (in grids)\n # W: width of the image (in grids)\n # for each grid cell, there are A*(4+1+num_classes) parameters\n nB = output.detach().shape[0] # batch\n nA = self.num_anchors # anchor_num\n nC = self.num_classes\n nH = output.detach().shape[2]\n nW = output.detach().shape[3]\n # resize the output (all parameters for each anchor can be reached)\n output = paddle.reshape(output, [nB, nA, (5 + nC), nH, nW])\n # anchor's parameter tx\n x = F.sigmoid(\n paddle.reshape(paddle.index_select(output, paddle.to_tensor([0], dtype='int64').cuda(), axis=2),\n [nB, nA, nH, nW]))\n x.stop_gradient = False\n # anchor's parameter ty\n y = F.sigmoid(",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:113-137"
+ },
+ "8089": {
+ "file_id": 596,
+ "content": "This code reshapes the output tensor for each anchor's parameters and applies sigmoid activation to the transformed tensor. The output tensor is of shape B*A*(4+1+num_classes)*H*W, which represents the coordinates (tx, ty), width, height, confidence score, and class probabilities for each anchor box in the image grid. By applying sigmoid activation functions to tx and ty, the code scales the anchor's parameter values between 0 and 1, preparing them for the subsequent operations in the YOLOv4 model.",
+ "type": "comment"
+ },
+ "8090": {
+ "file_id": 596,
+ "content": " paddle.reshape(paddle.index_select(output, paddle.to_tensor([1], dtype='int64').cuda(), axis=2),\n [nB, nA, nH, nW]))\n y.stop_gradient = False\n # anchor's parameter tw\n w = paddle.reshape(paddle.index_select(output, paddle.to_tensor([2], dtype='int64').cuda(), axis=2),\n [nB, nA, nH, nW])\n w.stop_gradient = False\n # anchor's parameter th\n h = paddle.reshape(paddle.index_select(output, paddle.to_tensor([3], dtype='int64').cuda(), axis=2),\n [nB, nA, nH, nW])\n h.stop_gradient = False\n # confidence score for each anchor\n conf = F.sigmoid(\n paddle.reshape(paddle.index_select(output, paddle.to_tensor([4], dtype='int64').cuda(), axis=2),\n [nB, nA, nH, nW]))\n conf.stop_gradient = False\n # anchor's parameter class label\n cls = paddle.index_select(output, paddle.linspace(5, 5 + nC - 1, nC, 'int64').cuda(), axis=2)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:138-155"
+ },
+ "8091": {
+ "file_id": 596,
+ "content": "The code reshapes and assigns stop_gradient to output slices of the tensor \"output\" corresponding to anchor parameters (paddle, w, h) and a confidence score (conf), as well as class labels (cls). All are assigned stop_gradient=False.",
+ "type": "comment"
+ },
+ "8092": {
+ "file_id": 596,
+ "content": " cls.stop_gradient = False\n # resize the data structure so that for every anchor there is a class label in the last dimension\n cls = paddle.reshape(paddle.transpose(paddle.reshape(cls, [nB * nA, nC, nH * nW]), [0, 2, 1]),\n [nB * nA * nH * nW, nC])\n # for the prediction of localization of each bounding box, there exist 4 parameters (tx, ty, tw, th)\n # pred_boxes = torch.cuda.FloatTensor(4, nB*nA*nH*nW)\n pred_boxes = paddle.zeros([4, nB * nA * nH * nW], dtype='float32').cuda()\n # tx and ty\n grid_x = paddle.reshape(paddle.tile(paddle.tile(paddle.linspace(0, nW - 1, nW), [nH, 1]), [nB * nA, 1, 1]),\n [nB * nA * nH * nW]).cuda()\n grid_y = paddle.reshape(paddle.tile(paddle.tile(paddle.linspace(0, nH - 1, nH), [nW, 1]).t(), [nB * nA, 1, 1]),\n [nB * nA * nH * nW]).cuda()\n # for each anchor there are anchor_step variables (with the structure num_anchor*anchor_step)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:156-169"
+ },
+ "8093": {
+ "file_id": 596,
+ "content": "This code resizes the data structure to have a class label for each anchor, initializes prediction boxes, and creates grid coordinates for localization. It uses PaddlePaddle's linear algebra functions like paddle.reshape, paddle.transpose, and paddle.linspace. The code aims to prepare the input data for object detection model training.",
+ "type": "comment"
+ },
+ "8094": {
+ "file_id": 596,
+ "content": " # for each row(anchor), the first variable is anchor's width, second is anchor's height\n # pw and ph\n anchor_w = paddle.index_select(paddle.reshape(paddle.to_tensor(self.anchors), [nA, self.anchor_step]),\n paddle.to_tensor([0], dtype='int64'), axis=1).cuda()\n anchor_h = paddle.index_select(paddle.reshape(paddle.to_tensor(self.anchors), [nA, self.anchor_step]),\n paddle.to_tensor([1], dtype='int64'), axis=1).cuda()\n # for each pixel (grid) repeat the above process (obtain width and height of each grid)\n anchor_w = paddle.reshape(paddle.tile(paddle.tile(anchor_w, [nB, 1]), [1, 1, nH * nW]), [nB * nA * nH * nW])\n anchor_h = paddle.reshape(paddle.tile(paddle.tile(anchor_h, [nB, 1]), [1, 1, nH * nW]), [nB * nA * nH * nW])\n # prediction of bounding box localization\n # x.data and y.data: top left corner of the anchor\n # grid_x, grid_y: tx and ty predictions made by yowo",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:170-181"
+ },
+ "8095": {
+ "file_id": 596,
+ "content": "This code is preparing anchor width and height values for the YOWO loss function. It reshapes the anchors, index selects the width and height values, tiles them to match grid dimensions, and assigns the prediction of bounding box localization for each grid cell.",
+ "type": "comment"
+ },
+ "8096": {
+ "file_id": 596,
+ "content": " x_data = paddle.reshape(x.detach(), [-1])\n y_data = paddle.reshape(y.detach(), [-1])\n w_data = paddle.reshape(w.detach(), [-1])\n h_data = paddle.reshape(h.detach(), [-1])\n pred_boxes[0] = paddle.cast(x_data, dtype='float32') + paddle.cast(grid_x, dtype='float32') # bx\n pred_boxes[1] = paddle.cast(y_data, dtype='float32') + paddle.cast(grid_y, dtype='float32') # by\n pred_boxes[2] = paddle.exp(paddle.cast(w_data, dtype='float32')) * paddle.cast(anchor_w, dtype='float32') # bw\n pred_boxes[3] = paddle.exp(paddle.cast(h_data, dtype='float32')) * paddle.cast(anchor_h, dtype='float32') # bh\n # the size -1 is inferred from other dimensions\n # pred_boxes (nB*nA*nH*nW, 4)\n pred_boxes = self.convert2cpu(\n paddle.cast(paddle.reshape(paddle.transpose(pred_boxes, (1, 0)), [-1, 4]), dtype='float32'))\n nGT, nCorrect, coord_mask, conf_mask, cls_mask, tx, ty, tw, th, tconf, tcls = build_targets(pred_boxes,\n ",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:183-199"
+ },
+ "8097": {
+ "file_id": 596,
+ "content": "This code reshapes and casts input tensors, calculates predicted bounding box coordinates based on input features, and calls a function to build targets for the model. It then reshapes and transposes the predicted boxes tensor before passing it to the build_targets function. The function is part of the YOLOv3-style loss calculation in PaddleVideo.",
+ "type": "comment"
+ },
+ "8098": {
+ "file_id": 596,
+ "content": " target.detach(),\n self.anchors, nA,\n nC, \\\n nH, nW,\n self.noobject_scale,\n self.object_scale,\n self.thresh)\n cls_mask = (cls_mask == 1)\n # keep those with high box confidence scores (greater than 0.25) as our final predictions\n nProposals = int((conf > 0.25).sum().detach().item())\n tx = (tx).cuda()",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:199-210"
+ },
+ "8099": {
+ "file_id": 596,
+ "content": "This code is setting up a loss function for object detection. It takes in target values, anchors, number of anchors (nA), number of classes (nC), and the image dimensions (nH, nW). The noobject_scale and object_scale variables control how the loss is applied depending on whether an object is present or not. The cls_mask variable filters out proposals with low box confidence scores. The final predictions are kept if their confidence score is greater than 0.25. The tensor tx is moved to the GPU (cuda).",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/81.json b/docs/data/81.json
new file mode 100644
index 000000000..9ccda8778
--- /dev/null
+++ b/docs/data/81.json
@@ -0,0 +1,542 @@
+{
+ "8100": {
+ "file_id": 596,
+ "content": " tx.stop_gradient = False\n ty = ty.cuda()\n ty.stop_gradient = False\n tw = tw.cuda()\n tw.stop_gradient = False\n th = th.cuda()\n th.stop_gradient = False\n tconf = tconf.cuda()\n tconf.stop_gradient = False\n tcls = paddle.reshape(tcls, [-1]).astype('int64')[paddle.reshape(cls_mask, [-1])].cuda()\n tcls.stop_gradient = False\n coord_mask = coord_mask.cuda()\n coord_mask.stop_gradient = False\n conf_mask = conf_mask.cuda().sqrt()\n coord_mask.stop_gradient = False\n cls_mask = paddle.tile(paddle.reshape(cls_mask, [-1, 1]), [1, nC]).cuda()\n cls_mask.stop_gradient = False\n cls = paddle.reshape(cls[cls_mask], [-1, nC])\n # losses between predictions and targets (ground truth)\n # In total 6 aspects are considered as losses:\n # 4 for bounding box location, 2 for prediction confidence and classification seperately\n L1_loss = nn.SmoothL1Loss(reduction='sum')\n loss_x = self.coord_scale * L1_loss(paddle.cast(x, dtype=\"float32\") * coord_mask, tx * coord_mask) / 2.0",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:211-237"
+ },
+ "8101": {
+ "file_id": 596,
+ "content": "This code is moving variables to the GPU and setting their gradient flags to False. Then, it calculates losses for bounding box location, prediction confidence, and classification separately using SmoothL1Loss.",
+ "type": "comment"
+ },
+ "8102": {
+ "file_id": 596,
+ "content": " loss_y = self.coord_scale * L1_loss(paddle.cast(y, dtype=\"float32\") * coord_mask, ty * coord_mask) / 2.0\n loss_w = self.coord_scale * L1_loss(paddle.cast(w * coord_mask, dtype=\"float32\"), tw * coord_mask) / 2.0\n loss_h = self.coord_scale * L1_loss(paddle.cast(h * coord_mask, dtype=\"float32\"), th * coord_mask) / 2.0\n loss_conf = nn.MSELoss(reduction='sum')(paddle.cast(conf, dtype=\"float32\") * conf_mask, tconf * conf_mask) / 2.0\n # try focal loss with gamma = 2\n loss_cls = self.class_scale * self.focalloss(cls, tcls)\n # sum of loss\n loss = loss_x + loss_y + loss_w + loss_h + loss_conf + loss_cls\n return loss, nCorrect",
+ "type": "code",
+ "location": "/paddlevideo/modeling/losses/yowo_loss.py:238-249"
+ },
+ "8103": {
+ "file_id": 596,
+ "content": "This code calculates the loss for an object detection model, consisting of L1_loss for coordinates (x, y, w, h) and MSELoss for confidence. It applies focal loss for classification with a gamma value of 2, sums all losses together, and returns the total loss and count of correct predictions.",
+ "type": "comment"
+ },
+ "8104": {
+ "file_id": 597,
+ "content": "/paddlevideo/modeling/registry.py",
+ "type": "filepath"
+ },
+ "8105": {
+ "file_id": 597,
+ "content": "This code registers various model types (backbones, heads, recognizers) using a Registry class for efficient organization and management in a larger model architecture or framework implementation. Registries are created for 'bbox_coder', 'estimator', 'multimodal', and 'segment'.",
+ "type": "summary"
+ },
+ "8106": {
+ "file_id": 597,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..utils import Registry\nBACKBONES = Registry('backbone')\nHEADS = Registry('head')\nRECOGNIZERS = Registry('recognizer')\nSEGMENTERS = Registry('Segmenters')\nLOCALIZERS = Registry('localizer')\nPARTITIONERS = Registry('partitioner')\nLOSSES = Registry('loss')\nROI_EXTRACTORS = Registry('roi_extractor')\nDETECTORS = Registry('detectors')\nBBOX_ASSIGNERS = Registry('bbox_assigner')\nBBOX_SAMPLERS = Registry('bbox_sampler')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/registry.py:1-27"
+ },
+ "8107": {
+ "file_id": 597,
+ "content": "This code is registering different types of models (backbones, heads, recognizers, etc.) using a Registry class from the utils module. The Registry will help in organizing and managing these different model types efficiently. This code snippet seems to be part of a larger model architecture or framework implementation.",
+ "type": "comment"
+ },
+ "8108": {
+ "file_id": 597,
+ "content": "BBOX_CODERS = Registry('bbox_coder')\nESTIMATORS = Registry('estimator')\nMULTIMODAL = Registry('multimodal')\nSEGMENT = Registry('segment')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/registry.py:28-31"
+ },
+ "8109": {
+ "file_id": 597,
+ "content": "Registry is created for 'bbox_coder', 'estimator', 'multimodal', and 'segment'. These Registries organize and manage the different types of models or coding methods, allowing for easy access and maintenance.",
+ "type": "comment"
+ },
+ "8110": {
+ "file_id": 598,
+ "content": "/paddlevideo/modeling/samplers/__init__.py",
+ "type": "filepath"
+ },
+ "8111": {
+ "file_id": 598,
+ "content": "This code is licensing information and imports the RandomSampler class from a submodule, then defines the __all__ variable to include only the RandomSampler class.",
+ "type": "summary"
+ },
+ "8112": {
+ "file_id": 598,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .random_sampler import RandomSampler\n__all__ = ['RandomSampler']",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/__init__.py:1-17"
+ },
+ "8113": {
+ "file_id": 598,
+ "content": "This code is licensing information and imports the RandomSampler class from a submodule, then defines the __all__ variable to include only the RandomSampler class.",
+ "type": "comment"
+ },
+ "8114": {
+ "file_id": 599,
+ "content": "/paddlevideo/modeling/samplers/random_sampler.py",
+ "type": "filepath"
+ },
+ "8115": {
+ "file_id": 599,
+ "content": "The code initializes a SamplingResult class for bbox sampling and defines a RandomSampler class to sample positive and negative bboxes from assigned results, ensuring enough samples are available in each case.",
+ "type": "summary"
+ },
+ "8116": {
+ "file_id": 599,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport numpy as np\nfrom ..registry import BBOX_SAMPLERS\nclass SamplingResult():\n \"\"\"Bbox sampling result. \"\"\"\n def __init__(self, pos_inds, neg_inds, bboxes, gt_bboxes, assign_result,\n gt_flags):\n self.pos_inds = pos_inds\n self.neg_inds = neg_inds\n self.pos_bboxes = paddle.index_select(bboxes,pos_inds)\n # neg_inds may be empty\n if neg_inds.shape[0]!=0:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/random_sampler.py:1-28"
+ },
+ "8117": {
+ "file_id": 599,
+ "content": "This code is importing necessary libraries and defining a class called \"SamplingResult\" that holds the result of bbox sampling. The class has attributes for positive indices, negative indices, bboxes, gt_bboxes, assign_result, and gt_flags. It uses paddle library to index select the bboxes based on pos_inds. Negative_inds may be empty.",
+ "type": "comment"
+ },
+ "8118": {
+ "file_id": 599,
+ "content": " self.neg_bboxes = paddle.index_select(bboxes,neg_inds)\n else:\n self.neg_bboxes=None\n self.pos_is_gt = paddle.index_select(gt_flags,pos_inds)\n self.num_gts = gt_bboxes.shape[0]\n self.pos_assigned_gt_inds = paddle.index_select(assign_result.gt_inds,pos_inds) - 1\n if float(gt_bboxes.numel()) == 0:\n assert self.pos_assigned_gt_inds.numel() == 0\n self.pos_gt_bboxes = paddle.empty_like(gt_bboxes).view(-1, 4)\n else:\n if len(gt_bboxes.shape) < 2:\n gt_bboxes = gt_bboxes.view(-1, 4)\n self.pos_gt_bboxes = paddle.index_select(gt_bboxes, self.pos_assigned_gt_inds)\n if assign_result.labels is not None:\n self.pos_gt_labels = paddle.index_select(assign_result.labels, pos_inds)\n else:\n self.pos_gt_labels = None\n @property\n def bboxes(self):\n if self.neg_bboxes is not None:\n ret = paddle.concat([self.pos_bboxes, self.neg_bboxes])\n else:",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/random_sampler.py:29-55"
+ },
+ "8119": {
+ "file_id": 599,
+ "content": "This code initializes the negative bounding boxes, positive ground truth (gt) bounding boxes and labels for a sampler. It checks if there are any gt bboxes available, if not, it sets up a placeholder for them. The 'pos_bboxes' are then concatenated with the neg_bboxes if they exist. If assign_result.labels is not None, it also extracts and stores positive gt labels.",
+ "type": "comment"
+ },
+ "8120": {
+ "file_id": 599,
+ "content": " # neg bbox may be empty\n ret = self.pos_bboxes\n return ret\n@BBOX_SAMPLERS.register()\nclass RandomSampler():\n def __init__(self,\n num,\n pos_fraction,\n neg_pos_ub=-1,\n add_gt_as_proposals=True,\n **kwargs):\n self.num = num\n self.pos_fraction = pos_fraction\n self.neg_pos_ub = neg_pos_ub\n self.add_gt_as_proposals = add_gt_as_proposals\n def sample(self,\n assign_result,\n bboxes,\n gt_bboxes,\n gt_labels=None,\n **kwargs):\n \"\"\"Sample positive and negative bboxes. \"\"\"\n if len(bboxes.shape) < 2:\n bboxes = bboxes[None, :]\n bboxes = bboxes[:, :4]\n gt_flags = paddle.full([bboxes.shape[0], ], 0, dtype='int32')\n if self.add_gt_as_proposals and len(gt_bboxes) > 0:\n if gt_labels is None:\n raise ValueError(\n 'gt_labels must be given when add_gt_as_proposals is True')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/random_sampler.py:56-92"
+ },
+ "8121": {
+ "file_id": 599,
+ "content": "This code defines a RandomSampler class which samples positive and negative bboxes from assigned results. It takes arguments like num, pos_fraction, neg_pos_ub, add_gt_as_proposals, etc. If add_gt_as_proposals is True and gt_bboxes are present, it raises a ValueError if gt_labels are not given. The sample method takes assign_result, bboxes, gt_bboxes, and gt_labels as arguments. It checks the shape of bboxes, converts them to 4-column format, and creates gt_flags.",
+ "type": "comment"
+ },
+ "8122": {
+ "file_id": 599,
+ "content": " bboxes = paddle.concat([gt_bboxes, bboxes])\n assign_result.add_gt_(gt_labels)\n gt_ones = paddle.full([gt_bboxes.shape[0], ], 1, dtype='int32')\n gt_flags = paddle.concat([gt_ones, gt_flags])\n #1. 得到正样本的数量, inds\n num_expected_pos = int(self.num * self.pos_fraction)\n pos_inds = self._sample_pos( assign_result, num_expected_pos, bboxes=bboxes, **kwargs)\n pos_inds = paddle.to_tensor(np.unique(pos_inds.numpy()))\n #2. 得到负样本的数量, inds\n num_sampled_pos = pos_inds.numel()\n num_expected_neg = self.num - num_sampled_pos\n neg_inds = self._sample_neg(\n assign_result, num_expected_neg, bboxes=bboxes, **kwargs)\n neg_inds = paddle.to_tensor(np.unique(neg_inds.numpy()))\n #3. 得到sampling result\n sampling_result = SamplingResult(pos_inds, neg_inds, bboxes, gt_bboxes,\n assign_result, gt_flags)\n return sampling_result\n def random_choice(self, gallery, num):",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/random_sampler.py:93-114"
+ },
+ "8123": {
+ "file_id": 599,
+ "content": "This code samples positive and negative indices for assigning ground truth labels to objects, ensuring a desired ratio of positive and negative samples. It then creates a SamplingResult object containing these indices along with bounding boxes and other information. The random_choice function is used to randomly select a specific number of samples from a given set of objects.",
+ "type": "comment"
+ },
+ "8124": {
+ "file_id": 599,
+ "content": " \"\"\"Random select some elements from the gallery. \"\"\"\n assert len(gallery) >= num\n perm = paddle.arange(gallery.numel())[:num]\n perm = paddle.randperm(gallery.numel())[:num] \n rand_inds = paddle.index_select(gallery, perm)\n return rand_inds\n def _sample_pos(self, assign_result, num_expected, **kwargs):\n \"\"\"Randomly sample some positive samples.\"\"\"\n #1.首先看一下给的bboxes里面有哪些label是大于0的 得到了他们的index\n pos_inds = paddle.nonzero(assign_result.gt_inds, as_tuple=False)\n #2. 只要这个pos_inds的数目不是0个 这些就都可以是positive sample\n # 当pos_inds的数目小于num_expected(想要的sample的最大数目), 就直接用这个pos_inds\n # 反之就从这么多index里随机采样num_expected个出来\n if float(pos_inds.numel()) != 0:\n pos_inds = pos_inds.squeeze() \n if float(pos_inds.numel()) <= num_expected:\n return pos_inds\n else:\n return self.random_choice(pos_inds, num_expected)\n def _sample_neg(self, assign_result, num_expected, **kwargs):\n \"\"\"Randomly sample some negative samples.\"\"\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/random_sampler.py:115-139"
+ },
+ "8125": {
+ "file_id": 599,
+ "content": "The code defines a random sampler that randomly selects elements from the gallery. It has two functions: _sample_pos, which randomly samples positive samples, and _sample_neg, which randomly samples negative samples. The _sample_pos function first finds indexes of assign_result with label greater than 0 (i.e., positive samples), then checks if the number of positive samples is less than or equal to num_expected. If it's less, returns the indices; otherwise, selects num_expected random samples from the available indices using the random_choice method. The _sample_neg function does a similar process for negative samples but doesn't return the indices if their number is 0.",
+ "type": "comment"
+ },
+ "8126": {
+ "file_id": 599,
+ "content": " neg_inds = paddle.nonzero(assign_result.gt_inds == 0, as_tuple=False)\n if float(neg_inds.numel()) != 0:\n neg_inds = neg_inds.squeeze() \n if (float(neg_inds.numel())) <= float(num_expected):\n return neg_inds\n else:\n return self.random_choice(neg_inds, num_expected)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/samplers/random_sampler.py:140-146"
+ },
+ "8127": {
+ "file_id": 599,
+ "content": "This code checks the assign_result's gt_inds for zero values, extracts their indices in neg_inds, and if there are non-zero values, squeezes them. If the number of non-zero values is less than or equal to expected, it returns neg_inds. Otherwise, it uses random_choice() to select required indices from neg_inds.",
+ "type": "comment"
+ },
+ "8128": {
+ "file_id": 600,
+ "content": "/paddlevideo/modeling/weight_init.py",
+ "type": "filepath"
+ },
+ "8129": {
+ "file_id": 600,
+ "content": "This code initializes layer weights in PaddlePaddle, applying truncated normal or other initializations like Gaussian and Kaiming uniform. It adjusts for different modes and supports numpy arrays and Paddle tensors.",
+ "type": "summary"
+ },
+ "8130": {
+ "file_id": 600,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn.initializer as init\nimport numpy as np\nfrom scipy import special\ndef weight_init_(layer,\n func,\n weight_name=None,\n bias_name=None,\n bias_value=0.0,\n **kwargs):\n \"\"\"\n In-place params init function.\n Usage:\n .. code-block:: python\n import paddle\n import numpy as np\n data = np.ones([3, 4], dtype='float32')",
+ "type": "code",
+ "location": "/paddlevideo/modeling/weight_init.py:1-36"
+ },
+ "8131": {
+ "file_id": 600,
+ "content": "This code defines a function that initializes the weights of a PaddlePaddle layer using specified functions. It can also set bias values and is compatible with numpy arrays and Paddle tensors.",
+ "type": "comment"
+ },
+ "8132": {
+ "file_id": 600,
+ "content": " linear = paddle.nn.Linear(4, 4)\n input = paddle.to_tensor(data)\n print(linear.weight)\n linear(input)\n weight_init_(linear, 'Normal', 'fc_w0', 'fc_b0', std=0.01, mean=0.1)\n print(linear.weight)\n \"\"\"\n if hasattr(layer, 'weight') and layer.weight is not None:\n getattr(init, func)(**kwargs)(layer.weight)\n if weight_name is not None:\n # override weight name\n layer.weight.name = weight_name\n if hasattr(layer, 'bias') and layer.bias is not None:\n init.Constant(bias_value)(layer.bias)\n if bias_name is not None:\n # override bias name\n layer.bias.name = bias_name\ndef _no_grad_trunc_normal_(tensor, mean, std, a, b):\n def norm_cdf(x):\n # Computes standard normal cumulative distribution function\n return (1. + math.erf(x / math.sqrt(2.))) / 2.\n if (mean < a - 2 * std) or (mean > b + 2 * std):\n print(\"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. \"\n \"The distribution of values may be incorrect.\")",
+ "type": "code",
+ "location": "/paddlevideo/modeling/weight_init.py:37-66"
+ },
+ "8133": {
+ "file_id": 600,
+ "content": "Code initializes a Linear layer, applies truncated normal initialization to its weights with specified mean and std deviation, and optionally changes the weight name. If the layer has bias, it initializes the bias with a constant value and optionally changes the bias name. The _no_grad_trunc_normal_ function is used internally by nn.init.trunc_normal_.",
+ "type": "comment"
+ },
+ "8134": {
+ "file_id": 600,
+ "content": " with paddle.no_grad():\n # Values are generated by using a truncated uniform distribution and\n # then using the inverse CDF for the normal distribution.\n # Get upper and lower cdf values\n l = norm_cdf((a - mean) / std)\n u = norm_cdf((b - mean) / std)\n # Uniformly fill tensor with values from [l, u], then translate to [2l-1, 2u-1].\n tmp = np.random.uniform(2 * l - 1, 2 * u - 1,\n size=list(tensor.shape)).astype(np.float32)\n # Use inverse cdf transform for normal distribution to get truncated\n # standard normal\n tmp = special.erfinv(tmp)\n # Transform to proper mean, std\n tmp *= (std * math.sqrt(2.0))\n tmp += mean\n # Clamp to ensure it's in the proper range\n tmp = np.clip(tmp, a, b)\n tensor.set_value(paddle.to_tensor(tmp))\n return tensor\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = tensor.dim()\n if dimensions < 2:\n raise ValueError(\n \"Fan in and fan out can not be computed for tensor with fewer than 2 dimensions\"",
+ "type": "code",
+ "location": "/paddlevideo/modeling/weight_init.py:68-98"
+ },
+ "8135": {
+ "file_id": 600,
+ "content": "This code generates weights for a tensor following a truncated Gaussian distribution. It computes the lower and upper bounds, uniformly fills the tensor with values between these bounds, transforms them to a standard Gaussian distribution, adjusts the mean and standard deviation, clamps the values within the original bounds, and sets the tensor's value.",
+ "type": "comment"
+ },
+ "8136": {
+ "file_id": 600,
+ "content": " )\n num_input_fmaps = tensor.shape[1]\n num_output_fmaps = tensor.shape[0]\n receptive_field_size = 1\n if tensor.dim() > 2:\n receptive_field_size = tensor[0][0].numel()\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef trunc_normal_(tensor, mean=0., std=1., a=-2., b=2.):\n return _no_grad_trunc_normal_(tensor, mean, std, a, b)\ndef kaiming_normal_(tensor, a=0., mode='fan_in', nonlinearity='leaky_relu'):\n def _calculate_correct_fan(tensor, mode):\n mode = mode.lower()\n valid_modes = ['fan_in', 'fan_out']\n if mode not in valid_modes:\n raise ValueError(\n \"Mode {} not supported, please use one of {}\".format(\n mode, valid_modes))\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n return fan_in if mode == 'fan_in' else fan_out\n def calculate_gain(nonlinearity, param=None):\n linear_fns = [\n 'linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d',",
+ "type": "code",
+ "location": "/paddlevideo/modeling/weight_init.py:99-130"
+ },
+ "8137": {
+ "file_id": 600,
+ "content": "This code initializes weights in a convolutional layer using either truncated normal or Kaiming uniform initialization. It calculates the fan-in and fan-out based on input and output feature maps, receptive field size, and optionally adjusts for different modes. The `trunc_normal_` function generates random values within specific bounds using truncated normal distribution, while `kaiming_normal_` sets weights using Kaiming uniform initialization with an optional nonlinearity parameter.",
+ "type": "comment"
+ },
+ "8138": {
+ "file_id": 600,
+ "content": " 'conv_transpose2d', 'conv_transpose3d'\n ]\n if nonlinearity in linear_fns or nonlinearity == 'sigmoid':\n return 1\n elif nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if param is None:\n negative_slope = 0.01\n elif not isinstance(param, bool) and isinstance(\n param, int) or isinstance(param, float):\n negative_slope = param\n else:\n raise ValueError(\n \"negative_slope {} not a valid number\".format(param))\n return math.sqrt(2.0 / (1 + negative_slope**2))\n else:\n raise ValueError(\"Unsupported nonlinearity {}\".format(nonlinearity))\n fan = _calculate_correct_fan(tensor, mode)\n gain = calculate_gain(nonlinearity, a)\n std = gain / math.sqrt(fan)\n with paddle.no_grad():\n paddle.nn.initializer.Normal(0, std)(tensor)",
+ "type": "code",
+ "location": "/paddlevideo/modeling/weight_init.py:131-156"
+ },
+ "8139": {
+ "file_id": 600,
+ "content": "This function initializes the weights of a neural network layer with respect to the nonlinearity used. It returns different values depending on the nonlinearity type, calculates the fan for each layer and then applies normal initialization using Paddle's Normal initializer.",
+ "type": "comment"
+ },
+ "8140": {
+ "file_id": 600,
+ "content": " return tensor",
+ "type": "code",
+ "location": "/paddlevideo/modeling/weight_init.py:157-157"
+ },
+ "8141": {
+ "file_id": 600,
+ "content": "Initializes a tensor with specified values and returns it.",
+ "type": "comment"
+ },
+ "8142": {
+ "file_id": 601,
+ "content": "/paddlevideo/solver/__init__.py",
+ "type": "filepath"
+ },
+ "8143": {
+ "file_id": 601,
+ "content": "This code snippet appears to import the \"build_optimizer\" and \"build_lr\" functions from their respective modules within the \"paddlevideo.solver\" package. The comments at the top of the file indicate that this code is protected by copyright and licensed under the Apache License, Version 2.0.",
+ "type": "summary"
+ },
+ "8144": {
+ "file_id": 601,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .optimizer import build_optimizer\nfrom .lr import build_lr",
+ "type": "code",
+ "location": "/paddlevideo/solver/__init__.py:1-16"
+ },
+ "8145": {
+ "file_id": 601,
+ "content": "This code snippet appears to import the \"build_optimizer\" and \"build_lr\" functions from their respective modules within the \"paddlevideo.solver\" package. The comments at the top of the file indicate that this code is protected by copyright and licensed under the Apache License, Version 2.0.",
+ "type": "comment"
+ },
+ "8146": {
+ "file_id": 602,
+ "content": "/paddlevideo/solver/custom_lr.py",
+ "type": "filepath"
+ },
+ "8147": {
+ "file_id": 602,
+ "content": "The code introduces CustomWarmupCosineDecay and CustomWarmupPiecewiseDecay schedulers for PaddleVideo, combining warm-up, cosine decay, and piecewise decay. The CustomWarmupAdjustDecay scheduler combines warmup and cosine decay and adjusts based on epoch number.",
+ "type": "summary"
+ },
+ "8148": {
+ "file_id": 602,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nfrom paddle.optimizer.lr import *\nimport numpy as np\n\"\"\"\nPaddleVideo Learning Rate Schedule:\nYou can use paddle.optimizer.lr\nor define your custom_lr in this file.\n\"\"\"\nclass CustomWarmupCosineDecay(LRScheduler):\n r\"\"\"\n We combine warmup and stepwise-cosine which is used in slowfast model.\n Args:\n warmup_start_lr (float): start learning rate used in warmup stage.\n warmup_epochs (int): the number epochs of warmup.",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:1-31"
+ },
+ "8149": {
+ "file_id": 602,
+ "content": "This code defines a custom learning rate scheduler, CustomWarmupCosineDecay, which combines warmup and stepwise-cosine decay for use in PaddleVideo. It is part of the PaddlePaddle framework and can be used to adjust learning rates during training.",
+ "type": "comment"
+ },
+ "8150": {
+ "file_id": 602,
+ "content": " cosine_base_lr (float|int, optional): base learning rate in cosine schedule.\n max_epoch (int): total training epochs.\n num_iters(int): number iterations of each epoch.\n last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.\n verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .\n Returns:\n ``CosineAnnealingDecay`` instance to schedule learning rate.\n \"\"\"\n def __init__(self,\n warmup_start_lr,\n warmup_epochs,\n cosine_base_lr,\n max_epoch,\n num_iters,\n last_epoch=-1,\n verbose=False):\n self.warmup_start_lr = warmup_start_lr\n self.warmup_epochs = warmup_epochs\n self.cosine_base_lr = cosine_base_lr\n self.max_epoch = max_epoch\n self.num_iters = num_iters\n #call step() in base class, last_lr/last_epoch/base_lr will be update",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:32-54"
+ },
+ "8151": {
+ "file_id": 602,
+ "content": "This code defines a class `CosineAnnealingDecay` that schedules the learning rate for training. It takes parameters like warmup start lr, warmup epochs, cosine base lr, max epoch, num_iters, last_epoch (optional), and verbose (optional). The class initializes these parameters and provides a `step()` method to update the learning rate based on cosine annealing schedule. If verbose is set to True, it will print messages for each update.",
+ "type": "comment"
+ },
+ "8152": {
+ "file_id": 602,
+ "content": " super(CustomWarmupCosineDecay, self).__init__(last_epoch=last_epoch,\n verbose=verbose)\n def step(self, epoch=None):\n \"\"\"\n ``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .\n The new learning rate will take effect on next ``optimizer.step`` .\n Args:\n epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.\n Returns:\n None\n \"\"\"\n if epoch is None:\n if self.last_epoch == -1:\n self.last_epoch += 1\n else:\n self.last_epoch += 1 / self.num_iters # update step with iters\n else:\n self.last_epoch = epoch\n self.last_lr = self.get_lr()\n if self.verbose:\n print('Epoch {}: {} set learning rate to {}.'.format(\n self.last_epoch, self.__class__.__name__, self.last_lr))\n def _lr_func_cosine(self, cur_epoch, cosine_base_lr, max_epoch):",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:55-80"
+ },
+ "8153": {
+ "file_id": 602,
+ "content": "This code defines a custom learning rate scheduler for PaddleVideo, implementing the CustomWarmupCosineDecay class. The step() method updates the learning rate based on current epoch and calls get_lr() to set the new learning rate. The _lr_func_cosine() function calculates the learning rate using a cosine annealing schedule.",
+ "type": "comment"
+ },
+ "8154": {
+ "file_id": 602,
+ "content": " return cosine_base_lr * (math.cos(math.pi * cur_epoch / max_epoch) +\n 1.0) * 0.5\n def get_lr(self):\n \"\"\"Define lr policy\"\"\"\n lr = self._lr_func_cosine(self.last_epoch, self.cosine_base_lr,\n self.max_epoch)\n lr_end = self._lr_func_cosine(self.warmup_epochs, self.cosine_base_lr,\n self.max_epoch)\n # Perform warm up.\n if self.last_epoch < self.warmup_epochs:\n lr_start = self.warmup_start_lr\n alpha = (lr_end - lr_start) / self.warmup_epochs\n lr = self.last_epoch * alpha + lr_start\n return lr\nclass CustomWarmupPiecewiseDecay(LRScheduler):\n r\"\"\"\n This op combine warmup and stepwise-cosine which is used in slowfast model.\n Args:\n warmup_start_lr (float): start learning rate used in warmup stage.\n warmup_epochs (int): the number epochs of warmup.\n step_base_lr (float|int, optional): base learning rate in step schedule.",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:81-106"
+ },
+ "8155": {
+ "file_id": 602,
+ "content": "The code defines a custom learning rate (LR) scheduler that includes warmup and stepwise-cosine decay. It first performs a warmup stage with a linear increase in LR from the warmup_start_lr to lr_end over warmup_epochs, then applies cosine annealing for the rest of the epochs, resulting in a learning rate that decreases from the base value according to the cosine function.",
+ "type": "comment"
+ },
+ "8156": {
+ "file_id": 602,
+ "content": " max_epoch (int): total training epochs.\n num_iters(int): number iterations of each epoch.\n last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.\n verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .\n Returns:\n ``CustomWarmupPiecewiseDecay`` instance to schedule learning rate.\n \"\"\"\n def __init__(self,\n warmup_start_lr,\n warmup_epochs,\n step_base_lr,\n lrs,\n gamma,\n steps,\n max_epoch,\n num_iters,\n last_epoch=0,\n verbose=False):\n self.warmup_start_lr = warmup_start_lr\n self.warmup_epochs = warmup_epochs\n self.step_base_lr = step_base_lr\n self.lrs = lrs\n self.gamma = gamma\n self.steps = steps\n self.max_epoch = max_epoch\n self.num_iters = num_iters",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:107-133"
+ },
+ "8157": {
+ "file_id": 602,
+ "content": "This code defines a class `CustomWarmupPiecewiseDecay` which schedules learning rate for training. The class takes parameters like warmup_start_lr, warmup_epochs, step_base_lr, lrs, gamma, steps, max_epoch, num_iters, last_epoch, and verbose. It initializes these parameters in the constructor (__init__). The learning rate is scheduled to decay over time following a piecewise function with warm-up and custom decays.",
+ "type": "comment"
+ },
+ "8158": {
+ "file_id": 602,
+ "content": " self.last_epoch = last_epoch\n self.last_lr = self.warmup_start_lr # used in first iter\n self.verbose = verbose\n self._var_name = None\n def step(self, epoch=None, rebuild=False):\n \"\"\"\n ``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .\n The new learning rate will take effect on next ``optimizer.step`` .\n Args:\n epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.\n Returns:\n None\n \"\"\"\n if epoch is None:\n if not rebuild:\n self.last_epoch += 1 / self.num_iters # update step with iters\n else:\n self.last_epoch = epoch\n self.last_lr = self.get_lr()\n if self.verbose:\n print(\n 'step Epoch {}: {} set learning rate to {}.self.num_iters={}, 1/self.num_iters={}'\n .format(self.last_epoch, self.__class__.__name__, self.last_lr,",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:134-158"
+ },
+ "8159": {
+ "file_id": 602,
+ "content": "This code defines a custom learning rate scheduler for the PaddleVideo library. The `step` method updates the learning rate based on the current epoch and returns None. It should be called after `optimizer.step`. If no epoch is specified, it increments the last epoch by the number of iterations divided by the total number of iterations. The last learning rate is stored in `self.last_lr`, and if verbose is set to True, it prints the current epoch, scheduler name, and updated learning rate.",
+ "type": "comment"
+ },
+ "8160": {
+ "file_id": 602,
+ "content": " self.num_iters, 1 / self.num_iters))\n def _lr_func_steps_with_relative_lrs(self, cur_epoch, lrs, base_lr, steps,\n max_epoch):\n # get step index\n steps = steps + [max_epoch]\n for ind, step in enumerate(steps):\n if cur_epoch < step:\n break\n if self.verbose:\n print(\n '_lr_func_steps_with_relative_lrs, cur_epoch {}: {}, steps {}, ind {}, step{}, max_epoch{}'\n .format(cur_epoch, self.__class__.__name__, steps, ind, step,\n max_epoch))\n return lrs[ind - 1] * base_lr\n def get_lr(self):\n \"\"\"Define lr policy\"\"\"\n lr = self._lr_func_steps_with_relative_lrs(\n self.last_epoch,\n self.lrs,\n self.step_base_lr,\n self.steps,\n self.max_epoch,\n )\n lr_end = self._lr_func_steps_with_relative_lrs(\n self.warmup_epochs,\n self.lrs,\n self.step_base_lr,",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:159-188"
+ },
+ "8161": {
+ "file_id": 602,
+ "content": "This function defines a learning rate (lr) policy that varies based on the current epoch, predefined learning rates, base lr, steps, and maximum epoch. It calculates the learning rate for each step using a relative learning rate function and returns it. The function also includes a warmup phase where the learning rate gradually increases from 0 to its final value over the specified number of epochs.",
+ "type": "comment"
+ },
+ "8162": {
+ "file_id": 602,
+ "content": " self.steps,\n self.max_epoch,\n )\n # Perform warm up.\n if self.last_epoch < self.warmup_epochs:\n lr_start = self.warmup_start_lr\n alpha = (lr_end - lr_start) / self.warmup_epochs\n lr = self.last_epoch * alpha + lr_start\n if self.verbose:\n print(\n 'get_lr, Epoch {}: {}, lr {}, lr_end {}, self.lrs{}, self.step_base_lr{}, self.steps{}, self.max_epoch{}'\n .format(self.last_epoch, self.__class__.__name__, lr, lr_end,\n self.lrs, self.step_base_lr, self.steps,\n self.max_epoch))\n return lr\nclass CustomPiecewiseDecay(PiecewiseDecay):\n def __init__(self, **kargs):\n kargs.pop('num_iters')\n super().__init__(**kargs)\nclass CustomWarmupCosineStepDecay(LRScheduler):\n def __init__(self,\n warmup_iters,\n warmup_ratio=0.1,\n min_lr=0,\n base_lr=3e-5,\n max_epoch=30,",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:189-222"
+ },
+ "8163": {
+ "file_id": 602,
+ "content": "This code implements a CustomWarmupCosineStepDecay learning rate scheduler, which performs warm up and then applies piecewise decay. The learning rate is determined based on the current epoch, warmup epochs, warmup start and end rates, and the number of steps. A CustomPiecewiseDecay class is also defined, which inherits from PiecewiseDecay and overrides the num_iters parameter.",
+ "type": "comment"
+ },
+ "8164": {
+ "file_id": 602,
+ "content": " last_epoch=-1,\n num_iters=None,\n verbose=False):\n self.warmup_ratio = warmup_ratio\n self.min_lr = min_lr\n self.warmup_epochs = warmup_iters\n self.warmup_iters = warmup_iters * num_iters\n self.cnt_iters = 0\n self.cnt_epoch = 0\n self.num_iters = num_iters\n self.tot_iters = max_epoch * num_iters\n self.max_epoch = max_epoch\n self.cosine_base_lr = base_lr # initial lr for all param groups\n self.regular_lr = self.get_regular_lr()\n super().__init__(last_epoch=last_epoch, verbose=verbose)\n def annealing_cos(self, start, end, factor, weight=1):\n cos_out = math.cos(math.pi * factor) + 1\n return end + 0.5 * weight * (start - end) * cos_out\n def get_regular_lr(self):\n progress = self.cnt_epoch\n max_progress = self.max_epoch\n target_lr = self.min_lr\n return self.annealing_cos(self.cosine_base_lr, target_lr, progress /\n max_progress) # self.cosine_base_lr",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:223-249"
+ },
+ "8165": {
+ "file_id": 602,
+ "content": "This function initializes the custom learning rate scheduler. It sets warmup ratio, minimum learning rate, and warmup iterations. The total number of iterations, maximum epochs, base learning rate for cosine annealing, and a regular learning rate are calculated. The function also defines a helper method 'annealing_cos' for cosine annealing.",
+ "type": "comment"
+ },
+ "8166": {
+ "file_id": 602,
+ "content": " def get_warmup_lr(self, cur_iters):\n k = (1 - cur_iters / self.warmup_iters) * (1 - self.warmup_ratio)\n warmup_lr = self.regular_lr * (1 - k) # 3e-5 * (1-k)\n return warmup_lr\n def step(self, epoch=None):\n self.regular_lr = self.get_regular_lr()\n self.last_lr = self.get_lr()\n self.cnt_epoch = (self.cnt_iters +\n 1) // self.num_iters # update step with iters\n self.cnt_iters += 1\n if self.verbose:\n print('Epoch {}: {} set learning rate to {}.'.format(\n self.last_epoch, self.__class__.__name__, self.last_lr))\n def get_lr(self):\n \"\"\"Define lr policy\"\"\"\n cur_iter = self.cnt_iters\n if cur_iter >= self.warmup_iters:\n return self.regular_lr\n else:\n warmup_lr = self.get_warmup_lr(cur_iter)\n return warmup_lr\nclass CustomWarmupAdjustDecay(LRScheduler):\n r\"\"\"\n We combine warmup and stepwise-cosine which is used in slowfast model.\n Args:\n step_base_lr (float): start learning rate used in warmup stage.",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:251-282"
+ },
+ "8167": {
+ "file_id": 602,
+ "content": "This code defines a custom learning rate scheduler that combines warmup and stepwise-cosine decay. The get_warmup_lr function calculates the warmup learning rate, while the get_lr function determines whether the current iteration is in the warmup stage or not, returning either the regular learning rate or the warmed-up learning rate. The step function updates the learning rate and counts the number of iterations.",
+ "type": "comment"
+ },
+ "8168": {
+ "file_id": 602,
+ "content": " warmup_epochs (int): the number epochs of warmup.\n lr_decay_rate (float|int, optional): base learning rate decay rate.\n step (int): step in change learning rate.\n last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.\n verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .\n Returns:\n ``CosineAnnealingDecay`` instance to schedule learning rate.\n \"\"\"\n def __init__(self,\n step_base_lr,\n warmup_epochs,\n lr_decay_rate,\n boundaries,\n num_iters=None,\n last_epoch=-1,\n verbose=False):\n self.step_base_lr = step_base_lr\n self.warmup_epochs = warmup_epochs\n self.lr_decay_rate = lr_decay_rate\n self.boundaries = boundaries\n self.num_iters = num_iters\n #call step() in base class, last_lr/last_epoch/base_lr will be update",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:283-305"
+ },
+ "8169": {
+ "file_id": 602,
+ "content": "Custom learning rate scheduler with warmup, decay, and boundary steps. Initializes the LR scheduler with step base LR, warmup epochs, decay rate, boundaries, number of iterations (optional), last epoch (optional), and verbosity level (optional).",
+ "type": "comment"
+ },
+ "8170": {
+ "file_id": 602,
+ "content": " super(CustomWarmupAdjustDecay, self).__init__(last_epoch=last_epoch,\n verbose=verbose)\n def step(self, epoch=None):\n \"\"\"\n ``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .\n The new learning rate will take effect on next ``optimizer.step`` .\n Args:\n epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.\n Returns:\n None\n \"\"\"\n if epoch is None:\n if self.last_epoch == -1:\n self.last_epoch += 1\n else:\n self.last_epoch += 1 / self.num_iters # update step with iters\n else:\n self.last_epoch = epoch\n self.last_lr = self.get_lr()\n if self.verbose:\n print('Epoch {}: {} set learning rate to {}.'.format(\n self.last_epoch, self.__class__.__name__, self.last_lr))\n def get_lr(self):",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:306-332"
+ },
+ "8171": {
+ "file_id": 602,
+ "content": "The code defines a custom learning rate scheduler, CustomWarmupAdjustDecay, which adjusts the learning rate based on epoch number. It initializes the scheduler and provides a step method for updating the learning rate after optimizer.step is called. The get_lr method returns the current learning rate. The last_epoch variable keeps track of the current epoch. If no epoch is specified, it auto-increments from the last_epoch value. If an epoch is provided, the last_epoch is set to that value. Finally, if verbose is True, it prints the current epoch and the learning rate set.",
+ "type": "comment"
+ },
+ "8172": {
+ "file_id": 602,
+ "content": " if self.last_epoch < self.warmup_epochs:\n lr = self.step_base_lr * (self.last_epoch + 1) / self.warmup_epochs\n else:\n lr = self.step_base_lr * (self.lr_decay_rate**np.sum(\n self.last_epoch >= np.array(self.boundaries)))\n return lr",
+ "type": "code",
+ "location": "/paddlevideo/solver/custom_lr.py:333-338"
+ },
+ "8173": {
+ "file_id": 602,
+ "content": "This code calculates the learning rate based on whether the current epoch is within the warmup phase or not. If in warmup, it linearly increases the base learning rate. Otherwise, it applies a decay rate to determine the learning rate.",
+ "type": "comment"
+ },
+ "8174": {
+ "file_id": 603,
+ "content": "/paddlevideo/solver/lr.py",
+ "type": "filepath"
+ },
+ "8175": {
+ "file_id": 603,
+ "content": "This code constructs a learning rate scheduler based on the 'OPTIMIZER' configuration provided, returns it with specified iterations, and handles custom cases such as converting 'learning_rate' to a custom object.",
+ "type": "summary"
+ },
+ "8176": {
+ "file_id": 603,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom typing import Dict\nfrom paddle.optimizer.lr import LRScheduler\nfrom . import custom_lr\ndef build_lr(cfg: Dict, num_iters: int) -> LRScheduler:\n \"\"\"Build a learning rate scheduler accroding to ```OPTIMIZER``` configuration, and it always pass into the optimizer.\n In configuration:\n learning_rate:\n name: 'PiecewiseDecay'\n boundaries: [20, 60]\n values: [0.00025, 0.000025, 0.0000025]",
+ "type": "code",
+ "location": "/paddlevideo/solver/lr.py:1-28"
+ },
+ "8177": {
+ "file_id": 603,
+ "content": "This code is building a learning rate scheduler according to the \"OPTIMIZER\" configuration provided in the cfg dictionary. The scheduler is based on the 'PiecewiseDecay' name, and has boundaries and values for adjusting the learning rate at specified iterations.",
+ "type": "comment"
+ },
+ "8178": {
+ "file_id": 603,
+ "content": " Args:\n cfg (Dict): learning rate configuration.\n num_iters (int): The number of iterations that may be used when calculating the learning rate\n Returns:\n LRScheduler: learning rate scheduler.\n \"\"\"\n cfg_copy = cfg.copy()\n #when learning_rate is LRScheduler\n if cfg_copy.get('learning_rate') and isinstance(cfg_copy['learning_rate'],\n dict):\n cfg_copy['learning_rate'] = build_lr(\n cfg_copy['learning_rate'],\n num_iters) #not support only inner iter_step\n lr_name = cfg_copy.pop('name')\n if cfg_copy.get('iter_step'):\n cfg_copy['num_iters'] = num_iters\n cfg_copy.pop('iter_step')\n return getattr(custom_lr, lr_name)(**cfg_copy)",
+ "type": "code",
+ "location": "/paddlevideo/solver/lr.py:30-52"
+ },
+ "8179": {
+ "file_id": 603,
+ "content": "This function takes a learning rate configuration and the number of iterations, and returns a learning rate scheduler. If the configuration includes a 'learning_rate' key with a dictionary value, it converts it to a custom learning rate object using the build_lr() function. It also handles cases where 'iter_step' is present in the configuration, replacing it with 'num_iters'. The returned scheduler is obtained from the 'custom_lr' module with the specified 'name'.",
+ "type": "comment"
+ },
+ "8180": {
+ "file_id": 604,
+ "content": "/paddlevideo/solver/optimizer.py",
+ "type": "filepath"
+ },
+ "8181": {
+ "file_id": 604,
+ "content": "This code initializes optimizer configurations, handles weight decay, grad clip, and excludes parameters for L2 decay. It sets learning rate with LRScheduler, supports multi-precision, and creates an optimizer based on inputs.",
+ "type": "summary"
+ },
+ "8182": {
+ "file_id": 604,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport inspect\n# for python3.11\nif not hasattr(inspect, 'getargspec'):\n inspect.getargspec = inspect.getfullargspec\nfrom typing import Dict\nimport paddle\nfrom paddle.optimizer.lr import LRScheduler\nfrom paddle.regularizer import L1Decay, L2Decay\nfrom paddlevideo.utils import get_logger\ndef build_optimizer(cfg: Dict,\n lr_scheduler: LRScheduler,\n model: paddle.nn.Layer,\n use_amp: bool = False,",
+ "type": "code",
+ "location": "/paddlevideo/solver/optimizer.py:1-31"
+ },
+ "8183": {
+ "file_id": 604,
+ "content": "This code is from the \"optimizer.py\" file in the PaddleVideo library, and it's responsible for building an optimizer. It imports necessary modules, checks compatibility with Python versions, defines a function build_optimizer that takes parameters such as configuration (cfg), learning rate scheduler (lr_scheduler), model, and optional AMP usage (use_amp). This file also includes some license information and comments.",
+ "type": "comment"
+ },
+ "8184": {
+ "file_id": 604,
+ "content": " amp_level: str = None) -> paddle.optimizer.Optimizer:\n \"\"\"Build an optimizer and learning rate scheduler to optimize parameters accroding to ```OPTIMIZER``` field in configuration.\n In configuration:\n OPTIMIZER:\n name: Momentum\n momentum: 0.9\n weight_decay: 0.001\n or\n OPTIMIZER:\n name: Momentum\n momentum: 0.9\n weight_decay:\n name: \"L1\"\n value: 0.001\n Momentum optimizer will be applied to optimize network and L1Decay regularizer will be applied to avoid overfit.\n OPTIMIZER:\n name: Adam\n weight_decay:\n name: \"L2\"\n value: 0.001\n Adam optimizer will be applied to optimize network and L2Decay regularizer will applied to avoid overfit.\n Refer to ```https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/regularizer/L2Decay_en.html``` for more details.\n Args:\n cfg (Dict): optimizer configuration.\n lr_scheduler (LRScheduler): learning rate scheduler.",
+ "type": "code",
+ "location": "/paddlevideo/solver/optimizer.py:32-63"
+ },
+ "8185": {
+ "file_id": 604,
+ "content": "Builds an optimizer and learning rate scheduler according to the OPTIMIZER field in the configuration. The Momentum or Adam optimizers are applied to optimize the network, and L1Decay or L2Decay regularizers are used to avoid overfitting. The function takes optimizer configuration (cfg) and learning rate scheduler (lr_scheduler) as arguments.",
+ "type": "comment"
+ },
+ "8186": {
+ "file_id": 604,
+ "content": " model (paddle.nn.Layer, optional): model which contains parameters to be optimized. Defaults to None.\n use_amp (bool, optional): Whether use amp. Defaults to False.\n amp_level (str, optional): amp level when amp is enabled. Defaults to None.\n Returns:\n paddle.optimizer.Optimizer: an optimizer for the input model.\n \"\"\"\n logger = get_logger(\"paddlevideo\")\n cfg_copy = cfg.copy()\n # NOTE: check none and illegal cfg!!!\n opt_name = cfg_copy.pop('name')\n # deal with weight decay\n if cfg_copy.get('weight_decay'):\n if isinstance(cfg_copy.get('weight_decay'),\n float): # just an float factor\n cfg_copy['weight_decay'] = cfg_copy.get('weight_decay')\n elif 'L1' in cfg_copy.get('weight_decay').get(\n 'name').upper(): # specify L2 wd and it's float factor\n cfg_copy['weight_decay'] = L1Decay(\n cfg_copy.get('weight_decay').get('value'))\n elif 'L2' in cfg_copy.get('weight_decay').get(",
+ "type": "code",
+ "location": "/paddlevideo/solver/optimizer.py:64-85"
+ },
+ "8187": {
+ "file_id": 604,
+ "content": "This code defines a function that creates an optimizer for a given model. It accepts parameters such as the model, whether to use AMP or not, and the AMP level. The function also handles weight decay by checking if a 'weight_decay' configuration is present and applying the appropriate settings (L1 or L2 decay).",
+ "type": "comment"
+ },
+ "8188": {
+ "file_id": 604,
+ "content": " 'name').upper(): # specify L1 wd and it's float factor\n cfg_copy['weight_decay'] = L2Decay(\n cfg_copy.get('weight_decay').get('value'))\n else:\n raise ValueError\n # deal with grad clip\n if cfg_copy.get('grad_clip'):\n if isinstance(cfg_copy.get('grad_clip'), float):\n cfg_copy['grad_clip'] = cfg_copy.get('grad_clip').get('value')\n elif 'global' in cfg_copy.get('grad_clip').get('name').lower():\n cfg_copy['grad_clip'] = paddle.nn.ClipGradByGlobalNorm(\n cfg_copy.get('grad_clip').get('value'))\n else:\n raise ValueError\n # Set for optimizers that cannot be applied to l2decay, i.e. AdamW\n if cfg_copy.get('no_weight_decay_name'):\n no_weight_decay_name = cfg_copy.pop('no_weight_decay_name')\n no_weight_decay_name_list = no_weight_decay_name.split(' ')\n # NOTE: use param.name not name\n no_weight_decay_param_list = [\n param.name for name, param in model.named_parameters()",
+ "type": "code",
+ "location": "/paddlevideo/solver/optimizer.py:86-109"
+ },
+ "8189": {
+ "file_id": 604,
+ "content": "This code is initializing the configuration for an optimizer, handling L1 and L2 weight decay, grad clip, and no_weight_decay parameters. If 'name' is specified for L1 wd, it sets the 'weight_decay' to the float factor. For grad clip, if a float value is given, it is set as the 'grad_clip', or if 'global' in name, creates a ClipGradByGlobalNorm object. If 'no_weight_decay_name' is specified, it extracts the list of parameters to exclude from L2 decay.",
+ "type": "comment"
+ },
+ "8190": {
+ "file_id": 604,
+ "content": " if any(key_word in name for key_word in no_weight_decay_name_list)\n ] # get the full param name of no weight decay\n _apply_decay_param_fun = lambda name: name not in no_weight_decay_param_list\n cfg_copy['apply_decay_param_fun'] = _apply_decay_param_fun\n logger.info(\n f\"No weight Decay list :({len(no_weight_decay_param_list)})\",\n no_weight_decay_param_list)\n cfg_copy.pop('learning_rate')\n # set multi_precision\n optimizer_setting = {\n 'learning_rate': lr_scheduler,\n 'parameters': model.parameters(),\n **cfg_copy\n }\n optimizer_init_args = inspect.getargspec(\n getattr(paddle.optimizer, opt_name).__init__).args\n if use_amp and amp_level == \"O2\" and \"multi_precision\" in optimizer_init_args:\n # support \"multi_precision\" arg in optimizer's __init__ function.\n optimizer_setting.update({\"multi_precision\": True})\n logger.info(\n \"Set multi_precision=True for optimizer when use_amp=True and amp_level='O2'\"",
+ "type": "code",
+ "location": "/paddlevideo/solver/optimizer.py:110-133"
+ },
+ "8191": {
+ "file_id": 604,
+ "content": "This code checks if there are any parameters without weight decay, and sets the learning rate using a LRScheduler. It also handles multi-precision for optimizer when use_amp is True and amp_level is 'O2'. The code updates the optimizer_setting with no_weight_decay_param_list and \"multi_precision\" if required, logging relevant information throughout.",
+ "type": "comment"
+ },
+ "8192": {
+ "file_id": 604,
+ "content": " )\n return getattr(paddle.optimizer, opt_name)(**optimizer_setting)",
+ "type": "code",
+ "location": "/paddlevideo/solver/optimizer.py:134-136"
+ },
+ "8193": {
+ "file_id": 604,
+ "content": "This code is creating and returning an optimizer based on the given \"opt_name\" and \"optimizer_setting\". The optimizer type is determined by using \"paddle.optimizer[opt_name]\" and the parameters are passed through **optimizer_settings** to initialize the optimizer object.",
+ "type": "comment"
+ },
+ "8194": {
+ "file_id": 605,
+ "content": "/paddlevideo/tasks/__init__.py",
+ "type": "filepath"
+ },
+ "8195": {
+ "file_id": 605,
+ "content": "This code is the initialization file for PaddleVideo's tasks module. It imports functions from various task-specific modules and adds them to `__all__` for export. The code also includes a license notice, copyright information, and a disclaimer.",
+ "type": "summary"
+ },
+ "8196": {
+ "file_id": 605,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .train import train_model\nfrom .test import test_model\nfrom .train_dali import train_dali\nfrom .train_multigrid import train_model_multigrid\n__all__ = ['train_model', 'test_model', 'train_dali', 'train_model_multigrid']",
+ "type": "code",
+ "location": "/paddlevideo/tasks/__init__.py:1-20"
+ },
+ "8197": {
+ "file_id": 605,
+ "content": "This code is the initialization file for PaddleVideo's tasks module. It imports functions from various task-specific modules and adds them to `__all__` for export. The code also includes a license notice, copyright information, and a disclaimer.",
+ "type": "comment"
+ },
+ "8198": {
+ "file_id": 606,
+ "content": "/paddlevideo/tasks/test.py",
+ "type": "filepath"
+ },
+ "8199": {
+ "file_id": 606,
+ "content": "This Python function uses PaddlePaddle to test models, enabling parallel processing and logging. It initializes the device, constructs model, dataset, and dataloader in test mode with adjustable parameters. The code builds a dataloader, loads state_dicts, sets up metrics, and iterates over batches for output or metric updates before accumulating the final result.",
+ "type": "summary"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/82.json b/docs/data/82.json
new file mode 100644
index 000000000..b7431f937
--- /dev/null
+++ b/docs/data/82.json
@@ -0,0 +1,546 @@
+{
+ "8200": {
+ "file_id": 606,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddlevideo.utils import get_logger, load\nfrom ..loader.builder import build_dataloader, build_dataset\nfrom ..metrics import build_metric\nfrom ..modeling.builder import build_model\nlogger = get_logger(\"paddlevideo\")\n@paddle.no_grad()\ndef test_model(cfg, weights, parallel=True):\n \"\"\"Test model entry\n Args:\n cfg (dict): configuration.\n weights (str): weights path to load.\n parallel (bool): Whether to do multi-cards testing. Default: True.",
+ "type": "code",
+ "location": "/paddlevideo/tasks/test.py:1-32"
+ },
+ "8201": {
+ "file_id": 606,
+ "content": "The code is a Python function for testing a model using PaddlePaddle framework. It takes configuration (cfg) and weights path (weights) as inputs, and allows for parallel processing. The logger captures any log messages from the function execution.",
+ "type": "comment"
+ },
+ "8202": {
+ "file_id": 606,
+ "content": " \"\"\"\n if cfg.get('use_npu', False):\n places = paddle.set_device('npu')\n elif cfg.get('use_xpu', False):\n places = paddle.set_device('xpu')\n else:\n places = paddle.set_device('gpu')\n # 1. Construct model.\n if cfg.MODEL.get('backbone') and cfg.MODEL.backbone.get('pretrained'):\n cfg.MODEL.backbone.pretrained = '' # disable pretrain model init\n model = build_model(cfg.MODEL)\n if parallel:\n model = paddle.DataParallel(model)\n # 2. Construct dataset and dataloader.\n cfg.DATASET.test.test_mode = True\n dataset = build_dataset((cfg.DATASET.test, cfg.PIPELINE.test))\n batch_size = cfg.DATASET.get(\"test_batch_size\", 8)\n # default num worker: 0, which means no subprocess will be created\n num_workers = cfg.DATASET.get('num_workers', 0)\n num_workers = cfg.DATASET.get('test_num_workers', num_workers)\n dataloader_setting = dict(batch_size=batch_size,\n num_workers=num_workers,\n places=places,",
+ "type": "code",
+ "location": "/paddlevideo/tasks/test.py:34-61"
+ },
+ "8203": {
+ "file_id": 606,
+ "content": "This code block initializes the model's device, constructs and configures the model, dataset, and dataloader. It also sets test mode and adjusts batch size and number of workers.",
+ "type": "comment"
+ },
+ "8204": {
+ "file_id": 606,
+ "content": " drop_last=False,\n shuffle=False)\n data_loader = build_dataloader(\n dataset, **dataloader_setting) if cfg.model_name not in ['CFBI'\n ] else dataset\n model.eval()\n state_dicts = load(weights)\n model.set_state_dict(state_dicts)\n # add params to metrics\n cfg.METRIC.data_size = len(dataset)\n cfg.METRIC.batch_size = batch_size\n Metric = build_metric(cfg.METRIC)\n if cfg.MODEL.framework == \"FastRCNN\":\n Metric.set_dataset_info(dataset.info, len(dataset))\n for batch_id, data in enumerate(data_loader):\n if cfg.model_name in [\n 'CFBI'\n ]: # for VOS task, dataset for video and dataloader for frames in each video\n Metric.update(batch_id, data, model)\n else:\n outputs = model(data, mode='test')\n Metric.update(batch_id, data, outputs)\n Metric.accumulate()",
+ "type": "code",
+ "location": "/paddlevideo/tasks/test.py:62-90"
+ },
+ "8205": {
+ "file_id": 606,
+ "content": "The code builds a dataloader for the dataset, loads state_dicts into the model, and sets up metrics. It then iterates over batches of data from the dataloader to either update the metric directly or get outputs from the model before updating the metric. After processing all batches, it accumulates the final result in the metric.",
+ "type": "comment"
+ },
+ "8206": {
+ "file_id": 607,
+ "content": "/paddlevideo/tasks/train.py",
+ "type": "filepath"
+ },
+ "8207": {
+ "file_id": 607,
+ "content": "The code utilizes PaddlePaddle's Fleet API for distributed training, defines models/metrics, and uses AMP to speed up gradient descent via DataParallel. It logs performance data, evaluates using PaddleVideo, saves the best model/optimizer, and periodically saves state during training.",
+ "type": "summary"
+ },
+ "8208": {
+ "file_id": 607,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport time\nimport paddle\nimport paddle.amp as amp\nimport paddle.distributed as dist\nimport paddle.distributed.fleet as fleet\nfrom paddlevideo.utils import (add_profiler_step, build_record, get_logger,\n load, log_batch, log_epoch, mkdir, save)\nfrom ..loader.builder import build_dataloader, build_dataset\nfrom ..metrics.ava_utils import collect_results_cpu\nfrom ..modeling.builder import build_model",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:1-27"
+ },
+ "8209": {
+ "file_id": 607,
+ "content": "The code imports necessary libraries, defines functions to build data loaders, datasets, models, and metrics using a builder pattern. It also includes functions for logging progress and saving results. The code is licensed under the Apache License 2.0, and it might be part of a larger framework or application dealing with video analysis tasks.",
+ "type": "comment"
+ },
+ "8210": {
+ "file_id": 607,
+ "content": "from ..solver import build_lr, build_optimizer\nfrom ..utils import do_preciseBN\ndef train_model(cfg,\n weights=None,\n parallel=True,\n validate=True,\n use_amp=False,\n amp_level=None,\n max_iters=None,\n use_fleet=False,\n profiler_options=None):\n \"\"\"Train model entry\n Args:\n cfg (dict): configuration.\n weights (str, optional): weights path for finetuning. Defaults to None.\n parallel (bool, optional): whether multi-cards training. Defaults to True.\n validate (bool, optional): whether to do evaluation. Defaults to True.\n use_amp (bool, optional): whether to use automatic mixed precision during training. Defaults to False.\n amp_level (str, optional): amp optmization level, must be 'O1' or 'O2' when use_amp is True. Defaults to None.\n max_iters (int, optional): max running iters in an epoch. Defaults to None.\n use_fleet (bool, optional): whether to use fleet. Defaults to False.",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:28-51"
+ },
+ "8211": {
+ "file_id": 607,
+ "content": "The code defines a train_model function for training the model using given configuration (cfg). It takes optional arguments like weights path, parallel training flag, validation enablement, automatic mixed precision usage, and more.",
+ "type": "comment"
+ },
+ "8212": {
+ "file_id": 607,
+ "content": " profiler_options (str, optional): configuration for the profiler function. Defaults to None.\n \"\"\"\n if use_fleet:\n fleet.init(is_collective=True)\n logger = get_logger(\"paddlevideo\")\n batch_size = cfg.DATASET.get('batch_size', 8)\n valid_batch_size = cfg.DATASET.get('valid_batch_size', batch_size)\n # gradient accumulation settings\n use_gradient_accumulation = cfg.get('GRADIENT_ACCUMULATION', None)\n if use_gradient_accumulation and dist.get_world_size() >= 1:\n global_batch_size = cfg.GRADIENT_ACCUMULATION.get(\n 'global_batch_size', None)\n num_gpus = dist.get_world_size()\n assert isinstance(\n global_batch_size, int\n ), f\"global_batch_size must be int, but got {type(global_batch_size)}\"\n assert batch_size <= global_batch_size, \\\n f\"global_batch_size({global_batch_size}) must not be less than batch_size({batch_size})\"\n cur_global_batch_size = batch_size * num_gpus # The number of batches calculated by all GPUs at one time",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:52-75"
+ },
+ "8213": {
+ "file_id": 607,
+ "content": "This code sets up gradient accumulation and global batch size for distributed training using PaddlePaddle's Fleet API. It retrieves batch and validation batch sizes from the configuration, then checks if gradient accumulation is enabled and the world size of the distributed setup. If so, it calculates the global batch size based on these settings and asserts that global_batch_size is greater than the current batch size.",
+ "type": "comment"
+ },
+ "8214": {
+ "file_id": 607,
+ "content": " assert global_batch_size % cur_global_batch_size == 0, \\\n f\"The global batchsize({global_batch_size}) must be divisible by cur_global_batch_size({cur_global_batch_size})\"\n cfg.GRADIENT_ACCUMULATION[\n \"num_iters\"] = global_batch_size // cur_global_batch_size\n # The number of iterations required to reach the global batchsize\n logger.info(\n f\"Using gradient accumulation training strategy, \"\n f\"global_batch_size={global_batch_size}, \"\n f\"num_gpus={num_gpus}, \"\n f\"num_accumulative_iters={cfg.GRADIENT_ACCUMULATION.num_iters}\")\n if cfg.get('use_npu', False):\n places = paddle.set_device('npu')\n elif cfg.get('use_xpu', False):\n places = paddle.set_device('xpu')\n else:\n places = paddle.set_device('gpu')\n # default num worker: 0, which means no subprocess will be created\n num_workers = cfg.DATASET.get('num_workers', 0)\n valid_num_workers = cfg.DATASET.get('valid_num_workers', num_workers)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:76-96"
+ },
+ "8215": {
+ "file_id": 607,
+ "content": "The code ensures the global batch size is divisible by cur_global_batch_size, sets the number of iterations needed to reach the global batch size, and sets the device type (NPU, XPU, or GPU) based on config values. It also allows for setting the number of workers for training and validation data loading.",
+ "type": "comment"
+ },
+ "8216": {
+ "file_id": 607,
+ "content": " model_name = cfg.model_name\n output_dir = cfg.get(\"output_dir\", f\"./output/{model_name}\")\n mkdir(output_dir)\n # 1. Construct model\n model = build_model(cfg.MODEL)\n if cfg.get('to_static', False):\n specs = None\n model = paddle.jit.to_static(model, input_spec=specs)\n logger.info(\n \"Successfully to apply @to_static with specs: {}\".format(specs))\n # 2. Construct dataset and dataloader for training and evaluation\n train_dataset = build_dataset((cfg.DATASET.train, cfg.PIPELINE.train))\n train_dataloader_setting = dict(\n batch_size=batch_size,\n num_workers=num_workers,\n collate_fn_cfg=cfg.get('MIX', None),\n places=places)\n train_loader = build_dataloader(train_dataset, **train_dataloader_setting)\n if validate:\n valid_dataset = build_dataset((cfg.DATASET.valid, cfg.PIPELINE.valid))\n validate_dataloader_setting = dict(\n batch_size=valid_batch_size,\n num_workers=valid_num_workers,\n places=places,",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:97-124"
+ },
+ "8217": {
+ "file_id": 607,
+ "content": "Code snippet builds a model, creates dataset and dataloader for training and validation, and optionally converts the model to static using Paddle.jit.to_static(). It saves the output in the specified directory and logs if @to_static is applied successfully.",
+ "type": "comment"
+ },
+ "8218": {
+ "file_id": 607,
+ "content": " drop_last=False,\n shuffle=cfg.DATASET.get(\n 'shuffle_valid',\n False) # NOTE: attention_LSTM needs to shuffle valid data.\n )\n valid_loader = build_dataloader(valid_dataset,\n **validate_dataloader_setting)\n # 3. Construct learning rate scheduler(lr) and optimizer\n lr = build_lr(cfg.OPTIMIZER.learning_rate, len(train_loader))\n optimizer = build_optimizer(\n cfg.OPTIMIZER, lr, model=model, use_amp=use_amp, amp_level=amp_level)\n # 4. Construct scalar and convert parameters for amp(optional)\n if use_amp:\n scaler = amp.GradScaler(\n init_loss_scaling=2.0**16,\n incr_every_n_steps=2000,\n decr_every_n_nan_or_inf=1)\n # convert model parameters to fp16 when amp_level is O2(pure fp16)\n model, optimizer = amp.decorate(\n models=model,\n optimizers=optimizer,\n level=amp_level,\n master_weight=True,\n save_dtype=None)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:125-150"
+ },
+ "8219": {
+ "file_id": 607,
+ "content": "This code is setting up a training process for the PaddleVideo framework. It first creates train and validation dataloaders with specified settings, then constructs a learning rate scheduler and optimizer based on provided configurations. Optionally, it converts model parameters to fp16 using AMP if needed.",
+ "type": "comment"
+ },
+ "8220": {
+ "file_id": 607,
+ "content": " # NOTE: save_dtype is set to float32 now.\n logger.info(f\"Training in amp mode, amp_level={amp_level}.\")\n else:\n assert amp_level is None, f\"amp_level must be None when training in fp32 mode, but got {amp_level}.\"\n logger.info(\"Training in fp32 mode.\")\n # 5. Resume(optional)\n resume_epoch = cfg.get(\"resume_epoch\", 0)\n if resume_epoch:\n filename = osp.join(output_dir,\n model_name + f\"_epoch_{resume_epoch:05d}\")\n resume_model_dict = load(filename + '.pdparams')\n resume_opt_dict = load(filename + '.pdopt')\n model.set_state_dict(resume_model_dict)\n optimizer.set_state_dict(resume_opt_dict)\n logger.info(\"Resume from checkpoint: {}\".format(filename))\n # 6. Finetune(optional)\n if weights:\n assert resume_epoch == 0, f\"Conflict occurs when finetuning, please switch resume function off by setting resume_epoch to 0 or not indicating it.\"\n model_dict = load(weights)\n model.set_state_dict(model_dict)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:151-172"
+ },
+ "8221": {
+ "file_id": 607,
+ "content": "The code checks if training in amp mode or fp32 mode. If in amp mode, it asserts that the amp_level is not None and logs the current level. If in fp32 mode, it asserts that amp_level is None and logs the mode. It then handles optional resume and finetuning steps if specified by loading model weights from a file, setting the model state dictionary to the loaded dictionary, and logging the checkpoint used.",
+ "type": "comment"
+ },
+ "8222": {
+ "file_id": 607,
+ "content": " logger.info(\"Finetune from checkpoint: {}\".format(weights))\n # 7. Parallelize(optional)\n if parallel:\n model = paddle.DataParallel(model)\n if use_fleet:\n model = fleet.distributed_model(model)\n optimizer = fleet.distributed_optimizer(optimizer)\n # 8. Train Model\n best = 0.0\n for epoch in range(0, cfg.epochs):\n if epoch < resume_epoch:\n logger.info(\n f\"| epoch: [{epoch + 1}] <= resume_epoch: [{resume_epoch}], continue...\"\n )\n continue\n model.train()\n record_list = build_record(cfg.MODEL)\n tic = time.time()\n for i, data in enumerate(train_loader):\n \"\"\"Next two line of code only used in test_tipc,\n ignore it most of the time\"\"\"\n if max_iters is not None and i >= max_iters:\n break\n record_list['reader_time'].update(time.time() - tic)\n # Collect performance information when profiler_options is activate\n add_profiler_step(profiler_options)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:173-204"
+ },
+ "8223": {
+ "file_id": 607,
+ "content": "The code finetunes a model from a specified checkpoint. It optionally parallelizes the training process using Paddle's DataParallel API and Fleet distributed computing for further optimization. The code trains the model for a specified number of epochs, continuing from a previous resume_epoch if needed. Performance information is collected when profiler options are activated.",
+ "type": "comment"
+ },
+ "8224": {
+ "file_id": 607,
+ "content": " # 8.1 forward\n # AMP #\n if use_amp:\n with amp.auto_cast(\n custom_black_list={\"reduce_mean\", \"conv3d\"},\n level=amp_level):\n outputs = model(data, mode='train')\n avg_loss = outputs['loss']\n if use_gradient_accumulation:\n # clear grad at when epoch begins\n if i == 0:\n optimizer.clear_grad()\n # Loss normalization\n avg_loss /= cfg.GRADIENT_ACCUMULATION.num_iters\n # Loss scaling\n scaled = scaler.scale(avg_loss)\n # 8.2 backward\n scaled.backward()\n # 8.3 minimize\n if (i + 1) % cfg.GRADIENT_ACCUMULATION.num_iters == 0:\n scaler.minimize(optimizer, scaled)\n optimizer.clear_grad()\n else: # general case\n # Loss scaling",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:206-229"
+ },
+ "8225": {
+ "file_id": 607,
+ "content": "Applies Automatic Mixed Precision (AMP) for faster training, calculates average loss, performs gradient accumulation, and scales backpropagation to reduce memory usage.",
+ "type": "comment"
+ },
+ "8226": {
+ "file_id": 607,
+ "content": " scaled = scaler.scale(avg_loss)\n # 8.2 backward\n scaled.backward()\n # 8.3 minimize\n scaler.minimize(optimizer, scaled)\n optimizer.clear_grad()\n else:\n outputs = model(data, mode='train')\n avg_loss = outputs['loss']\n if use_gradient_accumulation:\n # clear grad at when epoch begins\n if i == 0:\n optimizer.clear_grad()\n # Loss normalization\n avg_loss /= cfg.GRADIENT_ACCUMULATION.num_iters\n # 8.2 backward\n avg_loss.backward()\n # 8.3 minimize\n if (i + 1) % cfg.GRADIENT_ACCUMULATION.num_iters == 0:\n optimizer.step()\n optimizer.clear_grad()\n else: # general case\n # 8.2 backward\n avg_loss.backward()",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:230-253"
+ },
+ "8227": {
+ "file_id": 607,
+ "content": "This code calculates the average loss, scales it if necessary, performs backward pass, and applies gradient descent to minimize the loss. If gradient accumulation is used, the gradients are cleared at the start of each epoch and after every accumulated number of iterations.",
+ "type": "comment"
+ },
+ "8228": {
+ "file_id": 607,
+ "content": " # 8.3 minimize\n optimizer.step()\n optimizer.clear_grad()\n # log record\n record_list['lr'].update(optimizer.get_lr(), batch_size)\n for name, value in outputs.items():\n if name in record_list:\n record_list[name].update(value, batch_size)\n record_list['batch_time'].update(time.time() - tic)\n tic = time.time()\n if i % cfg.get(\"log_interval\", 10) == 0:\n ips = \"ips: {:.5f} instance/sec,\".format(\n batch_size / record_list[\"batch_time\"].val)\n cur_progress = ((i + 1) + epoch * len(train_loader)) / (\n len(train_loader) * cfg.epochs)\n eta = int(record_list[\"batch_time\"].sum *\n (1 - cur_progress) / cur_progress + 0.5)\n log_batch(record_list, i, epoch + 1, cfg.epochs, \"train\", ips,\n eta)\n # learning rate iter step",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:254-277"
+ },
+ "8229": {
+ "file_id": 607,
+ "content": "Optimizer step and gradient clearance followed by logging records, updating logs, calculating instantaneous performance (ips), determining progress and estimated time of arrival (eta), and calling log_batch function.",
+ "type": "comment"
+ },
+ "8230": {
+ "file_id": 607,
+ "content": " if cfg.OPTIMIZER.learning_rate.get(\"iter_step\"):\n lr.step()\n # learning rate epoch step\n if not cfg.OPTIMIZER.learning_rate.get(\"iter_step\"):\n lr.step()\n ips = \"avg_ips: {:.5f} instance/sec.\".format(\n batch_size * record_list[\"batch_time\"].count /\n record_list[\"batch_time\"].sum)\n log_epoch(record_list, epoch + 1, \"train\", ips)\n def evaluate(best):\n model.eval()\n results = []\n record_list = build_record(cfg.MODEL)\n record_list.pop('lr')\n tic = time.time()\n if parallel:\n rank = dist.get_rank()\n # single_gpu_test and multi_gpu_test\n for i, data in enumerate(valid_loader):\n \"\"\"Next two line of code only used in test_tipc,\n ignore it most of the time\"\"\"\n if max_iters is not None and i >= max_iters:\n break\n if use_amp:\n with amp.auto_cast(",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:278-306"
+ },
+ "8231": {
+ "file_id": 607,
+ "content": "This code snippet is from the PaddleVideo library and it contains code for training a model. It uses an optimizer with a learning rate that can be stepped based on whether it's an iterative step or not. After performing an epoch, it logs the average instances per second processed. The code then evaluates the model by setting it to evaluation mode and collecting test results using a record list. It also records the time taken for testing in 'tic'.",
+ "type": "comment"
+ },
+ "8232": {
+ "file_id": 607,
+ "content": " custom_black_list={\"reduce_mean\", \"conv3d\"},\n level=amp_level):\n outputs = model(data, mode='valid')\n else:\n outputs = model(data, mode='valid')\n if cfg.MODEL.framework == \"FastRCNN\":\n results.extend(outputs)\n # log_record\n if cfg.MODEL.framework != \"FastRCNN\":\n for name, value in outputs.items():\n if name in record_list:\n record_list[name].update(value, batch_size)\n record_list['batch_time'].update(time.time() - tic)\n tic = time.time()\n if i % cfg.get(\"log_interval\", 10) == 0:\n ips = \"ips: {:.5f} instance/sec.\".format(\n valid_batch_size / record_list[\"batch_time\"].val)\n log_batch(record_list, i, epoch + 1, cfg.epochs, \"val\", ips)\n if cfg.MODEL.framework == \"FastRCNN\":",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:307-330"
+ },
+ "8233": {
+ "file_id": 607,
+ "content": "This code snippet is from the PaddleVideo library and appears to be handling model training for a specific framework. It calculates outputs, updates records for non-FastRCNN models, logs batch information, and handles FastRCNN-specific operations. The code also includes functionality for updating batch time and logging progress at regular intervals.",
+ "type": "comment"
+ },
+ "8234": {
+ "file_id": 607,
+ "content": " if parallel:\n results = collect_results_cpu(results, len(valid_dataset))\n if not parallel or (parallel and rank == 0):\n eval_res = valid_dataset.evaluate(results)\n for name, value in eval_res.items():\n record_list[name].update(value, valid_batch_size)\n ips = \"avg_ips: {:.5f} instance/sec.\".format(\n valid_batch_size * record_list[\"batch_time\"].count /\n record_list[\"batch_time\"].sum)\n log_epoch(record_list, epoch + 1, \"val\", ips)\n best_flag = False\n if cfg.MODEL.framework == \"FastRCNN\" and (not parallel or\n (parallel and rank == 0)):\n if record_list[\"mAP@0.5IOU\"].val > best:\n best = record_list[\"mAP@0.5IOU\"].val\n best_flag = True\n return best, best_flag\n if cfg.MODEL.framework == \"YOWOLocalizer\" and (not parallel or",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:331-351"
+ },
+ "8235": {
+ "file_id": 607,
+ "content": "Code section checks if parallel processing is enabled, collects results for CPU, and evaluates the dataset. It calculates average instance processing speed and logs it. If using specific models like FastRCNN or YOWOLocalizer, compares current performance metrics with the best values achieved so far and returns them along with a flag indicating if a new best value was found.",
+ "type": "comment"
+ },
+ "8236": {
+ "file_id": 607,
+ "content": " (parallel and rank == 0)):\n if record_list[\"fscore\"].avg > best:\n best = record_list[\"fscore\"].avg\n best_flag = True\n return best, best_flag\n # forbest2, cfg.MODEL.framework != \"FastRCNN\":\n for top_flag in ['hit_at_one', 'top1', 'rmse', \"F1@0.50\"]:\n if record_list.get(top_flag):\n if top_flag != 'rmse' and record_list[top_flag].avg > best:\n best = record_list[top_flag].avg\n best_flag = True\n elif top_flag == 'rmse' and (\n best == 0.0 or record_list[top_flag].avg < best):\n best = record_list[top_flag].avg\n best_flag = True\n return best, best_flag\n # use precise bn to improve acc\n if cfg.get(\"PRECISEBN\") and (\n epoch % cfg.PRECISEBN.preciseBN_interval == 0",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:352-373"
+ },
+ "8237": {
+ "file_id": 607,
+ "content": "This code is updating the best value and flag based on various metrics (fscore, hit_at_one, top1, rmse, F1@0.50) in a parallel setting with rank 0. It also checks if using precise batch normalization improves accuracy every 'preciseBN_interval' epochs.",
+ "type": "comment"
+ },
+ "8238": {
+ "file_id": 607,
+ "content": " or epoch == cfg.epochs - 1):\n do_preciseBN(model, train_loader, parallel,\n min(cfg.PRECISEBN.num_iters_preciseBN,\n len(train_loader)), use_amp, amp_level)\n # 9. Validation\n if validate and (epoch % cfg.get(\"val_interval\", 1) == 0\n or epoch == cfg.epochs - 1):\n with paddle.no_grad():\n best, save_best_flag = evaluate(best)\n # save best\n if save_best_flag:\n save(optimizer.state_dict(),\n osp.join(output_dir, model_name + \"_best.pdopt\"))\n save_student_model_flag = True if \"Distillation\" in cfg.MODEL.framework else False\n save(\n model.state_dict(),\n osp.join(output_dir, model_name + \"_best.pdparams\"),\n save_student_model=save_student_model_flag)\n if model_name == \"AttentionLstm\":\n logger.info(\n f\"Already save the best model (hit_at_one){best}\")",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:374-395"
+ },
+ "8239": {
+ "file_id": 607,
+ "content": "This code block is responsible for the precise Batch Normalization and validation steps in a deep learning training process. It applies PreciseBN for specific number of iterations, performs validation every \"val_interval\" epochs or at the last epoch, saves best model state if validation accuracy improves, and handles model saving differently depending on the framework used (Distillation vs others).",
+ "type": "comment"
+ },
+ "8240": {
+ "file_id": 607,
+ "content": " elif cfg.MODEL.framework == \"FastRCNN\":\n logger.info(\n f\"Already save the best model (mAP@0.5IOU){int(best * 10000) / 10000}\"\n )\n elif cfg.MODEL.framework == \"DepthEstimator\":\n logger.info(\n f\"Already save the best model (rmse){int(best * 10000) / 10000}\"\n )\n elif cfg.MODEL.framework in ['MSTCN', 'ASRF']:\n logger.info(\n f\"Already save the best model (F1@0.50){int(best * 10000) / 10000}\"\n )\n elif cfg.MODEL.framework in ['YOWOLocalizer']:\n logger.info(\n f\"Already save the best model (fsocre){int(best * 10000) / 10000}\"\n )\n else:\n logger.info(\n f\"Already save the best model (top1 acc){int(best * 10000) / 10000}\"\n )\n # 10. Save model and optimizer",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:396-417"
+ },
+ "8241": {
+ "file_id": 607,
+ "content": "This code block checks the current model framework and logs the metric used to identify the best model saved, followed by saving the best model and optimizer.",
+ "type": "comment"
+ },
+ "8242": {
+ "file_id": 607,
+ "content": " if epoch % cfg.get(\"save_interval\", 1) == 0 or epoch == cfg.epochs - 1:\n save(optimizer.state_dict(),\n osp.join(output_dir,\n model_name + f\"_epoch_{epoch + 1:05d}.pdopt\"))\n save(model.state_dict(),\n osp.join(output_dir,\n model_name + f\"_epoch_{epoch + 1:05d}.pdparams\"))\n logger.info(f'training {model_name} finished')",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train.py:418-426"
+ },
+ "8243": {
+ "file_id": 607,
+ "content": "This code saves the optimizer and model state dictionaries at specific intervals during training. The optimizer state is saved with a .pdopt extension and the model state is saved with a .pdparams extension. This occurs if the current epoch is either divisible by the save_interval or is the final epoch, to preserve progress during training. Finally, it logs that training for the specified model has finished.",
+ "type": "comment"
+ },
+ "8244": {
+ "file_id": 608,
+ "content": "/paddlevideo/tasks/train_dali.py",
+ "type": "filepath"
+ },
+ "8245": {
+ "file_id": 608,
+ "content": "The code sets up libraries, initializes DALI and TSN model, creates a dataloader, builds solver, trains model with optimization steps, logs performance metrics, updates learning rates, supports resuming training/finetuning, and saves states at intervals.",
+ "type": "summary"
+ },
+ "8246": {
+ "file_id": 608,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport time\nimport os.path as osp\nimport paddle\nfrom ..modeling.builder import build_model\nfrom ..solver import build_lr, build_optimizer\nfrom ..utils import do_preciseBN\nfrom paddlevideo.utils import get_logger, coloring\nfrom paddlevideo.utils import (AverageMeter, build_record, log_batch, log_epoch,\n save, load, mkdir)\nfrom paddlevideo.loader import TSN_Dali_loader, get_input_data",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_dali.py:1-25"
+ },
+ "8247": {
+ "file_id": 608,
+ "content": "This code imports necessary libraries and modules, sets up licenses, and imports functions from other files for model building, solver configuration, and additional utility functions. It also defines a loader for TSN-Dali dataset and functions for input data preparation.",
+ "type": "comment"
+ },
+ "8248": {
+ "file_id": 608,
+ "content": "\"\"\"\nWe only supported DALI training for TSN model now.\n\"\"\"\ndef train_dali(cfg, weights=None, parallel=True):\n \"\"\"Train model entry\n Args:\n \tcfg (dict): configuration.\n weights (str): weights path for finetuning.\n \tparallel (bool): Whether multi-cards training. Default: True.\n \"\"\"\n logger = get_logger(\"paddlevideo\")\n batch_size = cfg.DALI_LOADER.get('batch_size', 8)\n places = paddle.set_device('gpu')\n model_name = cfg.model_name\n output_dir = cfg.get(\"output_dir\", f\"./output/{model_name}\")\n mkdir(output_dir)\n # 1. Construct model\n model = build_model(cfg.MODEL)\n if parallel:\n model = paddle.DataParallel(model)\n # 2. Construct dali dataloader\n train_loader = TSN_Dali_loader(cfg.DALI_LOADER).build_dali_reader()\n # 3. Construct solver.\n lr = build_lr(cfg.OPTIMIZER.learning_rate, None)\n optimizer = build_optimizer(cfg.OPTIMIZER, lr, model=model)\n # Resume\n resume_epoch = cfg.get(\"resume_epoch\", 0)\n if resume_epoch:\n filename = osp.join(output_dir,",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_dali.py:26-63"
+ },
+ "8249": {
+ "file_id": 608,
+ "content": "This code snippet initializes and trains a DALI (Data Augmentation and Input Pipeline Library) for the TSN model. It first constructs the model, creates a Dali dataloader, builds a solver with specified optimizer and learning rate, and then resumes training from the last checkpoint if provided.",
+ "type": "comment"
+ },
+ "8250": {
+ "file_id": 608,
+ "content": " model_name + f\"_epoch_{resume_epoch:05d}\")\n resume_model_dict = load(filename + '.pdparams')\n resume_opt_dict = load(filename + '.pdopt')\n model.set_state_dict(resume_model_dict)\n optimizer.set_state_dict(resume_opt_dict)\n # Finetune:\n if weights:\n assert resume_epoch == 0, f\"Conflict occurs when finetuning, please switch resume function off by setting resume_epoch to 0 or not indicating it.\"\n model_dict = load(weights)\n model.set_state_dict(model_dict)\n # 4. Train Model\n for epoch in range(0, cfg.epochs):\n if epoch < resume_epoch:\n logger.info(\n f\"| epoch: [{epoch+1}] <= resume_epoch: [{ resume_epoch}], continue... \"\n )\n continue\n model.train()\n record_list = build_record(cfg.MODEL)\n tic = time.time()\n for i, data in enumerate(train_loader):\n data = get_input_data(data)\n record_list['reader_time'].update(time.time() - tic)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_dali.py:64-88"
+ },
+ "8251": {
+ "file_id": 608,
+ "content": "This code snippet is part of a model training pipeline. It first checks if the resume_epoch is 0 or if weights are provided for finetuning, then loads and sets the corresponding state dictionaries for the model and optimizer. The model is trained for specified epochs, with the option to continue from a previous epoch or start from scratch depending on the resume_epoch and weights inputs. It also records reader time during training loop iterations.",
+ "type": "comment"
+ },
+ "8252": {
+ "file_id": 608,
+ "content": " # 4.1 forward\n outputs = model(data, mode='train')\n # 4.2 backward\n avg_loss = outputs['loss']\n avg_loss.backward()\n # 4.3 minimize\n optimizer.step()\n optimizer.clear_grad()\n # log record\n record_list['lr'].update(optimizer._global_learning_rate(),\n batch_size)\n for name, value in outputs.items():\n record_list[name].update(value, batch_size)\n record_list['batch_time'].update(time.time() - tic)\n tic = time.time()\n if i % cfg.get(\"log_interval\", 10) == 0:\n ips = \"ips: {:.5f} instance/sec.\".format(\n batch_size / record_list[\"batch_time\"].val)\n log_batch(record_list, i, epoch + 1, cfg.epochs, \"train\", ips)\n # learning rate iter step\n if cfg.OPTIMIZER.learning_rate.get(\"iter_step\"):\n lr.step()\n # learning rate epoch step",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_dali.py:89-116"
+ },
+ "8253": {
+ "file_id": 608,
+ "content": "This code is training a model. It performs forward, backward pass, and optimization steps before logging performance metrics and updating learning rates. The model takes input data and calculates outputs in 'train' mode. Then, it calculates the average loss from the outputs. Next, it updates gradients using backward propagation, optimizes the model with step and clears gradients. It records log information such as learning rate and batch time for later analysis. The code also checks if there is an interval in the training to log current metrics and provides an instance per second rate (ips) as performance indicator. Lastly, it updates learning rates using both iteration steps and epoch steps, based on configuration settings.",
+ "type": "comment"
+ },
+ "8254": {
+ "file_id": 608,
+ "content": " if not cfg.OPTIMIZER.learning_rate.get(\"iter_step\"):\n lr.step()\n ips = \"ips: {:.5f} instance/sec.\".format(\n batch_size * record_list[\"batch_time\"].count /\n record_list[\"batch_time\"].sum)\n log_epoch(record_list, epoch + 1, \"train\", ips)\n # use precise bn to improve acc\n if cfg.get(\"PRECISEBN\") and (epoch % cfg.PRECISEBN.preciseBN_interval\n == 0 or epoch == cfg.epochs - 1):\n do_preciseBN(\n model, train_loader, parallel,\n min(cfg.PRECISEBN.num_iters_preciseBN, len(train_loader)))\n # 5. Save model and optimizer\n if epoch % cfg.get(\"save_interval\", 1) == 0 or epoch == cfg.epochs - 1:\n save(\n optimizer.state_dict(),\n osp.join(output_dir,\n model_name + f\"_epoch_{epoch+1:05d}.pdopt\"))\n save(\n model.state_dict(),\n osp.join(output_dir,\n model_name + f\"_epoch_{epoch+1:05d}.pdparams\"))",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_dali.py:117-141"
+ },
+ "8255": {
+ "file_id": 608,
+ "content": "This code chunk performs the following actions:\n1. Checks if learning rate should be updated based on iteration count.\n2. Calculates and logs the training instance speed (ips).\n3. Optionally applies precise Batch Normalization (bn) to improve accuracy.\n4. Saves the model's and optimizer's state every 'save_interval' epochs.",
+ "type": "comment"
+ },
+ "8256": {
+ "file_id": 608,
+ "content": " logger.info(f'training {model_name} finished')",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_dali.py:143-143"
+ },
+ "8257": {
+ "file_id": 608,
+ "content": "This line logs the completion of training a specific model using the \"logger.info\" function, indicating that the training process for the specified \"model_name\" has ended.",
+ "type": "comment"
+ },
+ "8258": {
+ "file_id": 609,
+ "content": "/paddlevideo/tasks/train_multigrid.py",
+ "type": "filepath"
+ },
+ "8259": {
+ "file_id": 609,
+ "content": "The code prepares the environment for training PaddleVideo models, builds a multigrid configuration, handles device and parallelism, trains the model, optimizes it using specified optimizer, logs progress/learning rate updates, evaluates performance, saves state, and saves model & optimizer.",
+ "type": "summary"
+ },
+ "8260": {
+ "file_id": 609,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport time\nimport os.path as osp\nimport paddle\nimport paddle.distributed as dist\nfrom ..loader.builder import build_dataloader, build_dataset\nfrom ..modeling.builder import build_model\nfrom ..solver import build_lr, build_optimizer\nfrom ..utils import do_preciseBN\nfrom paddlevideo.utils import get_logger, coloring\nfrom paddlevideo.utils import (AverageMeter, build_record, log_batch, log_epoch,\n save, load, mkdir)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:1-27"
+ },
+ "8261": {
+ "file_id": 609,
+ "content": "The code snippet is the opening section of the file \"train_multigrid.py\" within the PaddleVideo library. It starts by declaring copyright, licensing information, and importing necessary modules. It also includes functions to build datasets, models, loaders, solvers, and utilities for logging, saving, and loading model parameters and progress. This section sets up the environment for training video models in the PaddleVideo framework.",
+ "type": "comment"
+ },
+ "8262": {
+ "file_id": 609,
+ "content": "from paddlevideo.utils.multigrid import MultigridSchedule, aggregate_sub_bn_stats, subn_load, subn_save, is_eval_epoch\ndef construct_loader(cfg, places, validate, precise_bn, num_iters_precise_bn,\n world_size):\n batch_size = cfg.DATASET.get('batch_size', 2)\n train_dataset = build_dataset((cfg.DATASET.train, cfg.PIPELINE.train))\n precise_bn_dataloader_setting = dict(\n batch_size=batch_size,\n num_workers=cfg.DATASET.get('num_workers', 0),\n places=places,\n )\n if precise_bn:\n cfg.DATASET.train.num_samples_precise_bn = num_iters_precise_bn * batch_size * world_size\n precise_bn_dataset = build_dataset((cfg.DATASET.train,\n cfg.PIPELINE.train))\n precise_bn_loader = build_dataloader(precise_bn_dataset,\n **precise_bn_dataloader_setting)\n cfg.DATASET.train.num_samples_precise_bn = None\n else:\n precise_bn_loader = None\n if cfg.MULTIGRID.SHORT_CYCLE:",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:28-50"
+ },
+ "8263": {
+ "file_id": 609,
+ "content": "This function constructs data loaders for training a model with the multigrid approach. It takes several arguments including configuration (cfg), places to distribute the data, whether to use precise batch normalization (precise_bn), number of iterations for precise BN (num_iters_precise_bn), and world size. If precise BN is enabled, it adjusts the number of samples in the training dataset, creates a separate loader for precise BN, and sets the adjusted number of samples back to None. If not, it sets the precise BN loader to None. The code also checks if a short cycle multigrid approach is being used.",
+ "type": "comment"
+ },
+ "8264": {
+ "file_id": 609,
+ "content": " # get batch size list in short cycle schedule\n bs_factor = [\n int(\n round((float(cfg.PIPELINE.train.transform[1]['MultiCrop'][\n 'target_size']) / (s * cfg.MULTIGRID.default_crop_size))\n **2)) for s in cfg.MULTIGRID.short_cycle_factors\n ]\n batch_sizes = [\n batch_size * bs_factor[0],\n batch_size * bs_factor[1],\n batch_size,\n ]\n train_dataloader_setting = dict(\n batch_size=batch_sizes,\n multigrid=True,\n num_workers=cfg.DATASET.get('num_workers', 0),\n places=places,\n )\n else:\n train_dataloader_setting = precise_bn_dataloader_setting\n train_loader = build_dataloader(train_dataset, **train_dataloader_setting)\n if validate:\n valid_dataset = build_dataset((cfg.DATASET.valid, cfg.PIPELINE.valid))\n validate_dataloader_setting = dict(\n batch_size=batch_size,\n num_workers=cfg.DATASET.get('num_workers', 0),",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:51-77"
+ },
+ "8265": {
+ "file_id": 609,
+ "content": "The code adjusts the batch size in a short cycle schedule based on target image size, multi-grid factors and default crop size. It then sets up a train_dataloader with these batch sizes and other parameters. If validate is True, it also builds a valid_dataset and valid_dataloader with the given configurations.",
+ "type": "comment"
+ },
+ "8266": {
+ "file_id": 609,
+ "content": " places=places,\n drop_last=False,\n shuffle=False)\n valid_loader = build_dataloader(valid_dataset,\n **validate_dataloader_setting)\n else:\n valid_loader = None\n return train_loader, valid_loader, precise_bn_loader\ndef build_trainer(cfg, places, parallel, validate, precise_bn,\n num_iters_precise_bn, world_size):\n \"\"\"\n Build training model and its associated tools, including optimizer,\n dataloaders and meters.\n Args:\n cfg (CfgNode): configs.\n Returns:\n model: training model.\n optimizer: optimizer.\n train_loader: training data loader.\n val_loader: validatoin data loader.\n precise_bn_loader: training data loader for computing\n precise BN.\n \"\"\"\n model = build_model(cfg.MODEL)\n if parallel:\n model = paddle.DataParallel(model)\n train_loader, valid_loader, precise_bn_loader = \\\n construct_loader(cfg,\n places,",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:78-110"
+ },
+ "8267": {
+ "file_id": 609,
+ "content": "This code is creating training and validation data loaders for a PaddleVideo model. It also builds the model, and if parallelization is enabled, it wraps the model with Paddle's DataParallel API to distribute computation across multiple GPUs. The function returns the trained model, its optimizer, and the various data loaders required for training and validation.",
+ "type": "comment"
+ },
+ "8268": {
+ "file_id": 609,
+ "content": " validate,\n precise_bn,\n num_iters_precise_bn,\n world_size,\n )\n lr = build_lr(cfg.OPTIMIZER.learning_rate, len(train_loader))\n optimizer = build_optimizer(cfg.OPTIMIZER, lr, model=model)\n return (\n model,\n lr,\n optimizer,\n train_loader,\n valid_loader,\n precise_bn_loader,\n )\ndef train_model_multigrid(cfg, world_size=1, validate=True):\n \"\"\"Train model entry\n Args:\n \tcfg (dict): configuration.\n \tparallel (bool): Whether multi-card training. Default: True\n validate (bool): Whether to do evaluation. Default: False.\n \"\"\"\n # Init multigrid.\n multigrid = None\n if cfg.MULTIGRID.LONG_CYCLE or cfg.MULTIGRID.SHORT_CYCLE:\n multigrid = MultigridSchedule()\n cfg = multigrid.init_multigrid(cfg)\n if cfg.MULTIGRID.LONG_CYCLE:\n cfg, _ = multigrid.update_long_cycle(cfg, cur_epoch=0)\n multi_save_epoch = [i[-1] - 1 for i in multigrid.schedule]",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:111-146"
+ },
+ "8269": {
+ "file_id": 609,
+ "content": "This code initializes a multigrid training configuration and builds the model, learning rate, optimizer, and loaders for training, validation, and precise Batch Normalization. It also includes an optional multigrid schedule for long or short cycles if specified in the configuration.",
+ "type": "comment"
+ },
+ "8270": {
+ "file_id": 609,
+ "content": " parallel = world_size != 1\n logger = get_logger(\"paddlevideo\")\n batch_size = cfg.DATASET.get('batch_size', 2)\n if cfg.get('use_npu', False):\n places = paddle.set_device('npu')\n elif cfg.get('use_xpu', False):\n places = paddle.set_device('xpu')\n else:\n places = paddle.set_device('gpu')\n model_name = cfg.model_name\n output_dir = cfg.get(\"output_dir\", f\"./output/{model_name}\")\n mkdir(output_dir)\n local_rank = dist.ParallelEnv().local_rank\n precise_bn = cfg.get(\"PRECISEBN\")\n num_iters_precise_bn = cfg.PRECISEBN.num_iters_preciseBN\n # 1. Construct model\n model = build_model(cfg.MODEL)\n if parallel:\n model = paddle.DataParallel(model)\n # 2. Construct dataloader\n train_loader, valid_loader, precise_bn_loader = \\\n construct_loader(cfg,\n places,\n validate,\n precise_bn,\n num_iters_precise_bn,\n world_size,\n )",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:148-179"
+ },
+ "8271": {
+ "file_id": 609,
+ "content": "This code sets the device (npu, xpu or gpu) based on configuration and creates the model, dataloaders for training, validation, and precise BN if needed. It also initializes a logger and handles distributed training using parallel models and dataloaders.",
+ "type": "comment"
+ },
+ "8272": {
+ "file_id": 609,
+ "content": " # 3. Construct optimizer\n lr = build_lr(cfg.OPTIMIZER.learning_rate, len(train_loader))\n optimizer = build_optimizer(\n cfg.OPTIMIZER, lr, parameter_list=model.parameters())\n # Resume\n resume_epoch = cfg.get(\"resume_epoch\", 0)\n if resume_epoch:\n filename = osp.join(\n output_dir,\n model_name + str(local_rank) + '_' + f\"{resume_epoch:05d}\")\n subn_load(model, filename, optimizer)\n # 4. Train Model\n best = 0.\n total_epochs = int(cfg.epochs * cfg.MULTIGRID.epoch_factor)\n for epoch in range(total_epochs):\n if epoch < resume_epoch:\n logger.info(\n f\"| epoch: [{epoch+1}] <= resume_epoch: [{ resume_epoch}], continue... \"\n )\n continue\n if cfg.MULTIGRID.LONG_CYCLE:\n cfg, changed = multigrid.update_long_cycle(cfg, epoch)\n if changed:\n logger.info(\"====== Rebuild model/optimizer/loader =====\")\n (\n model,\n lr,",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:181-210"
+ },
+ "8273": {
+ "file_id": 609,
+ "content": "Constructing the optimizer, resuming training from a previous checkpoint if specified in the config file, and updating the long cycle configuration for multi-grid training.",
+ "type": "comment"
+ },
+ "8274": {
+ "file_id": 609,
+ "content": " optimizer,\n train_loader,\n valid_loader,\n precise_bn_loader,\n ) = build_trainer(cfg, places, parallel, validate, precise_bn,\n num_iters_precise_bn, world_size)\n #load checkpoint after re-build model\n if epoch != 0:\n #epoch no need to -1, haved add 1 when save\n filename = osp.join(\n output_dir,\n model_name + str(local_rank) + '_' + f\"{(epoch):05d}\")\n subn_load(model, filename, optimizer)\n #update lr last epoch, not to use saved params\n lr.last_epoch = epoch\n lr.step(rebuild=True)\n model.train()\n record_list = build_record(cfg.MODEL)\n tic = time.time()\n for i, data in enumerate(train_loader):\n record_list['reader_time'].update(time.time() - tic)\n # 4.1 forward\n outputs = model(data, mode='train')",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:211-235"
+ },
+ "8275": {
+ "file_id": 609,
+ "content": "The code builds a trainer with specified configurations, optimizer, train and validation loaders. It loads checkpoints if the epoch is not zero and updates the learning rate for the next epoch before training the model on the given data.",
+ "type": "comment"
+ },
+ "8276": {
+ "file_id": 609,
+ "content": " # 4.2 backward\n avg_loss = outputs['loss']\n avg_loss.backward()\n # 4.3 minimize\n optimizer.step()\n optimizer.clear_grad()\n # log record\n record_list['lr'].update(\n float(optimizer._global_learning_rate()), batch_size)\n for name, value in outputs.items():\n record_list[name].update(float(value), batch_size)\n record_list['batch_time'].update(time.time() - tic)\n tic = time.time()\n if i % cfg.get(\"log_interval\", 10) == 0:\n ips = \"ips: {:.5f} instance/sec.\".format(\n batch_size / record_list[\"batch_time\"].val)\n log_batch(record_list, i, epoch + 1, total_epochs, \"train\", ips)\n # learning rate iter step\n if cfg.OPTIMIZER.learning_rate.get(\"iter_step\"):\n lr.step()\n # learning rate epoch step\n if not cfg.OPTIMIZER.learning_rate.get(\"iter_step\"):\n lr.step()",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:236-262"
+ },
+ "8277": {
+ "file_id": 609,
+ "content": "Performing backward pass, optimizing using given optimizer, logging progress, and updating learning rate in both iteration step and epoch step.",
+ "type": "comment"
+ },
+ "8278": {
+ "file_id": 609,
+ "content": " ips = \"ips: {:.5f} instance/sec.\".format(\n batch_size * record_list[\"batch_time\"].count /\n record_list[\"batch_time\"].sum)\n log_epoch(record_list, epoch + 1, \"train\", ips)\n def evaluate(best):\n model.eval()\n record_list = build_record(cfg.MODEL)\n record_list.pop('lr')\n tic = time.time()\n for i, data in enumerate(valid_loader):\n outputs = model(data, mode='valid')\n # log_record\n for name, value in outputs.items():\n record_list[name].update(float(value), batch_size)\n record_list['batch_time'].update(time.time() - tic)\n tic = time.time()\n if i % cfg.get(\"log_interval\", 10) == 0:\n ips = \"ips: {:.5f} instance/sec.\".format(\n batch_size / record_list[\"batch_time\"].val)\n log_batch(record_list, i, epoch + 1, total_epochs, \"val\",\n ips)",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:264-288"
+ },
+ "8279": {
+ "file_id": 609,
+ "content": "This code snippet evaluates the model's performance during training and updates the record list with new values. It also logs the progress at certain intervals, displaying the number of instances processed per second (ips). The function 'evaluate' is called to perform this evaluation for each data batch in the valid_loader, updating the record list accordingly.",
+ "type": "comment"
+ },
+ "8280": {
+ "file_id": 609,
+ "content": " ips = \"ips: {:.5f} instance/sec.\".format(\n batch_size * record_list[\"batch_time\"].count /\n record_list[\"batch_time\"].sum)\n log_epoch(record_list, epoch + 1, \"val\", ips)\n best_flag = False\n if record_list.get('top1') and record_list['top1'].avg > best:\n best = record_list['top1'].avg\n best_flag = True\n return best, best_flag\n # use precise bn to improve acc\n if is_eval_epoch(cfg, epoch, total_epochs, multigrid.schedule):\n logger.info(f\"do precise BN in {epoch+1} ...\")\n do_preciseBN(model, precise_bn_loader, parallel,\n min(num_iters_precise_bn, len(precise_bn_loader)))\n # aggregate sub_BN stats\n logger.info(\"Aggregate sub_BatchNorm stats...\")\n aggregate_sub_bn_stats(model)\n # 5. Validation\n if is_eval_epoch(cfg, epoch, total_epochs, multigrid.schedule):\n logger.info(f\"eval in {epoch+1} ...\")",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:290-313"
+ },
+ "8281": {
+ "file_id": 609,
+ "content": "The code calculates the instantaneous processing speed (ips) and checks if a new best performance has been achieved. It then logs this information. If it's an evaluation epoch, it performs precise batch normalization, aggregates sub-batch normalization stats, and validates the model.",
+ "type": "comment"
+ },
+ "8282": {
+ "file_id": 609,
+ "content": " with paddle.no_grad():\n best, save_best_flag = evaluate(best)\n # save best\n if save_best_flag:\n save(optimizer.state_dict(),\n osp.join(output_dir, model_name + \"_best.pdopt\"))\n save(model.state_dict(),\n osp.join(output_dir, model_name + \"_best.pdparams\"))\n logger.info(\n f\"Already save the best model (top1 acc){int(best * 10000) / 10000}\"\n )\n # 6. Save model and optimizer\n if is_eval_epoch(\n cfg, epoch,\n total_epochs, multigrid.schedule) or epoch % cfg.get(\n \"save_interval\", 10) == 0 or epoch in multi_save_epoch:\n logger.info(\"[Save parameters] ======\")\n subn_save(output_dir, model_name + str(local_rank) + '_', epoch + 1,\n model, optimizer)\n logger.info(f'training {model_name} finished')",
+ "type": "code",
+ "location": "/paddlevideo/tasks/train_multigrid.py:314-335"
+ },
+ "8283": {
+ "file_id": 609,
+ "content": "The code saves the best model if it outperforms previous results, and periodically saves the current model parameters during training. It uses the evaluate function to measure performance, the save function to store state dictionaries, and the subn_save function for saving models and optimizers at certain epochs. The logger is used for informative messages about saving and training completion.",
+ "type": "comment"
+ },
+ "8284": {
+ "file_id": 610,
+ "content": "/paddlevideo/utils/__init__.py",
+ "type": "filepath"
+ },
+ "8285": {
+ "file_id": 610,
+ "content": "This code imports various functions and classes from different modules within the PaddleVideo library. It also sets up logger and profiler functionality, provides a build function for creating objects, and handles saving and loading data.",
+ "type": "summary"
+ },
+ "8286": {
+ "file_id": 610,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import Registry\nfrom .build_utils import build\nfrom .config import *\nfrom .logger import setup_logger, coloring, get_logger\nfrom .record import AverageMeter, build_record, log_batch, log_epoch\nfrom .dist_utils import get_dist_info, main_only\nfrom .save_load import save, load, load_ckpt, mkdir\nfrom .precise_bn import do_preciseBN\nfrom .profiler import add_profiler_step\n__all__ = ['Registry', 'build']",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/__init__.py:1-24"
+ },
+ "8287": {
+ "file_id": 610,
+ "content": "This code imports various functions and classes from different modules within the PaddleVideo library. It also sets up logger and profiler functionality, provides a build function for creating objects, and handles saving and loading data.",
+ "type": "comment"
+ },
+ "8288": {
+ "file_id": 611,
+ "content": "/paddlevideo/utils/build_utils.py",
+ "type": "filepath"
+ },
+ "8289": {
+ "file_id": 611,
+ "content": "The \"build\" function takes a config dictionary and registry, constructs an object from the configuration, checks for required keys, retrieves class from the registry, and returns the instance.",
+ "type": "summary"
+ },
+ "8290": {
+ "file_id": 611,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\ndef build(cfg, registry, key='name'):\n \"\"\"Build a module from config dict.\n Args:\n cfg (dict): Config dict. It should at least contain the key.\n registry (XXX): The registry to search the type from.\n key (str): the key.\n Returns:\n obj: The constructed object.\n \"\"\"\n assert isinstance(cfg, dict) and key in cfg\n cfg_copy = cfg.copy()\n obj_type = cfg_copy.pop(key)\n obj_cls = registry.get(obj_type)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/build_utils.py:1-31"
+ },
+ "8291": {
+ "file_id": 611,
+ "content": "This code defines a function named \"build\" that takes a config dictionary and a registry, builds an object from the given configuration dictionary, and returns it. The function asserts that the input is a valid dictionary and checks if the required key exists. It then retrieves the object type from the dictionary and gets the corresponding class from the registry before returning the constructed object.",
+ "type": "comment"
+ },
+ "8292": {
+ "file_id": 611,
+ "content": " if obj_cls is None:\n raise KeyError('{} is not in the {} registry'.format(\n obj_type, registry.name))\n return obj_cls(**cfg_copy)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/build_utils.py:32-35"
+ },
+ "8293": {
+ "file_id": 611,
+ "content": "Checks if an object class is provided, raises a KeyError if not found in the registry, and returns an instance of the found class with provided configuration.",
+ "type": "comment"
+ },
+ "8294": {
+ "file_id": 612,
+ "content": "/paddlevideo/utils/config.py",
+ "type": "filepath"
+ },
+ "8295": {
+ "file_id": 612,
+ "content": "The code imports necessary modules, sets up a logger, creates an AttrDict class for config handling, and defines functions to load, visualize, and override dictionary from YAML file. It also includes 'options' and 'get_config' functions to apply overrides and print or return the updated configuration.",
+ "type": "summary"
+ },
+ "8296": {
+ "file_id": 612,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport yaml\nfrom paddlevideo.utils.logger import coloring, get_logger, setup_logger\n__all__ = ['get_config']\nlogger = setup_logger(\"./\", name=\"paddlevideo\", level=\"INFO\")\nclass AttrDict(dict):\n def __getattr__(self, key):\n return self[key]\n def __setattr__(self, key, value):\n if key in self.__dict__:\n self.__dict__[key] = value\n else:\n self[key] = value\ndef create_attr_dict(yaml_config):",
+ "type": "code",
+ "location": "/paddlevideo/utils/config.py:1-34"
+ },
+ "8297": {
+ "file_id": 612,
+ "content": "The code is importing necessary modules, defining an AttrDict class for config handling and setting up a logger. It also creates a function 'create_attr_dict' that takes in a yaml configuration file as input.",
+ "type": "comment"
+ },
+ "8298": {
+ "file_id": 612,
+ "content": " from ast import literal_eval\n for key, value in yaml_config.items():\n if type(value) is dict:\n yaml_config[key] = value = AttrDict(value)\n if isinstance(value, str):\n try:\n value = literal_eval(value)\n except BaseException:\n pass\n if isinstance(value, AttrDict):\n create_attr_dict(yaml_config[key])\n else:\n yaml_config[key] = value\ndef parse_config(cfg_file):\n \"\"\"Load a config file into AttrDict\"\"\"\n with open(cfg_file, 'r') as fopen:\n yaml_config = AttrDict(yaml.load(fopen, Loader=yaml.SafeLoader))\n create_attr_dict(yaml_config)\n return yaml_config\ndef print_dict(d, delimiter=0):\n \"\"\"\n Recursively visualize a dict and\n indenting acrrording by the relationship of keys.\n \"\"\"\n placeholder = \"-\" * 60\n for k, v in sorted(d.items()):\n if isinstance(v, dict):\n logger.info(\"{}{} : \".format(delimiter * \" \", coloring(k,\n \"HEADER\")))",
+ "type": "code",
+ "location": "/paddlevideo/utils/config.py:35-67"
+ },
+ "8299": {
+ "file_id": 612,
+ "content": "The code defines two functions: \"parse_config\" and \"print_dict\". The \"parse_config\" function loads a config file into an AttrDict, while the \"print_dict\" function recursively visualizes a dictionary by indenting according to the relationship of keys.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/83.json b/docs/data/83.json
new file mode 100644
index 000000000..adf322db1
--- /dev/null
+++ b/docs/data/83.json
@@ -0,0 +1,544 @@
+{
+ "8300": {
+ "file_id": 612,
+ "content": " print_dict(v, delimiter + 4)\n elif isinstance(v, list) and len(v) >= 1 and isinstance(v[0], dict):\n logger.info(\"{}{} : \".format(delimiter * \" \",\n coloring(str(k), \"HEADER\")))\n for value in v:\n print_dict(value, delimiter + 4)\n else:\n logger.info(\"{}{} : {}\".format(delimiter * \" \",\n coloring(k, \"HEADER\"),\n coloring(v, \"OKGREEN\")))\n if k.isupper():\n logger.info(placeholder)\ndef print_config(config):\n \"\"\"\n visualize configs\n Arguments:\n config: configs\n \"\"\"\n print_dict(config)\ndef check_config(config):\n \"\"\"\n Check config\n \"\"\"\n pass\ndef override(dl, ks, v):\n \"\"\"\n Recursively replace dict of list\n Args:\n dl(dict or list): dict or list to be replaced\n ks(list): list of keys\n v(str): value to be replaced\n \"\"\"\n def str2num(v):\n try:\n return eval(v)",
+ "type": "code",
+ "location": "/paddlevideo/utils/config.py:68-109"
+ },
+ "8301": {
+ "file_id": 612,
+ "content": "The code defines several functions related to handling and manipulating configurations. The \"print_config\" function visualizes the configs, while \"check_config\" checks the validity of the configs. The \"override\" function recursively replaces values in a dictionary or list using a provided key and value.",
+ "type": "comment"
+ },
+ "8302": {
+ "file_id": 612,
+ "content": " except Exception:\n return v\n assert isinstance(dl, (list, dict)), (\"{} should be a list or a dict\")\n assert len(ks) > 0, ('lenght of keys should larger than 0')\n if isinstance(dl, list):\n k = str2num(ks[0])\n if len(ks) == 1:\n assert k < len(dl), ('index({}) out of range({})'.format(k, dl))\n dl[k] = str2num(v)\n else:\n override(dl[k], ks[1:], v)\n else:\n if len(ks) == 1:\n #assert ks[0] in dl, ('{} is not exist in {}'.format(ks[0], dl))\n if not ks[0] in dl:\n logger.warning('A new filed ({}) detected!'.format(ks[0], dl))\n dl[ks[0]] = str2num(v)\n else:\n assert ks[0] in dl, (\n '({}) doesn\\'t exist in {}, a new dict field is invalid'.format(\n ks[0], dl))\n override(dl[ks[0]], ks[1:], v)\ndef override_config(config, options=None):\n \"\"\"\n Recursively override the config\n Args:\n config(dict): dict to be replaced",
+ "type": "code",
+ "location": "/paddlevideo/utils/config.py:110-139"
+ },
+ "8303": {
+ "file_id": 612,
+ "content": "This code is part of a function `override_config` that recursively overrides the given config with new options. The function first checks if the data `dl` is a list or dictionary, and then proceeds accordingly. If `dl` is a list, it extracts the first key from `ks`, converts it to an integer, and ensures that the index is within range of `dl`. It then updates the corresponding element in `dl` with the value `v`. If `ks` has more than one key, it calls the `override` function to update a specific field in `dl`. \n\nIf `dl` is a dictionary, it again handles two scenarios: when there's only one key and when there are multiple keys. In the single-key scenario, it checks if the key exists in `dl` (and warns if not) and updates its value. If there are multiple keys, it first asserts that the first key exists in `dl`, then calls the `override` function to update a specific field in `dl`.\n\nThe code also includes an exception handling block which simply returns the original value `v` in case of any error or exception.",
+ "type": "comment"
+ },
+ "8304": {
+ "file_id": 612,
+ "content": " options(list): list of pairs(key0.key1.idx.key2=value)\n such as: [\n epochs=20',\n 'PIPELINE.train.transform.1.ResizeImage.resize_short=300'\n ]\n Returns:\n config(dict): replaced config\n \"\"\"\n if options is not None:\n for opt in options:\n assert isinstance(opt,\n str), (\"option({}) should be a str\".format(opt))\n assert \"=\" in opt, (\n \"option({}) should contain a =\"\n \"to distinguish between key and value\".format(opt))\n pair = opt.split('=')\n assert len(pair) == 2, (\"there can be only a = in the option\")\n key, value = pair\n keys = key.split('.')\n override(config, keys, value)\n return config\ndef get_config(fname, overrides=None, show=True):\n \"\"\"\n Read config from file\n \"\"\"\n assert os.path.exists(fname), ('config file({}) is not exist'.format(fname))\n config = parse_config(fname)\n override_config(config, overrides)",
+ "type": "code",
+ "location": "/paddlevideo/utils/config.py:140-170"
+ },
+ "8305": {
+ "file_id": 612,
+ "content": "This code defines two functions, `options(list)` and `get_config(fname, overrides=None, show=True)`. The `options(list)` function takes a list of pairs (key-value) as input and replaces the config with new values. The `get_config(fname, overrides=None, show=True)` function reads the config from a file, applies any overrides, and if 'show' is True, prints the updated configuration.",
+ "type": "comment"
+ },
+ "8306": {
+ "file_id": 612,
+ "content": " if show:\n print_config(config)\n check_config(config)\n return config",
+ "type": "code",
+ "location": "/paddlevideo/utils/config.py:171-174"
+ },
+ "8307": {
+ "file_id": 612,
+ "content": "This code block checks if the 'show' variable is set to True, and if so, it calls a function named 'print_config' with the 'config' parameter. It then always executes another function called 'check_config' with the same 'config' argument before returning the 'config' variable.",
+ "type": "comment"
+ },
+ "8308": {
+ "file_id": 613,
+ "content": "/paddlevideo/utils/dist_utils.py",
+ "type": "filepath"
+ },
+ "8309": {
+ "file_id": 613,
+ "content": "This code is from PaddleVideo's EIVideo module and includes util functions for distributed computing. It defines a function get_dist_info() to retrieve the current rank and world size, and main_only() is a decorator that only runs the wrapped function if the rank is 0 (used in distributed environments).",
+ "type": "summary"
+ },
+ "8310": {
+ "file_id": 613,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport functools\nimport paddle\nimport paddle.distributed as dist\ndef get_dist_info():\n world_size = dist.get_world_size()\n rank = dist.get_rank()\n return rank, world_size\ndef main_only(func):\n @functools.wraps(func)\n def wrapper(*args, **kwargs):\n rank, _ = get_dist_info()\n if rank == 0:\n return func(*args, **kwargs)\n return wrapper",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/dist_utils.py:1-30"
+ },
+ "8311": {
+ "file_id": 613,
+ "content": "This code is from PaddleVideo's EIVideo module and includes util functions for distributed computing. It defines a function get_dist_info() to retrieve the current rank and world size, and main_only() is a decorator that only runs the wrapped function if the rank is 0 (used in distributed environments).",
+ "type": "comment"
+ },
+ "8312": {
+ "file_id": 614,
+ "content": "/paddlevideo/utils/logger.py",
+ "type": "filepath"
+ },
+ "8313": {
+ "file_id": 614,
+ "content": "This code sets up a colorful logging function for PaddleVideo, initializes logger with verbosity levels, and ensures non-propagation of logs. It configures logger for Python's logging module using different formats and handlers based on local rank.",
+ "type": "summary"
+ },
+ "8314": {
+ "file_id": 614,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport logging\nimport os\nimport sys\nimport datetime\nfrom paddle.distributed import ParallelEnv\nColor = {\n 'RED': '\\033[31m',\n 'HEADER': '\\033[35m', # deep purple\n 'PURPLE': '\\033[95m', # purple\n 'OKBLUE': '\\033[94m',\n 'OKGREEN': '\\033[92m',\n 'WARNING': '\\033[93m',\n 'FAIL': '\\033[91m',\n 'ENDC': '\\033[0m'\n}\ndef coloring(message, color=\"OKGREEN\"):\n assert color in Color.keys()\n if os.environ.get('COLORING', True):",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:1-38"
+ },
+ "8315": {
+ "file_id": 614,
+ "content": "This code is from the \"logger.py\" file in the PaddleVideo project, and it sets up a coloring function for logging messages with optional colors using ANSI escape sequences. The function takes a message and an optional color parameter, which should be one of the defined colors in the Color dictionary. It asserts that the provided color is indeed a key in the dictionary, and then returns the message with the specified color applied. The function also checks the environment variable \"COLORING\" to determine whether coloring should be enabled or not (default is True).",
+ "type": "comment"
+ },
+ "8316": {
+ "file_id": 614,
+ "content": " return Color[color] + str(message) + Color[\"ENDC\"]\n else:\n return message\nlogger_initialized = []\ndef setup_logger(output=None, name=\"paddlevideo\", level=\"INFO\"):\n \"\"\"\n Initialize the paddlevideo logger and set its verbosity level to \"INFO\".\n Args:\n output (str): a file name or a directory to save log. If None, will not save log file.\n If ends with \".txt\" or \".log\", assumed to be a file name.\n Otherwise, logs will be saved to `output/log.txt`.\n name (str): the root module name of this logger\n Returns:\n logging.Logger: a logger\n \"\"\"\n def time_zone(sec, fmt):\n real_time = datetime.datetime.now()\n return real_time.timetuple()\n logging.Formatter.converter = time_zone\n logger = logging.getLogger(name)\n if level == \"INFO\":\n logger.setLevel(logging.INFO)\n elif level==\"DEBUG\":\n logger.setLevel(logging.DEBUG)\n logger.propagate = False\n if level == \"DEBUG\":\n plain_formatter = logging.Formatter(",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:39-71"
+ },
+ "8317": {
+ "file_id": 614,
+ "content": "This code initializes the PaddleVideo logger and sets its verbosity level to \"INFO\" or \"DEBUG\", depending on the input argument. It also defines a custom time zone converter for logging, and ensures that the logger does not propagate logs to its parent loggers.",
+ "type": "comment"
+ },
+ "8318": {
+ "file_id": 614,
+ "content": " \"[%(asctime)s] %(name)s %(levelname)s: %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n else:\n plain_formatter = logging.Formatter(\n \"[%(asctime)s] %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n # stdout logging: master only\n local_rank = ParallelEnv().local_rank\n if local_rank == 0:\n ch = logging.StreamHandler(stream=sys.stdout)\n ch.setLevel(logging.DEBUG)\n formatter = plain_formatter\n ch.setFormatter(formatter)\n logger.addHandler(ch)\n # file logging: all workers\n if output is not None:\n if output.endswith(\".txt\") or output.endswith(\".log\"):\n filename = output\n else:\n filename = os.path.join(output, \".log.txt\")\n if local_rank > 0:\n filename = filename + \".rank{}\".format(local_rank)\n # PathManager.mkdirs(os.path.dirname(filename))\n os.makedirs(os.path.dirname(filename), exist_ok=True)\n # fh = logging.StreamHandler(_cached_log_stream(filename)\n fh = logging.FileHandler(filename, mode='a')",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:72-100"
+ },
+ "8319": {
+ "file_id": 614,
+ "content": "This code configures a logger for Python's logging module. It uses different formats and handlers (stdout, file) based on the local rank of the process, creating separate log files for each worker ranked greater than 0. If the output is a .txt or .log file, it will be used as-is; otherwise, a .log.txt file with optional rank appended will be created. The code also ensures that missing directories for the log file are created beforehand.",
+ "type": "comment"
+ },
+ "8320": {
+ "file_id": 614,
+ "content": " fh.setLevel(logging.DEBUG)\n fh.setFormatter(plain_formatter)\n logger.addHandler(fh)\n logger_initialized.append(name)\n return logger\ndef get_logger(name, output=None):\n logger = logging.getLogger(name)\n if name in logger_initialized:\n return logger\n return setup_logger(name=name, output=name)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py:101-113"
+ },
+ "8321": {
+ "file_id": 614,
+ "content": "This code initializes a logger object and sets its level to DEBUG, adds a file handler with a plain formatter, and appends the logger's name to an initialized list. The function returns the logger if it has been previously initialized for the given name; otherwise, it sets up the logger using the provided name and optional output.",
+ "type": "comment"
+ },
+ "8322": {
+ "file_id": 615,
+ "content": "/paddlevideo/utils/multigrid/__init__.py",
+ "type": "filepath"
+ },
+ "8323": {
+ "file_id": 615,
+ "content": "This code imports various functions and classes from different modules in the PaddleVideo library. The __all__ list specifies the exported public symbols including MultigridSchedule, get_norm, aggregate_sub_bn_stats, DistributedShortSampler, subn_save, subn_load, and is_eval_epoch.",
+ "type": "summary"
+ },
+ "8324": {
+ "file_id": 615,
+ "content": "from .multigrid import MultigridSchedule\nfrom .batchnorm_helper import get_norm, aggregate_sub_bn_stats\nfrom .short_sampler import DistributedShortSampler\nfrom .save_load_helper import subn_save, subn_load\nfrom .interval_helper import is_eval_epoch\n__all__ = [\n 'MultigridSchedule', 'get_norm', 'aggregate_sub_bn_stats',\n 'DistributedShortSampler', 'subn_save', 'subn_load', 'is_eval_epoch'\n]",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/__init__.py:1-10"
+ },
+ "8325": {
+ "file_id": 615,
+ "content": "This code imports various functions and classes from different modules in the PaddleVideo library. The __all__ list specifies the exported public symbols including MultigridSchedule, get_norm, aggregate_sub_bn_stats, DistributedShortSampler, subn_save, subn_load, and is_eval_epoch.",
+ "type": "comment"
+ },
+ "8326": {
+ "file_id": 616,
+ "content": "/paddlevideo/utils/multigrid/batchnorm_helper.py",
+ "type": "filepath"
+ },
+ "8327": {
+ "file_id": 616,
+ "content": "This code defines a PyTorch class for batch normalization, initializing a BatchNorm3D layer and including methods to compute mean and standard deviation. It also supports aggregating statistics from multiple splits and performs forward pass for training or evaluation.",
+ "type": "summary"
+ },
+ "8328": {
+ "file_id": 616,
+ "content": "from functools import partial\nimport paddle\ndef get_norm(bn_norm_type, bn_num_splits):\n \"\"\"\n Args:\n cfg (CfgNode): model building configs, details are in the comments of\n the config file.\n Returns:\n nn.Layer: the normalization layer.\n \"\"\"\n if bn_norm_type == \"batchnorm\":\n return paddle.nn.BatchNorm3D\n elif bn_norm_type == \"sub_batchnorm\":\n return partial(SubBatchNorm3D, num_splits=bn_num_splits)\n else:\n raise NotImplementedError(\n \"Norm type {} is not supported\".format(bn_norm_type))\ndef aggregate_sub_bn_stats(model):\n \"\"\"\n Recursively find all SubBN modules and aggregate sub-BN stats.\n Args:\n model (nn.Layer): model to be aggregate sub-BN stats\n Returns:\n count (int): number of SubBN module found.\n \"\"\"\n count = 0\n for child in model.children():\n if isinstance(child, SubBatchNorm3D):\n child.aggregate_stats()\n count += 1\n else:\n count += aggregate_sub_bn_stats(child)",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/batchnorm_helper.py:1-36"
+ },
+ "8329": {
+ "file_id": 616,
+ "content": "This code defines a function `get_norm` that returns the normalization layer based on the provided bn_norm_type and bn_num_splits. If bn_norm_type is 'batchnorm', it returns paddle.nn.BatchNorm3D, otherwise if it's 'sub_batchnorm', it returns a partially applied SubBatchNorm3D function with num_splits parameter set to bn_num_splits. If the norm type isn't supported, it raises a NotImplementedError. It also defines `aggregate_sub_bn_stats` function that recursively finds all SubBN modules in the given model and aggregates sub-BN stats by calling aggregate_stats() on each found SubBatchNorm3D module. It returns the count of SubBN modules found.",
+ "type": "comment"
+ },
+ "8330": {
+ "file_id": 616,
+ "content": " return count\nclass SubBatchNorm3D(paddle.nn.Layer):\n \"\"\"\n Implement based on paddle2.0.\n The standard BN layer computes stats across all examples in a GPU. In some\n cases it is desirable to compute stats across only a subset of examples\n SubBatchNorm3D splits the batch dimension into N splits, and run BN on\n each of them separately (so that the stats are computed on each subset of\n examples (1/N of batch) independently. During evaluation, it aggregates\n the stats from all splits into one BN.\n \"\"\"\n def __init__(self, num_splits, **args):\n \"\"\"\n Args:\n num_splits (int): number of splits.\n args (list): list of args\n \"\"\"\n super(SubBatchNorm3D, self).__init__()\n self.num_splits = num_splits\n self.num_features = args[\"num_features\"]\n self.weight_attr = args[\"weight_attr\"]\n self.bias_attr = args[\"bias_attr\"]\n # Keep only one set of weight and bias (outside).\n if self.weight_attr == False:\n self.weight = self.create_parameter(",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/batchnorm_helper.py:37-64"
+ },
+ "8331": {
+ "file_id": 616,
+ "content": "The code defines a SubBatchNorm3D class that implements Batch Normalization with the option to split the batch dimension into N splits. It computes stats for each subset of examples independently during training and aggregates them during evaluation. The class takes num_splits as an argument and other parameters such as num_features, weight_attr, and bias_attr are set in its constructor.",
+ "type": "comment"
+ },
+ "8332": {
+ "file_id": 616,
+ "content": " attr=None,\n shape=[self.num_features],\n default_initializer=paddle.nn.initializer.Constant(1.0))\n self.weight.stop_gradient = True\n else:\n self.weight = self.create_parameter(\n attr=self.weight_attr,\n shape=[self.num_features],\n default_initializer=paddle.nn.initializer.Constant(1.0))\n self.weight.stop_gradient = self.weight_attr is not None \\\n and self.weight_attr.learning_rate == 0.\n if self.bias_attr == False:\n self.bias = self.create_parameter(attr=None,\n shape=[self.num_features],\n is_bias=True)\n self.bias.stop_gradient = True\n else:\n self.bias = self.create_parameter(attr=self.bias_attr,\n shape=[self.num_features],\n is_bias=True)",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/batchnorm_helper.py:65-85"
+ },
+ "8333": {
+ "file_id": 616,
+ "content": "This code initializes the weight and bias parameters of a BatchNorm layer in PaddlePaddle. If learning rate is 0, it sets weight to have no gradient update, and if `bias_attr` is False, it sets the bias to True and stops its gradients from being updated.",
+ "type": "comment"
+ },
+ "8334": {
+ "file_id": 616,
+ "content": " self.bias.stop_gradient = self.bias_attr is not None \\\n and self.bias_attr.learning_rate == 0.\n # set weights and bias fixed (inner).\n args[\"weight_attr\"] = False\n args[\"bias_attr\"] = False\n self.bn = paddle.nn.BatchNorm3D(**args)\n # update number of features used in split_bn\n args[\"num_features\"] = self.num_features * self.num_splits\n self.split_bn = paddle.nn.BatchNorm3D(**args)\n def _get_aggregated_mean_std(self, means, stds, n):\n \"\"\"\n Calculate the aggregated mean and stds.\n Use the method of update mean and std when merge multi-part data.\n Args:\n means (tensor): mean values.\n stds (tensor): standard deviations.\n n (int): number of sets of means and stds.\n \"\"\"\n mean = paddle.sum(paddle.reshape(means, (n, -1)), axis=0) / n\n std = (paddle.sum(paddle.reshape(stds, (n, -1)), axis=0) / n +\n paddle.sum(paddle.reshape(",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/batchnorm_helper.py:86-108"
+ },
+ "8335": {
+ "file_id": 616,
+ "content": "Class is initializing a BatchNorm3D layer and storing two instances of it (self.bn and self.split_bn). The first instance has its weights and bias set as fixed (inner), while the second instance handles splitting the features for a specified number of splits. The function _get_aggregated_mean_std calculates the aggregated mean and standard deviation by summing each set's means and stds, then dividing them by the total count to get the average values.",
+ "type": "comment"
+ },
+ "8336": {
+ "file_id": 616,
+ "content": " paddle.pow((paddle.reshape(means, (n, -1)) - mean), 2),\n (n, -1)),\n axis=0) / n)\n return mean, std\n def aggregate_stats(self):\n \"\"\"\n Synchronize running_mean, and running_var to self.bn.\n Call this before eval, then call model.eval();\n When eval, forward function will call self.bn instead of self.split_bn,\n During this time the running_mean, and running_var of self.bn has been obtained from\n self.split_bn.\n \"\"\"\n if self.split_bn.training:\n bn_mean_tensor, bn_variance_tensor = self._get_aggregated_mean_std(\n self.split_bn._mean,\n self.split_bn._variance,\n self.num_splits,\n )\n self.bn._mean.set_value(bn_mean_tensor)\n self.bn._variance.set_value(bn_variance_tensor)\n def forward(self, x):\n if self.training:\n n, c, t, h, w = x.shape\n x = paddle.reshape(\n x, (n // self.num_splits, c * self.num_splits, t, h, w))",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/batchnorm_helper.py:109-135"
+ },
+ "8337": {
+ "file_id": 616,
+ "content": "This code is defining a class that implements batch normalization in PyTorch. The class has methods to compute the mean and standard deviation, aggregate statistics from multiple splits of batch normalization, and perform forward pass for training or evaluation.",
+ "type": "comment"
+ },
+ "8338": {
+ "file_id": 616,
+ "content": " x = self.split_bn(x)\n x = paddle.reshape(x, (n, c, t, h, w))\n else:\n x = self.bn(x)\n x = paddle.multiply(x, paddle.reshape(self.weight, (-1, 1, 1, 1)))\n x = paddle.add(x, paddle.reshape(self.bias, (-1, 1, 1, 1)))\n return x",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/batchnorm_helper.py:136-142"
+ },
+ "8339": {
+ "file_id": 616,
+ "content": "The code applies batch normalization to the input tensor and multiplies it by a weight matrix. Then, it adds a bias vector and returns the normalized tensor.",
+ "type": "comment"
+ },
+ "8340": {
+ "file_id": 617,
+ "content": "/paddlevideo/utils/multigrid/interval_helper.py",
+ "type": "filepath"
+ },
+ "8341": {
+ "file_id": 617,
+ "content": "The function `is_eval_epoch` determines whether the model should be evaluated at a given epoch based on the provided configs, current epoch, and multigrid training schedule. If the current epoch is equal to the total number of epochs or if there's a non-null multigrid schedule, it checks if the current epoch is a time for evaluation based on the schedule intervals. The function returns True when an evaluation should occur and False otherwise.",
+ "type": "summary"
+ },
+ "8342": {
+ "file_id": 617,
+ "content": "def is_eval_epoch(cfg, cur_epoch, total_epochs, multigrid_schedule):\n \"\"\"\n Determine if the model should be evaluated at the current epoch.\n Args:\n cfg (CfgNode): configs. Details can be found in\n slowfast/config/defaults.py\n cur_epoch (int): current epoch.\n multigrid_schedule (List): schedule for multigrid training.\n \"\"\"\n if cur_epoch + 1 == total_epochs:\n return True\n if multigrid_schedule is not None:\n prev_epoch = 0\n for s in multigrid_schedule:\n if cur_epoch < s[-1]:\n period = max(\n (s[-1] - prev_epoch) // cfg.MULTIGRID.EVAL_FREQ + 1, 1)\n return (s[-1] - 1 - cur_epoch) % period == 0\n prev_epoch = s[-1]",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/interval_helper.py:1-19"
+ },
+ "8343": {
+ "file_id": 617,
+ "content": "The function `is_eval_epoch` determines whether the model should be evaluated at a given epoch based on the provided configs, current epoch, and multigrid training schedule. If the current epoch is equal to the total number of epochs or if there's a non-null multigrid schedule, it checks if the current epoch is a time for evaluation based on the schedule intervals. The function returns True when an evaluation should occur and False otherwise.",
+ "type": "comment"
+ },
+ "8344": {
+ "file_id": 618,
+ "content": "/paddlevideo/utils/multigrid/multigrid.py",
+ "type": "filepath"
+ },
+ "8345": {
+ "file_id": 618,
+ "content": "The MultigridSchedule class manages multigrid training schedules, batch sizes, sampling rates, and long cycle updates. The update_long_cycle() function adjusts these parameters based on the epoch in PaddleVideo. It also calculates final learning rate schedules and provides a function for determining long cycle base shape.",
+ "type": "summary"
+ },
+ "8346": {
+ "file_id": 618,
+ "content": "\"\"\"Functions for multigrid training.\"\"\"\nimport numpy as np\nclass MultigridSchedule(object):\n \"\"\"\n This class defines multigrid training schedule and update cfg accordingly.\n \"\"\"\n def init_multigrid(self, cfg):\n \"\"\"\n Update cfg based on multigrid settings.\n Args:\n cfg (configs): configs that contains training and multigrid specific\n hyperparameters.\n Returns:\n cfg (configs): the updated cfg.\n \"\"\"\n self.schedule = None\n # We may modify cfg.DATASET.batch_size, cfg.PIPELINE.train.decode_sampler.num_frames, and\n # cfg.PIPELINE.train.transform[1]['MultiCrop']['target_size'] during training, so we store their original\n # value in cfg and use them as global variables.\n cfg.MULTIGRID.default_batch_size = cfg.DATASET.batch_size # total bs,64\n cfg.MULTIGRID.default_temporal_size = cfg.PIPELINE.train.decode_sampler.num_frames # 32\n cfg.MULTIGRID.default_crop_size = cfg.PIPELINE.train.transform[1][",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:1-25"
+ },
+ "8347": {
+ "file_id": 618,
+ "content": "This code defines a MultigridSchedule class for multigrid training schedule and updates cfg according to multigrid settings. The init_multigrid function takes in configs (cfg) as input, updates it based on multigrid settings, and returns the updated cfg. It stores original values of batch size, temporal size, and crop size in cfg's MULTIGRID subsection as global variables for later use.",
+ "type": "comment"
+ },
+ "8348": {
+ "file_id": 618,
+ "content": " 'MultiCrop']['target_size'] # 224\n if cfg.MULTIGRID.LONG_CYCLE:\n self.schedule = self.get_long_cycle_schedule(cfg)\n cfg.OPTIMIZER.learning_rate.steps = [0] + [\n s[-1] for s in self.schedule\n ]\n # Fine-tuning phase.\n cfg.OPTIMIZER.learning_rate.steps[-1] = (\n cfg.OPTIMIZER.learning_rate.steps[-2] +\n cfg.OPTIMIZER.learning_rate.steps[-1]) // 2\n cfg.OPTIMIZER.learning_rate.lrs = [\n cfg.OPTIMIZER.learning_rate.gamma**s[0] * s[1][0]\n for s in self.schedule\n ]\n # Fine-tuning phase.\n cfg.OPTIMIZER.learning_rate.lrs = cfg.OPTIMIZER.learning_rate.lrs[:-1] + [\n cfg.OPTIMIZER.learning_rate.lrs[-2],\n cfg.OPTIMIZER.learning_rate.lrs[-1],\n ]\n cfg.OPTIMIZER.learning_rate.max_epoch = self.schedule[-1][-1]\n elif cfg.MULTIGRID.SHORT_CYCLE:\n cfg.OPTIMIZER.learning_rate.steps = [",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:26-50"
+ },
+ "8349": {
+ "file_id": 618,
+ "content": "The code initializes the multi-grid training schedule for the given configuration (cfg). If a long cycle is enabled, it sets learning rate steps and adjusts them for fine-tuning. It also updates the maximum epoch count based on the schedule.",
+ "type": "comment"
+ },
+ "8350": {
+ "file_id": 618,
+ "content": " int(s * cfg.MULTIGRID.epoch_factor)\n for s in cfg.OPTIMIZER.learning_rate.steps\n ]\n cfg.OPTIMIZER.learning_rate.max_epoch = int(\n cfg.OPTIMIZER.learning_rate.max_epoch *\n cfg.OPTIMIZER.learning_rate.max_epoch)\n return cfg\n def update_long_cycle(self, cfg, cur_epoch):\n \"\"\"\n Before every epoch, check if long cycle shape should change. If it\n should, update cfg accordingly.\n Args:\n cfg (configs): configs that contains training and multigrid specific\n hyperparameters.\n cur_epoch (int): current epoch index.\n Returns:\n cfg (configs): the updated cfg.\n changed (bool): whether to change long cycle shape at this epoch\n \"\"\"\n base_b, base_t, base_s = get_current_long_cycle_shape(\n self.schedule, cur_epoch)\n if base_s != cfg.PIPELINE.train.transform[1]['MultiCrop'][\n 'target_size'] or base_t != cfg.PIPELINE.train.decode_sampler.num_frames:",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:51-74"
+ },
+ "8351": {
+ "file_id": 618,
+ "content": "This function, update_long_cycle(), checks if the long cycle shape should change before every epoch. If it should, it updates cfg accordingly. It takes in configs (cfg) and current epoch index (cur_epoch), and returns the updated cfg and a boolean indicating whether the long cycle shape changed. The function also retrieves the base_b, base_t, and base_s using get_current_long_cycle_shape(). If these values differ from the target size or number of frames in the cfg, it implies that the long cycle shape should change.",
+ "type": "comment"
+ },
+ "8352": {
+ "file_id": 618,
+ "content": " #NOTE Modify\n # no need to modify, used by pool_size in head, None when multigrid\n # cfg.MODEL.head.num_frames = base_t\n # cfg.MODEL.head.crop_size = base_s\n cfg.PIPELINE.train.decode_sampler.num_frames = base_t\n cfg.PIPELINE.train.transform[1]['MultiCrop']['target_size'] = base_s\n cfg.DATASET.batch_size = base_b * cfg.MULTIGRID.default_batch_size #change bs\n bs_factor = (float(cfg.DATASET.batch_size) /\n cfg.MULTIGRID.bn_base_size)\n if bs_factor == 1: #single bs == bn_base_size (== 8)\n cfg.MODEL.backbone.bn_norm_type = \"batchnorm\"\n else:\n cfg.MODEL.backbone.bn_norm_type = \"sub_batchnorm\"\n cfg.MODEL.backbone.bn_num_splits = int(bs_factor)\n cfg.MULTIGRID.long_cycle_sampling_rate = cfg.PIPELINE.train.decode_sampler.sampling_rate * (\n cfg.MULTIGRID.default_temporal_size // base_t)\n print(\"Long cycle updates:\")",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:75-94"
+ },
+ "8353": {
+ "file_id": 618,
+ "content": "This code sets the number of frames and crop size for the head and transform, adjusts batch size based on multigrid configuration, determines whether to use \"batchnorm\" or \"sub_batchnorm\", and sets the long cycle sampling rate. The output is a message stating if long cycle updates are enabled.",
+ "type": "comment"
+ },
+ "8354": {
+ "file_id": 618,
+ "content": " print(\"\\tbn_norm_type: {}\".format(cfg.MODEL.backbone.bn_norm_type))\n if cfg.MODEL.backbone.bn_norm_type == \"sub_batchnorm\":\n print(\"\\tbn_num_splits: {}\".format(\n cfg.MODEL.backbone.bn_num_splits))\n print(\"\\tTRAIN.batch_size[single card]: {}\".format(\n cfg.DATASET.batch_size))\n print(\"\\tDATA.NUM_FRAMES x LONG_CYCLE_SAMPLING_RATE: {}x{}\".format(\n cfg.PIPELINE.train.decode_sampler.num_frames,\n cfg.MULTIGRID.long_cycle_sampling_rate))\n print(\"\\tDATA.train_crop_size: {}\".format(\n cfg.PIPELINE.train.transform[1]['MultiCrop']['target_size']))\n return cfg, True\n else:\n return cfg, False\n def get_long_cycle_schedule(self, cfg):\n \"\"\"\n Based on multigrid hyperparameters, define the schedule of a long cycle.\n Args:\n cfg (configs): configs that contains training and multigrid specific\n hyperparameters.",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:95-115"
+ },
+ "8355": {
+ "file_id": 618,
+ "content": "The code is a function that checks the configuration for certain parameters related to multigrid training. It prints specific values and returns two values: a boolean indicating if the long cycle schedule should be used, and the original config unchanged.",
+ "type": "comment"
+ },
+ "8356": {
+ "file_id": 618,
+ "content": " Returns:\n schedule (list): Specifies a list long cycle base shapes and their\n corresponding training epochs.\n \"\"\"\n steps = cfg.OPTIMIZER.learning_rate.steps\n default_size = float(\n cfg.PIPELINE.train.decode_sampler.num_frames *\n cfg.PIPELINE.train.transform[1]['MultiCrop']['target_size']**\n 2) # 32 * 224 * 224 C*H*W\n default_iters = steps[-1] # 196\n # Get shapes and average batch size for each long cycle shape.\n avg_bs = []\n all_shapes = []\n # for t_factor, s_factor in cfg.MULTIGRID.long_cycle_factors:\n for item in cfg.MULTIGRID.long_cycle_factors:\n t_factor, s_factor = item[\"value\"]\n base_t = int(\n round(cfg.PIPELINE.train.decode_sampler.num_frames * t_factor))\n base_s = int(\n round(\n cfg.PIPELINE.train.transform[1]['MultiCrop']['target_size']\n * s_factor))\n if cfg.MULTIGRID.SHORT_CYCLE:",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:116-141"
+ },
+ "8357": {
+ "file_id": 618,
+ "content": "This code calculates the schedule for multi-grid training, iterating over long cycle factor pairs in `cfg.MULTIGRID.long_cycle_factors`. It determines base shapes for each cycle, calculating `base_t` based on `cfg.PIPELINE.train.decode_sampler.num_frames` and `t_factor`, and `base_s` based on target size from `cfg.PIPELINE.train.transform[1]['MultiCrop']['target_size']` and `s_factor`. It also considers short cycle training flag, `cfg.MULTIGRID.SHORT_CYCLE`.",
+ "type": "comment"
+ },
+ "8358": {
+ "file_id": 618,
+ "content": " shapes = [\n [\n base_t,\n cfg.MULTIGRID.default_crop_size *\n cfg.MULTIGRID.short_cycle_factors[0],\n ],\n [\n base_t,\n cfg.MULTIGRID.default_crop_size *\n cfg.MULTIGRID.short_cycle_factors[1],\n ],\n [base_t, base_s],\n ] #first two is short_cycle, last is the base long_cycle\n else:\n shapes = [[base_t, base_s]]\n # (T, S) -> (B, T, S)\n shapes = [[\n int(round(default_size / (s[0] * s[1] * s[1]))), s[0], s[1]\n ] for s in shapes]\n avg_bs.append(np.mean([s[0] for s in shapes]))\n all_shapes.append(shapes)\n # Get schedule regardless of cfg.MULTIGRID.epoch_factor.\n total_iters = 0\n schedule = []\n for step_index in range(len(steps) - 1):\n step_epochs = steps[step_index + 1] - steps[step_index]",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:142-169"
+ },
+ "8359": {
+ "file_id": 618,
+ "content": "This code defines the multigrid training schedule for PaddleVideo. It sets the shapes for different grid levels, converts them to batch sizes, and calculates the average batch size. The code then computes the total number of iterations and generates the multigrid training schedule based on the steps provided.",
+ "type": "comment"
+ },
+ "8360": {
+ "file_id": 618,
+ "content": " for long_cycle_index, shapes in enumerate(all_shapes):\n #ensure each of 4 sequences run the same num of iters\n cur_epochs = (step_epochs * avg_bs[long_cycle_index] /\n sum(avg_bs))\n # get cur_iters from cur_epochs\n cur_iters = cur_epochs / avg_bs[long_cycle_index]\n total_iters += cur_iters\n schedule.append((step_index, shapes[-1], cur_epochs))\n iter_saving = default_iters / total_iters # ratio between default iters and real iters\n final_step_epochs = cfg.OPTIMIZER.learning_rate.max_epoch - steps[-1]\n # We define the fine-tuning phase to have the same amount of iteration\n # saving as the rest of the training.\n #final_step_epochs / iter_saving make fine-tune having the same iters as training\n ft_epochs = final_step_epochs / iter_saving * avg_bs[-1]\n # schedule.append((step_index + 1, all_shapes[-1][2], ft_epochs))\n schedule.append((step_index + 1, all_shapes[-1][-1], ft_epochs))",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:171-191"
+ },
+ "8361": {
+ "file_id": 618,
+ "content": "This code calculates the number of iterations for each sequence based on average batch sizes, and then appends the schedule with corresponding step index, shape, and epochs. It also ensures that the fine-tuning phase has the same number of iterations as the rest of the training.",
+ "type": "comment"
+ },
+ "8362": {
+ "file_id": 618,
+ "content": " # Obtrain final schedule given desired cfg.MULTIGRID.epoch_factor.\n x = (cfg.OPTIMIZER.learning_rate.max_epoch *\n cfg.MULTIGRID.epoch_factor / sum(s[-1] for s in schedule))\n final_schedule = []\n total_epochs = 0\n for s in schedule:\n epochs = s[2] * x\n total_epochs += epochs\n final_schedule.append((s[0], s[1], int(round(total_epochs))))\n print_schedule(final_schedule)\n return final_schedule\ndef print_schedule(schedule):\n \"\"\"\n Log schedule.\n \"\"\"\n print(\n \"Long_cycle_index\\tBase_shape(bs_factor,temporal_size,crop_size)\\tEpochs\"\n )\n for s in schedule:\n print(\"{}\\t\\t\\t{}\\t\\t\\t\\t\\t{}\".format(s[0], s[1], s[2]))\ndef get_current_long_cycle_shape(schedule, epoch):\n \"\"\"\n Given a schedule and epoch index, return the long cycle base shape.\n Args:\n schedule (configs): configs that contains training and multigrid specific\n hyperparameters.\n cur_epoch (int): current epoch index.",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:193-224"
+ },
+ "8363": {
+ "file_id": 618,
+ "content": "This code calculates the final learning rate schedule for multigrid training based on a provided schedule, max_epoch, and epoch_factor. It then prints this new schedule. The function get_current_long_cycle_shape takes in this same schedule and current epoch index to return the long cycle base shape for the given epoch.",
+ "type": "comment"
+ },
+ "8364": {
+ "file_id": 618,
+ "content": " Returns:\n shapes (list): A list describing the base shape in a long cycle:\n [batch size relative to default,\n number of frames, spatial dimension].\n \"\"\"\n for s in schedule:\n if epoch < s[-1]:\n return s[1]\n return schedule[-1][1]",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/multigrid.py:225-233"
+ },
+ "8365": {
+ "file_id": 618,
+ "content": "This function returns a list describing the base shape in a long cycle based on the current epoch and a given schedule. It iterates through the schedule, returning the appropriate shape if the current epoch is less than the scheduled value, otherwise it returns the last shape in the schedule.",
+ "type": "comment"
+ },
+ "8366": {
+ "file_id": 619,
+ "content": "/paddlevideo/utils/multigrid/save_load_helper.py",
+ "type": "filepath"
+ },
+ "8367": {
+ "file_id": 619,
+ "content": "This function ensures state dict consistency by comparing optimizer and model parameters, saving/loading checkpoints, and converting sub-bn to normal bn. It checks if certain layers are set to load and prints a message for unloaded weights before loading pre-trained weights and setting the optimizer's state dictionary.",
+ "type": "summary"
+ },
+ "8368": {
+ "file_id": 619,
+ "content": "import os\nimport numpy as np\nimport paddle\nimport copy\ndef sub_to_normal_bn(sd):\n \"\"\"\n When save, Convert the Sub-BN paprameters to normal BN parameters in a state dict.\n There are two copies of BN layers in a Sub-BN implementation: `bn.bn` and\n `bn.split_bn`. `bn.split_bn` is used during training and\n \"compute_precise_bn\". Before saving or evaluation, its stats are copied to\n `bn.bn`. We rename `bn.bn` to `bn` and store it to be consistent with normal\n BN layers.\n Args:\n sd (OrderedDict): a dict of parameters which might contain Sub-BN\n parameters.\n Returns:\n new_sd (OrderedDict): a dict with Sub-BN parameters reshaped to\n normal parameters.\n \"\"\"\n modifications = [\n (\"bn.bn._mean\", \"bn._mean\"),\n (\"bn.bn._variance\", \"bn._variance\"),\n ]\n to_remove = [\"bn.bn.\", \".split_bn.\"]\n key_list = list(sd.keys()) #odict_keys to list\n for key in key_list:\n for before, after in modifications:\n if key.endswith(before):\n new_key = key.split(before)[0] + after",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:1-31"
+ },
+ "8369": {
+ "file_id": 619,
+ "content": "This function converts Sub-BN parameters to normal BN parameters in a state dict. It renames `bn.bn` to `bn`, and modifies `_mean` and `_variance` accordingly. This is done before saving or evaluation to maintain consistency with normal BN layers. The modifications are made by iterating through the dictionary and checking if the key ends with the appropriate string, then updating it accordingly.",
+ "type": "comment"
+ },
+ "8370": {
+ "file_id": 619,
+ "content": " sd[new_key] = sd.pop(key)\n for rm in to_remove:\n if rm in key and key in sd:\n del sd[key]\ndef normal_to_sub_bn(checkpoint_sd, model_sd):\n \"\"\"\n When load, Convert BN parameters to Sub-BN parameters if model contains Sub-BNs.\n Args:\n checkpoint_sd (OrderedDict): source dict of parameters.\n model_sd (OrderedDict): target dict of parameters.\n Returns:\n new_sd (OrderedDict): converted dict of parameters.\n \"\"\"\n for key in model_sd:\n if key not in checkpoint_sd:\n # not to replace bn.weight and bn.bias\n if \"bn.split_bn.\" in key and \"bn.weight\" not in key and \"bn.bias\" not in key:\n load_key = key.replace(\"bn.split_bn.\", \"bn.\")\n bn_key = key.replace(\"bn.split_bn.\", \"bn.bn.\")\n checkpoint_sd[key] = checkpoint_sd.pop(load_key)\n checkpoint_sd[bn_key] = checkpoint_sd[key]\n # match the shape of bn.split_bn._xx\n # model_sd: split_bn.rm.shape = num_feature*num_split",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:32-58"
+ },
+ "8371": {
+ "file_id": 619,
+ "content": "This function converts BN parameters to Sub-BN parameters when loading a checkpoint into a model containing Sub-BNs. It loops through the model's parameters, if a parameter has the \"bn.split_bn.\" prefix and is not the weight or bias of BN, it renames and moves the corresponding value from the checkpoint dict to the bn.bn key in the same subdict. Finally, it adjusts the shape of the Sub-BN parameters to match the original BN parameters' shape.",
+ "type": "comment"
+ },
+ "8372": {
+ "file_id": 619,
+ "content": " # checkpoint_sd: split_bn.rm.shape = bn.rm.shape = num_feature\n for key in model_sd:\n if key in checkpoint_sd:\n model_blob_shape = model_sd[key].shape #bn.split_bn\n c2_blob_shape = checkpoint_sd[key].shape #bn.bn\n if (len(model_blob_shape) == 1 and len(c2_blob_shape) == 1\n and model_blob_shape[0] > c2_blob_shape[0]\n and model_blob_shape[0] % c2_blob_shape[0] == 0):\n before_shape = checkpoint_sd[key].shape\n checkpoint_sd[key] = np.concatenate(\n [checkpoint_sd[key]] *\n (model_blob_shape[0] // c2_blob_shape[0]))\n if 'split_bn' not in key: #split_bn is excepted\n print(\"{} {} -> {}\".format(key, before_shape,\n checkpoint_sd[key].shape))\n return checkpoint_sd\ndef mapping_opt_dict(opt_dict, model_key_list):\n \"\"\"\n Paddle Name schedule: conv_1.w -> conv_2.w\n Sometimes: sub_bn -> bn",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:59-81"
+ },
+ "8373": {
+ "file_id": 619,
+ "content": "This code is comparing the shape of certain keys in the model and checkpoint dictionaries. If they match certain criteria, it will concatenate the checkpoint key to expand its size based on the model's shape. This is done for specific keys in the dictionary, except 'split_bn'. The function prints out the before and after shapes of the affected keys.",
+ "type": "comment"
+ },
+ "8374": {
+ "file_id": 619,
+ "content": " when re-build model, we desire the parameter name to be coincident,\n but the parameters name index will be added, as conv_1 to conv_2, not conv_1.\n It will raise error if we set old saved parameters to new created optimizer.\n as conv_2 cannot find in state_dict(only conv_1).\n Args:\n opt_dict: optimizer state dict, including the name and value of parameters gradient.\n model_key_list: the parameters name list of re-build model.\n Return: optimizer state dict with modified keys\n \"\"\"\n def get_name_info(PNAME, PN_key_list, key_list):\n min_index = float('inf')\n max_index = 0\n for name in PN_key_list[1:]:\n for key in key_list:\n if name in key:\n index = int(key.split('.')[0].split(name)[-1])\n if index < min_index:\n min_index = index\n if index > max_index:\n max_index = index\n num_name = max_index - min_index + 1\n PNAME[name].append((min_index, max_index, num_name))",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:82-103"
+ },
+ "8375": {
+ "file_id": 619,
+ "content": "This function takes an optimizer state dict and a list of parameter names from a rebuilt model. It aims to modify the keys in the optimizer state dict to match the new parameters' names, while also considering any added index for better compatibility. The function then returns the modified optimizer state dict.",
+ "type": "comment"
+ },
+ "8376": {
+ "file_id": 619,
+ "content": " min_index = float('inf')\n max_index = 0\n PNAME = {\n \"LR_Scheduler\": [],\n \"conv3d_\": [],\n \"linear_\": [],\n \"sub_batch_norm3d_\": [],\n \"batch_norm3d_\": [],\n }\n pd_key_list = list(opt_dict.keys())\n print(\"The number of parameters in saved optimizer state dict = {}\".format(\n len(pd_key_list)))\n print(\"The number of parameters in re-build model list = {}\".format(\n len(model_key_list)))\n # 1 may be LR_Scheduler\n PN_key_list = list(PNAME.keys())\n # get the number of each PNAME\n get_name_info(PNAME, PN_key_list, pd_key_list)\n get_name_info(PNAME, PN_key_list, model_key_list)\n print(\"[Parameters info] prefix: min_index, max_index, number_params: \\n\",\n PNAME)\n # whether to change name of bn layer\n change_name = False\n if PNAME[\"sub_batch_norm3d_\"][0][-1] == -float('inf'):\n PN_key_list.remove(\"sub_batch_norm3d_\")\n if PNAME[\"sub_batch_norm3d_\"][1][-1] != -float('inf'):\n print(\n \"Optimizer state dict saved bn, but Re-build model use sub_bn, changed name!\"",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:104-135"
+ },
+ "8377": {
+ "file_id": 619,
+ "content": "This code appears to be a part of a larger program that compares the parameters in an optimizer state dict with those in a re-built model. It calculates and prints information about the number of parameters associated with each prefix, checks if batch normalization layers need their names changed, and potentially removes the \"sub_batch_norm3d_\" prefix from consideration. The code assumes that the \"opt_dict\" and \"model\" variables have already been defined elsewhere.",
+ "type": "comment"
+ },
+ "8378": {
+ "file_id": 619,
+ "content": " )\n change_name = True\n else:\n print(\"Optimizer state dict saved bn, and Re-build model use bn\")\n else:\n PN_key_list.remove(\"batch_norm3d_\")\n if PNAME[\"sub_batch_norm3d_\"][1][-1] == -float('inf'):\n print(\n \"Optimizer state dict saved sub_bn, but Re-build model use bn, changed name!\"\n )\n change_name = True\n else:\n print(\n \"Optimizer state dict saved sub_bn, Re-build model use sub_bn\")\n #update key name\n # sub_bn -> bn name mapping, pre-define dict\n change_dict = {\n \"sub_batch_norm3d_\": \"batch_norm3d_\",\n \"batch_norm3d_\": \"sub_batch_norm3d_\"\n }\n for key in pd_key_list:\n for name in PN_key_list[1:]:\n if key.startswith(name):\n start = change_dict[name] if (\n change_name and \"batch_norm\" in name) else name\n str_index = key.split('.')[0].split(name)[-1]\n index = int(str_index)",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:136-163"
+ },
+ "8379": {
+ "file_id": 619,
+ "content": "The code checks if the optimizer state dict saved batch normalization (bn) or sub_batch_normalization and updates the key names accordingly. If the state dict saved bn but the model uses sub_bn, it prints a message and changes the name. If the state dict saved sub_bn and the model also uses sub_bn, it prints a separate message. The code then defines a change_dict mapping and iterates over the key list to update the names if required.",
+ "type": "comment"
+ },
+ "8380": {
+ "file_id": 619,
+ "content": " new_index = str(index +\n (PNAME[start][1][0] - PNAME[name][0][0]))\n end = key.split('.')[-1]\n update_key = start + new_index + '.' + end\n opt_dict[update_key] = opt_dict.pop(key)\n return opt_dict\ndef subn_save(save_dir, name_prefix, epoch, video_model, optimizer):\n if not os.path.isdir(save_dir):\n os.makedirs(save_dir)\n model_path = os.path.join(save_dir, name_prefix + \"{:05d}\".format(epoch))\n model_dict = video_model.state_dict()\n sub_to_normal_bn(model_dict)\n opti_dict = optimizer.state_dict()\n paddle.save(model_dict, model_path + '.pdparams')\n paddle.save(opti_dict, model_path + '.pdopt')\n print('[Saved Epoch {} parameters and optimizer state ]'.format(epoch))\ndef subn_load(model, ck_path, optimizer=None):\n \"\"\"\n Load the checkpoint from the given file.\n Args:\n model (model): model to load the weights from the checkpoint.\n optimizer (optim, optional): optimizer to load the historical state.",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:164-190"
+ },
+ "8381": {
+ "file_id": 619,
+ "content": "This code defines two functions: \"subn_save\" and \"subn_load\". \"subn_save\" saves a model's state dictionary along with the optimizer's state dictionary to specified directories in a specific format. It also converts sub-bn to normal bn before saving, and prints a message confirming the save operation. \"subn_load\" loads checkpoints from given files into the specified model and optionally an optimizer.",
+ "type": "comment"
+ },
+ "8382": {
+ "file_id": 619,
+ "content": " ck_path (str): checkpoint path\n Returns:\n (int): the number of training epoch of the checkpoint.\n \"\"\"\n assert os.path.exists(ck_path + \".pdparams\"), \\\n \"Given dir {}.pdparams not exist.\".format(ck_path)\n print(\"load checkpint from {}.pdparams\".format(ck_path))\n model_dict = model.state_dict()\n checkpoint_dict = paddle.load(ck_path + \".pdparams\")\n # checkpoint_dict = copy.deepcopy(checkpoint_dict_orig) #not modify when multi card\n pre_train_dict = normal_to_sub_bn(checkpoint_dict, model_dict)\n # Match pre-trained weights that have same shape as current model.\n pre_train_dict_match = {\n k: v\n for k, v in pre_train_dict.items()\n if k in model_dict and tuple(v.shape) == tuple(model_dict[k].shape)\n }\n # Weights that do not have match from the pre-trained model.\n not_load_layers = [\n k for k in model_dict.keys() if k not in pre_train_dict_match.keys()\n ]\n # Log weights that are not loaded with the pre-trained weights.",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:191-216"
+ },
+ "8383": {
+ "file_id": 619,
+ "content": "This function loads checkpoints from a specific path and returns the number of training epochs. It ensures that the given directory has .pdparams file, prints the checkpoint loading information, copies model state dictionary, and compares the shapes of pre-trained weights to current model weights for matching. It also identifies layers that are not loaded with pre-trained weights.",
+ "type": "comment"
+ },
+ "8384": {
+ "file_id": 619,
+ "content": " if not_load_layers:\n for k in not_load_layers:\n if 'bn.weight' not in k and 'bn.bias' not in k:\n print(\"Network weights {} not loaded.\".format(k))\n # Load pre-trained weights.\n model.set_state_dict(pre_train_dict_match)\n if optimizer:\n assert os.path.exists(ck_path + \".pdopt\"), \\\n \"Given dir {}.pdopt not exist.\".format(ck_path)\n print(\"load checkpint from {}.pdopt\".format(ck_path))\n opt_dict = paddle.load(ck_path + \".pdopt\")\n # get parameters that required gradient from re-build model\n model_key_list = []\n for param in model.parameters():\n if param.stop_gradient == False:\n model_key_list.append(param.name)\n new_opt_dict = mapping_opt_dict(opt_dict, model_key_list)\n optimizer.set_state_dict(new_opt_dict)",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/save_load_helper.py:217-237"
+ },
+ "8385": {
+ "file_id": 619,
+ "content": "This code block checks if certain layers in the model are not set to load, and prints a message if those weights are not loaded. It then loads the pre-trained weights for the model and checks if a specific file exists before loading the optimizer's state dictionary from that file. The function mapping_opt_dict is called to create a new dictionary containing only parameters that require gradient, which is then set as the state dictionary of the optimizer.",
+ "type": "comment"
+ },
+ "8386": {
+ "file_id": 620,
+ "content": "/paddlevideo/utils/multigrid/short_sampler.py",
+ "type": "filepath"
+ },
+ "8387": {
+ "file_id": 620,
+ "content": "DistributedShortSampler streamlines distributed data loading, dynamic batch sizes, and GPU support for PaddleVideo's multigrid. It efficiently calculates average batch size and offers sample dropping options.",
+ "type": "summary"
+ },
+ "8388": {
+ "file_id": 620,
+ "content": "from __future__ import print_function\nfrom __future__ import division\nimport numpy as np\nimport math\nimport paddle\n__all__ = [\"DistributedShortSampler\"]\nclass DistributedShortSampler(paddle.io.BatchSampler):\n \"\"\"Sampler that restricts data loading to a subset of the dataset.\n In such case, each process can pass a DistributedBatchSampler instance\n as a DataLoader sampler, and load a subset of the original dataset that\n is exclusive to it.\n .. note::\n Batch size is dynamic changed following short cycle schedule.\n Args:\n dataset(paddle.io.Dataset): this could be a `paddle.io.Dataset` implement\n or other python object which implemented\n `__len__` for BatchSampler to get sample\n number of data source.\n batch_sizes(list): batch size list of one cycle.\n num_replicas(int, optional): porcess number in distributed training.\n If :attr:`num_replicas` is None, :attr:`num_replicas` will be\n retrieved from :code:`paddle.fluid.dygraph.parallel.ParallenEnv`.",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/short_sampler.py:1-28"
+ },
+ "8389": {
+ "file_id": 620,
+ "content": "The code defines a DistributedShortSampler class which is a sampler for restricting data loading to a subset of the dataset in distributed training. It allows each process to load exclusive subsets by passing the DistributedBatchSampler as a DataLoader sampler and supports dynamic batch size changes following short cycle schedules. The class takes in a dataset, batch_sizes list, and optionally num_replicas (process number in distributed training).",
+ "type": "comment"
+ },
+ "8390": {
+ "file_id": 620,
+ "content": " Default None.\n rank(int, optional): the rank of the current process among :attr:`num_replicas`\n processes. If :attr:`rank` is None, :attr:`rank` is retrieved from\n :code:`paddle.fluid.dygraph.parallel.ParallenEnv`. Default None.\n shuffle(bool): whther to shuffle indices order before genrating\n batch indices. Default False.\n drop_last(bool): whether drop the last incomplete batch dataset size\n is not divisible by the batch size. Default False\n \"\"\"\n def __init__(self,\n dataset,\n batch_sizes,\n num_replicas=None,\n rank=None,\n shuffle=False,\n drop_last=False):\n self.dataset = dataset\n assert any(isinstance(batch_size, int) and batch_size > 0 for batch_size in batch_sizes), \\\n \"batch_size should be a positive integer\"\n self.batch_sizes = batch_sizes\n self.len_batch_sizes = len(self.batch_sizes)\n assert isinstance(shuffle, bool), \\",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/short_sampler.py:29-51"
+ },
+ "8391": {
+ "file_id": 620,
+ "content": "The `__init__` method initializes an instance of the class with a dataset, batch sizes, number of replicas (optional), rank (optional), whether to shuffle indices (optional), and whether to drop last incomplete batch (optional). The batch_sizes should be positive integers. The method performs assertions on the inputs to ensure validity.",
+ "type": "comment"
+ },
+ "8392": {
+ "file_id": 620,
+ "content": " \"shuffle should be a boolean value\"\n self.shuffle = shuffle\n assert isinstance(drop_last, bool), \\\n \"drop_last should be a boolean number\"\n if num_replicas is not None:\n assert isinstance(num_replicas, int) and num_replicas > 0, \\\n \"num_replicas should be a positive integer\"\n self.nranks = num_replicas\n else:\n self.nranks = paddle.distributed.ParallelEnv().nranks\n if rank is not None:\n assert isinstance(rank, int) and rank >= 0, \\\n \"rank should be a non-negative integer\"\n self.local_rank = rank\n else:\n self.local_rank = paddle.distributed.ParallelEnv().local_rank\n self.drop_last = drop_last\n self.epoch = 0\n self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.nranks))\n self.total_size = self.num_samples * self.nranks\n def __iter__(self):\n num_samples = len(self.dataset)\n indices = np.arange(num_samples).tolist()",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/short_sampler.py:52-79"
+ },
+ "8393": {
+ "file_id": 620,
+ "content": "The code initializes a MultigridSampler object, which manages the sampling of data across multiple ranks in distributed training. It checks for valid input values (boolean for shuffle and drop_last) and ensures positive integer for num_replicas. It determines the number of ranks and local rank based on provided values or environment. The total number of samples is calculated based on the dataset size and number of ranks, and an array of indices is created.",
+ "type": "comment"
+ },
+ "8394": {
+ "file_id": 620,
+ "content": " indices += indices[:(self.total_size -\n len(indices))] #completion last iter\n assert len(indices) == self.total_size\n if self.shuffle:\n np.random.RandomState(self.epoch).shuffle(indices)\n self.epoch += 1\n # subsample\n def _get_indices_by_batch_size(indices):\n total_batch_size = sum(self.batch_sizes)\n subsampled_indices = []\n last_batch_size = self.total_size % (\n total_batch_size * self.nranks) #number samples of last batch\n assert last_batch_size % self.nranks == 0\n last_local_batch_size = last_batch_size // self.nranks\n for i in range(self.local_rank * total_batch_size,\n len(indices) - last_batch_size,\n total_batch_size * self.nranks):\n subsampled_indices.extend(indices[i:i + total_batch_size])\n indices = indices[len(indices) - last_batch_size:]\n subsampled_indices.extend(",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/short_sampler.py:80-102"
+ },
+ "8395": {
+ "file_id": 620,
+ "content": "This code ensures that the number of samples selected is equal to the total size, and then subsamples them by batch sizes. It handles the last batch with potentially fewer samples due to modulo operations and shuffles the indices if desired.",
+ "type": "comment"
+ },
+ "8396": {
+ "file_id": 620,
+ "content": " indices[self.local_rank *\n last_local_batch_size:(self.local_rank + 1) *\n last_local_batch_size])\n return subsampled_indices\n if self.nranks > 1:\n indices = _get_indices_by_batch_size(indices)\n assert len(indices) == self.num_samples #index length in each card\n _sample_iter = iter(indices)\n batch_indices = []\n counter = 0\n batch_size = self.batch_sizes[0]\n for idx in _sample_iter:\n batch_indices.append(\n (idx, counter %\n self.len_batch_sizes)) #to be used in dataloader get_item\n if len(batch_indices) == batch_size:\n yield batch_indices\n counter += 1\n batch_size = self.batch_sizes[counter % self.len_batch_sizes]\n batch_indices = []\n if not self.drop_last and len(batch_indices) > 0:\n yield batch_indices\n def __len__(self):\n avg_batch_size = sum(self.batch_sizes) / float(self.len_batch_sizes)",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/short_sampler.py:103-130"
+ },
+ "8397": {
+ "file_id": 620,
+ "content": "This code is responsible for creating a sampler that supports dynamic batch sizes. It first sub-samples the input indices based on the local rank and local batch size. Then, it handles cases with multiple GPUs (ranks > 1), dividing the indices into batches of uniform size. Finally, it yields these batches until all samples have been used, or if the drop_last flag is set to False, it yields remaining samples even if they don't form a full batch. The average batch size is also calculated and stored in the class variable avg_batch_size.",
+ "type": "comment"
+ },
+ "8398": {
+ "file_id": 620,
+ "content": " if self.drop_last:\n return int(np.floor(self.num_samples / avg_batch_size))\n else:\n return int(np.ceil(self.num_samples / avg_batch_size))\n def set_epoch(self, epoch):\n \"\"\"\n Sets the epoch number. When :attr:`shuffle=True`, this number is used\n as seeds of random numbers. By default, users may not set this, all\n replicas (workers) use a different random ordering for each epoch.\n If set same number at each epoch, this sampler will yield the same\n ordering at all epoches.\n Arguments:\n epoch (int): Epoch number.\n \"\"\"\n self.epoch = epoch",
+ "type": "code",
+ "location": "/paddlevideo/utils/multigrid/short_sampler.py:131-146"
+ },
+ "8399": {
+ "file_id": 620,
+ "content": "This code defines a class for a sampler that can be used with PaddleVideo's multigrid. It calculates the number of samples to return based on batch size and either rounds down or up depending on whether drop_last is set. The set_epoch method sets the epoch number and, when shuffle is True, uses it as seeds for random numbers. This can result in the same ordering being yielded at all epochs if the same number is set each time.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/84.json b/docs/data/84.json
new file mode 100644
index 000000000..1a7d8c919
--- /dev/null
+++ b/docs/data/84.json
@@ -0,0 +1,543 @@
+{
+ "8400": {
+ "file_id": 621,
+ "content": "/paddlevideo/utils/precise_bn.py",
+ "type": "filepath"
+ },
+ "8401": {
+ "file_id": 621,
+ "content": "This code defines a function for precise batch normalization that recomputes and updates BN statistics, improving accuracy while speeding up training and saving memory.",
+ "type": "summary"
+ },
+ "8402": {
+ "file_id": 621,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport itertools\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n\"\"\"\nImplement precise bn, which is useful for improving accuracy.\n\"\"\"\n@paddle.no_grad() # speed up and save CUDA memory\ndef do_preciseBN(model,\n data_loader,\n parallel,\n num_iters=200,\n use_amp=False,\n amp_level=None):\n \"\"\"\n Recompute and update the batch norm stats to make them more precise. During",
+ "type": "code",
+ "location": "/paddlevideo/utils/precise_bn.py:1-34"
+ },
+ "8403": {
+ "file_id": 621,
+ "content": "The code is importing necessary libraries and defining a function for the precise batch normalization (BN) technique. This BN improves accuracy by recomputing and updating batch norm statistics to make them more precise, which can speed up training and save memory. The function takes in a model, data loader, parallel flag, number of iterations, whether to use automatic mixed precision, and the AMP level.",
+ "type": "comment"
+ },
+ "8404": {
+ "file_id": 621,
+ "content": " training both BN stats and the weight are changing after every iteration, so\n the running average can not precisely reflect the actual stats of the\n current model.\n In this function, the BN stats are recomputed with fixed weights, to make\n the running average more precise. Specifically, it computes the true average\n of per-batch mean/variance instead of the running average.\n This is useful to improve validation accuracy.\n Args:\n model: the model whose bn stats will be recomputed\n data_loader: an iterator. Produce data as input to the model\n num_iters: number of iterations to compute the stats.\n Return:\n the model with precise mean and variance in bn layers.\n \"\"\"\n bn_layers_list = [\n m for m in model.sublayers()\n if any((isinstance(m, bn_type)\n for bn_type in (paddle.nn.BatchNorm1D, paddle.nn.BatchNorm2D,\n paddle.nn.BatchNorm3D))) and m.training\n ]\n if len(bn_layers_list) == 0:\n return",
+ "type": "code",
+ "location": "/paddlevideo/utils/precise_bn.py:35-56"
+ },
+ "8405": {
+ "file_id": 621,
+ "content": "This function recomputes the BN stats for a given model using fixed weights to improve validation accuracy. It targets specific BN layers and runs iterations to compute precise mean and variance values. This is useful when training both BN stats and weights are changing with every iteration, affecting running averages.",
+ "type": "comment"
+ },
+ "8406": {
+ "file_id": 621,
+ "content": " # moving_mean=moving_mean*momentum+batch_mean*(1.−momentum)\n # we set momentum=0. to get the true mean and variance during forward\n momentum_actual = [bn._momentum for bn in bn_layers_list]\n for bn in bn_layers_list:\n bn._momentum = 0.\n running_mean = [paddle.zeros_like(bn._mean)\n for bn in bn_layers_list] # pre-ignore\n running_var = [paddle.zeros_like(bn._variance) for bn in bn_layers_list]\n ind = -1\n for ind, data in enumerate(itertools.islice(data_loader, num_iters)):\n logger.info(\"Computing precise BN {} / {}...\".format(\n ind + 1, num_iters))\n if use_amp:\n with paddle.amp.auto_cast(\n custom_black_list={\"reduce_mean\",\n \"conv3d\"}, level=amp_level):\n model(data, mode='train')\n else:\n model(data, mode='train')\n for i, bn in enumerate(bn_layers_list):\n # Accumulates the bn stats.\n running_mean[i] += (bn._mean - running_mean[i]) / (ind + 1)",
+ "type": "code",
+ "location": "/paddlevideo/utils/precise_bn.py:58-83"
+ },
+ "8407": {
+ "file_id": 621,
+ "content": "This code resets the momentum in Batch Normalization layers to 0, calculates precise Batch Normalization by accumulating batch means and variances across iterations, and then updates the running mean and variance.",
+ "type": "comment"
+ },
+ "8408": {
+ "file_id": 621,
+ "content": " running_var[i] += (bn._variance - running_var[i]) / (ind + 1)\n assert ind == num_iters - 1, (\n \"update_bn_stats is meant to run for {} iterations, but the dataloader stops at {} iterations.\"\n .format(num_iters, ind))\n # Sets the precise bn stats.\n for i, bn in enumerate(bn_layers_list):\n bn._mean.set_value(running_mean[i])\n bn._variance.set_value(running_var[i])\n bn._momentum = momentum_actual[i]",
+ "type": "code",
+ "location": "/paddlevideo/utils/precise_bn.py:84-94"
+ },
+ "8409": {
+ "file_id": 621,
+ "content": "This code updates batch normalization (BN) statistics based on the running mean, variance, and momentum. It asserts that the dataloader has run for the expected number of iterations before setting these values to the BN layers.",
+ "type": "comment"
+ },
+ "8410": {
+ "file_id": 622,
+ "content": "/paddlevideo/utils/profiler.py",
+ "type": "filepath"
+ },
+ "8411": {
+ "file_id": 622,
+ "content": "This code is part of PaddleVideo's profiler module, which allows performance analysis and optimization. It initializes a profiler object and starts/stops profiling based on step ID and specified batch range, generating summary reports in ms units.",
+ "type": "summary"
+ },
+ "8412": {
+ "file_id": 622,
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport sys\nimport paddle.profiler as profiler\n# A global variable to record the number of calling times for profiler\n# functions. It is used to specify the tracing range of training steps.\n_profiler_step_id = 0\n# A global variable to avoid parsing from string every time.\n_profiler_options = None\n_prof = None\nclass ProfilerOptions(object):\n '''\n Use a string to initialize a ProfilerOptions.\n The string should be in the format: \"key1=value1;key2=value;key3=value3\".",
+ "type": "code",
+ "location": "/paddlevideo/utils/profiler.py:1-29"
+ },
+ "8413": {
+ "file_id": 622,
+ "content": "This code is a part of PaddleVideo's profiler module, which allows for performance analysis and optimization. It imports the necessary libraries, initializes global variables, and defines the ProfilerOptions class to configure profiling options using a string in key-value format.",
+ "type": "comment"
+ },
+ "8414": {
+ "file_id": 622,
+ "content": " For example:\n \"profile_path=model.profile\"\n \"batch_range=[50, 60]; profile_path=model.profile\"\n \"batch_range=[50, 60]; tracer_option=OpDetail; profile_path=model.profile\"\n ProfilerOptions supports following key-value pair:\n batch_range - a integer list, e.g. [100, 110].\n state - a string, the optional values are 'CPU', 'GPU' or 'All'. \n sorted_key - a string, the optional values are 'calls', 'total',\n 'max', 'min' or 'ave.\n tracer_option - a string, the optional values are 'Default', 'OpDetail',\n 'AllOpDetail'.\n profile_path - a string, the path to save the serialized profile data,\n which can be used to generate a timeline.\n exit_on_finished - a boolean.\n '''\n def __init__(self, options_str):\n assert isinstance(options_str, str)\n self._options = {\n 'batch_range': [10, 20],\n 'state': 'All',\n 'sorted_key': 'total',",
+ "type": "code",
+ "location": "/paddlevideo/utils/profiler.py:30-53"
+ },
+ "8415": {
+ "file_id": 622,
+ "content": "The code defines a class \"ProfilerOptions\" with options for profiling. It takes an options string as input and has attributes for batch range (default [10, 20]), state (default 'All'), sorted key (default 'total'), tracer option (default 'Default'), profile path (empty string), and exit on finished flag (False).",
+ "type": "comment"
+ },
+ "8416": {
+ "file_id": 622,
+ "content": " 'tracer_option': 'Default',\n 'profile_path': '/tmp/profile',\n 'exit_on_finished': True,\n 'timer_only': True\n }\n self._parse_from_string(options_str)\n def _parse_from_string(self, options_str):\n for kv in options_str.replace(' ', '').split(';'):\n key, value = kv.split('=')\n if key == 'batch_range':\n value_list = value.replace('[', '').replace(']', '').split(',')\n value_list = list(map(int, value_list))\n if len(value_list) >= 2 and value_list[0] >= 0 and value_list[\n 1] > value_list[0]:\n self._options[key] = value_list\n elif key == 'exit_on_finished':\n self._options[key] = value.lower() in (\"yes\", \"true\", \"t\", \"1\")\n elif key in [\n 'state', 'sorted_key', 'tracer_option', 'profile_path'\n ]:\n self._options[key] = value\n elif key == 'timer_only':\n self._options[key] = value",
+ "type": "code",
+ "location": "/paddlevideo/utils/profiler.py:54-77"
+ },
+ "8417": {
+ "file_id": 622,
+ "content": "The code defines a class with an option parser. It parses options from a string, sets batch range if present, handles exit_on_finished flag, and updates other specified options (state, sorted_key, tracer_option, profile_path, timer_only).",
+ "type": "comment"
+ },
+ "8418": {
+ "file_id": 622,
+ "content": " def __getitem__(self, name):\n if self._options.get(name, None) is None:\n raise ValueError(\n \"ProfilerOptions does not have an option named %s.\" % name)\n return self._options[name]\ndef add_profiler_step(options_str=None):\n '''\n Enable the operator-level timing using PaddlePaddle's profiler.\n The profiler uses a independent variable to count the profiler steps.\n One call of this function is treated as a profiler step.\n Args:\n profiler_options - a string to initialize the ProfilerOptions.\n Default is None, and the profiler is disabled.\n '''\n if options_str is None:\n return\n global _prof \n global _profiler_step_id\n global _profiler_options\n if _profiler_options is None:\n _profiler_options = ProfilerOptions(options_str)\n # profile : https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/performance_improving/profiling_model.html#chakanxingnengshujudetongjibiaodan\n # timer_only = True only the model's throughput and time overhead are displayed",
+ "type": "code",
+ "location": "/paddlevideo/utils/profiler.py:79-105"
+ },
+ "8419": {
+ "file_id": 622,
+ "content": "This code provides a function to enable the operator-level timing using PaddlePaddle's profiler. The profiler step is initialized with options provided as a string. If no options are given, the profiler remains disabled. This can be used for performance analysis of models by measuring their throughput and time overhead.",
+ "type": "comment"
+ },
+ "8420": {
+ "file_id": 622,
+ "content": " # timer_only = False calling summary can print a statistical form that presents performance data from different perspectives.\n # timer_only = False the output Timeline information can be found in the profiler_log directory\n if _prof is None:\n _timer_only = str(_profiler_options['timer_only']) == str(True)\n _prof = profiler.Profiler(\n scheduler = (_profiler_options['batch_range'][0], _profiler_options['batch_range'][1]),\n on_trace_ready = profiler.export_chrome_tracing('./profiler_log'),\n timer_only = _timer_only)\n _prof.start()\n else:\n _prof.step()\n if _profiler_step_id == _profiler_options['batch_range'][1]:\n _prof.stop()\n _prof.summary(\n op_detail=True,\n thread_sep=False,\n time_unit='ms')\n _prof = None\n if _profiler_options['exit_on_finished']:\n sys.exit(0)\n _profiler_step_id += 1",
+ "type": "code",
+ "location": "/paddlevideo/utils/profiler.py:106-128"
+ },
+ "8421": {
+ "file_id": 622,
+ "content": "This code initializes a profiler object with specified scheduler range and timer_only option, then starts the profiling process. If the step ID matches the specified batch range, it stops the profiling, generates a summary report in ms units, clears the profiler, and exits the program if instructed to do so.",
+ "type": "comment"
+ },
+ "8422": {
+ "file_id": 623,
+ "content": "/paddlevideo/utils/record.py",
+ "type": "filepath"
+ },
+ "8423": {
+ "file_id": 623,
+ "content": "This code records metrics, calculates means, logs batch info and epoch progress in training processes with colored formatting for visibility. It uses PaddleVideo framework, AverageMeter and OrderedDict for efficient logging.",
+ "type": "summary"
+ },
+ "8424": {
+ "file_id": 623,
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport datetime\nfrom collections import OrderedDict\nimport paddle\nfrom .logger import coloring, get_logger\nlogger = get_logger(\"paddlevideo\")\n__all__ = ['AverageMeter', 'build_record', 'log_batch', 'log_epoch']\ndef build_record(cfg):\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"lr\", AverageMeter('lr', 'f', need_avg=False)),\n ]\n if 'Recognizer1D' in cfg.framework: #TODO: required specify str in framework",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:1-32"
+ },
+ "8425": {
+ "file_id": 623,
+ "content": "Code snippet imports necessary libraries and defines functions for building a record, logging batches and epochs. It also sets up logger for the PaddleVideo framework.",
+ "type": "comment"
+ },
+ "8426": {
+ "file_id": 623,
+ "content": " record_list.append((\"hit_at_one\", AverageMeter(\"hit_at_one\", '.5f')))\n record_list.append((\"perr\", AverageMeter(\"perr\", '.5f')))\n record_list.append((\"gap\", AverageMeter(\"gap\", '.5f')))\n elif 'Recognizer' in cfg.framework:\n record_list.append((\"top1\", AverageMeter(\"top1\", '.5f')))\n record_list.append((\"top5\", AverageMeter(\"top5\", '.5f')))\n elif 'FastRCNN' in cfg.framework:\n record_list.append(\n (\"recall@thr=0.5\", AverageMeter(\"recall@thr=0.5\", '.5f')))\n record_list.append((\"prec@thr=0.5\", AverageMeter(\"prec@thr=0.5\",\n '.5f')))\n record_list.append((\"recall@top3\", AverageMeter(\"recall@top3\", '.5f')))\n record_list.append((\"prec@top3\", AverageMeter(\"prec@top3\", '.5f')))\n record_list.append((\"recall@top5\", AverageMeter(\"recall@top5\", '.5f')))\n record_list.append((\"prec@top5\", AverageMeter(\"prec@top5\", '.5f')))\n record_list.append((\"mAP@0.5IOU\", AverageMeter(\"mAP@0.5IOU\", '.5f')))",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:33-48"
+ },
+ "8427": {
+ "file_id": 623,
+ "content": "Code appends specific metrics to the record list based on the framework specified in cfg. Frameworks include 'PaddleVideo', 'Recognizer', and 'FastRCNN'. Metrics are averaged using AverageMeter and include 'hit_at_one', 'perr', 'gap', 'top1', 'top5', recall@thr=0.5, prec@thr=0.5, recall@top3, prec@top3, recall@top5, prec@top5, and mAP@0.5IOU.",
+ "type": "comment"
+ },
+ "8428": {
+ "file_id": 623,
+ "content": " elif 'DepthEstimator' in cfg.framework:\n record_list.append((\"abs_rel\", AverageMeter(\"abs_rel\", '.5f')))\n record_list.append((\"sq_rel\", AverageMeter(\"sq_rel\", '.5f')))\n record_list.append((\"rmse\", AverageMeter(\"rmse\", '.5f')))\n record_list.append((\"rmse_log\", AverageMeter(\"rmse_log\", '.5f')))\n record_list.append((\"a1\", AverageMeter(\"a1\", '.5f')))\n record_list.append((\"a2\", AverageMeter(\"a2\", '.5f')))\n record_list.append((\"a3\", AverageMeter(\"a3\", '.5f')))\n record_list.append((\"losses_day\", AverageMeter(\"losses_day\", '.5f')))\n record_list.append((\"losses_night\", AverageMeter(\"losses_night\",\n '.5f')))\n elif 'MSTCN' in cfg.framework or 'ASRF' in cfg.framework:\n record_list.append((\"F1@0.50\", AverageMeter(\"F1@0.50\", '.5f')))\n elif 'YOWOLocalizer' in cfg.framework:\n record_list.append((\"nCorrect\", AverageMeter('nCorrect', '.1f')))\n record_list.append((\"fscore\", AverageMeter(\"fscore\", '.5f')))",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:49-65"
+ },
+ "8429": {
+ "file_id": 623,
+ "content": "The code is conditionally adding metrics to the record list based on the value of 'cfg.framework'. It handles three different cases: 'DepthEstimator', 'MSTCN' or 'ASRF', and 'YOWOLocalizer'. For 'DepthEstimator', it adds 9 metrics, for 'MSTCN' or 'ASRF', it adds one metric, and for 'YOWOLocalizer', it adds two metrics. Each metric is associated with an AverageMeter object that keeps track of its mean value over time.",
+ "type": "comment"
+ },
+ "8430": {
+ "file_id": 623,
+ "content": " record_list.append((\"batch_time\", AverageMeter('batch_cost', '.5f')))\n record_list.append((\"reader_time\", AverageMeter('reader_cost', '.5f')))\n record_list = OrderedDict(record_list)\n return record_list\nclass AverageMeter(object):\n \"\"\"\n Computes and stores the average and current value\n \"\"\"\n def __init__(self, name='', fmt='f', need_avg=True):\n self.name = name\n self.fmt = fmt\n self.need_avg = need_avg\n self.reset()\n def reset(self):\n \"\"\" reset \"\"\"\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def update(self, val, n=1):\n \"\"\" update \"\"\"\n if isinstance(val, paddle.Tensor):\n val = float(val)\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n @property\n def total(self):\n return '{self.name}_sum: {self.sum:{self.fmt}}'.format(self=self)\n @property\n def total_minute(self):\n return '{self.name}_sum: {s:{self.fmt}} min'.format(s=self.sum / 60,",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:67-105"
+ },
+ "8431": {
+ "file_id": 623,
+ "content": "This function creates a record dictionary containing two AverageMeter objects, one for batch time and another for reader time. It then converts the list to an OrderedDict and returns it. The AverageMeter class calculates and stores the average and current values of a given metric, allowing easy tracking of performance metrics during program execution.",
+ "type": "comment"
+ },
+ "8432": {
+ "file_id": 623,
+ "content": " self=self)\n @property\n def mean(self):\n return '{self.name}_avg: {self.avg:{self.fmt}}'.format(\n self=self) if self.need_avg else ''\n @property\n def value(self):\n return '{self.name}: {self.val:{self.fmt}}'.format(self=self)\ndef log_batch(metric_list,\n batch_id,\n epoch_id,\n total_epoch,\n mode,\n ips,\n eta_sec: int = None):\n batch_cost = str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = str(metric_list['reader_time'].value) + ' sec,'\n metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].value)\n metric_str = ' '.join([str(v) for v in metric_values])\n epoch_str = \"epoch:[{:>3d}/{:<3d}]\".format(epoch_id, total_epoch)\n step_str = \"{:s} step:{:<4d}\".format(mode, batch_id)\n if eta_sec is not None:\n eta_str = \"eta: {:s}\".format(",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:106-136"
+ },
+ "8433": {
+ "file_id": 623,
+ "content": "This code defines a class and functions for recording metrics, calculating means, and logging batch information. The `log_batch` function records the time taken for each batch, adds other metric values, and logs the total epoch, current epoch, mode, and step. It also calculates the remaining time for the current operation if provided.",
+ "type": "comment"
+ },
+ "8434": {
+ "file_id": 623,
+ "content": " str(datetime.timedelta(seconds=int(eta_sec))))\n else:\n eta_str = ''\n max_mem_reserved_str = \"\"\n max_mem_allocated_str = \"\"\n if paddle.device.is_compiled_with_cuda():\n max_mem_reserved_str = f\"max_mem_reserved: {format(paddle.device.cuda.max_memory_reserved() / (1024 ** 2), '.2f')} MB\"\n max_mem_allocated_str = f\"max_mem_allocated: {format(paddle.device.cuda.max_memory_allocated() / (1024 ** 2), '.2f')} MB\"\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {} {:s}, {} {}\".format(\n coloring(epoch_str, \"HEADER\") if batch_id == 0 else epoch_str,\n coloring(step_str, \"PURPLE\"), coloring(metric_str, 'OKGREEN'),\n coloring(batch_cost, \"OKGREEN\"), coloring(reader_cost, 'OKGREEN'), ips,\n eta_str, max_mem_reserved_str, max_mem_allocated_str))\ndef log_epoch(metric_list, epoch, mode, ips):\n batch_cost = 'avg_' + str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = 'avg_' + str(metric_list['reader_time'].value) + ' sec,'\n batch_sum = str(metric_list['batch_time'].total) + ' sec,'",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:137-155"
+ },
+ "8435": {
+ "file_id": 623,
+ "content": "This code logs the progress of an epoch in a training process. It formats the logged information with colors for better visibility. The logger displays the current step, metrics, time taken, batch cost and reader cost, estimated time remaining (ETA), and maximum memory reserved and allocated on CUDA devices if available.",
+ "type": "comment"
+ },
+ "8436": {
+ "file_id": 623,
+ "content": " metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].mean)\n metric_str = ' '.join([str(v) for v in metric_values])\n end_epoch_str = \"END epoch:{:<3d}\".format(epoch)\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {:s} {}\".format(\n coloring(end_epoch_str, \"RED\"), coloring(mode, \"PURPLE\"),\n coloring(metric_str, \"OKGREEN\"), coloring(batch_cost, \"OKGREEN\"),\n coloring(reader_cost, \"OKGREEN\"), coloring(batch_sum, \"OKGREEN\"), ips))",
+ "type": "code",
+ "location": "/paddlevideo/utils/record.py:157-168"
+ },
+ "8437": {
+ "file_id": 623,
+ "content": "This code calculates the mean of metrics except 'batch_time' and 'reader_time', then joins them into a string. It formats an info message with RED for \"END epoch\", PURPLE for mode, GREEN for metric values, batch cost, reader cost, and batch sum, as well as ips (inferences per second). The logger outputs this formatted message.",
+ "type": "comment"
+ },
+ "8438": {
+ "file_id": 624,
+ "content": "/paddlevideo/utils/registry.py",
+ "type": "filepath"
+ },
+ "8439": {
+ "file_id": 624,
+ "content": "The code defines a Registry class for mapping names to objects and provides methods for registering, getting, and unregistering objects. It utilizes the @BACKBONES.register() decorator or BACKBONES.register(ResNet) function for registration, and also verifies if an object with a given name exists in the registry using the `get` method.",
+ "type": "summary"
+ },
+ "8440": {
+ "file_id": 624,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nclass Registry(object):\n \"\"\"\n The registry that provides name -> object mapping, to support third-party users' custom modules.\n To register an object:\n .. code-block:: python\n BACKBONES = Registry('backbone')\n @BACKBONES.register()\n class ResNet:\n pass\n Or:\n .. code-block:: python\n BACKBONES = Registry('backbone')\n class ResNet:\n pass\n BACKBONES.register(ResNet)",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py:1-34"
+ },
+ "8441": {
+ "file_id": 624,
+ "content": "This code defines a Registry class that provides name to object mapping, allowing third-party users to register their custom modules. Users can register their objects by using the @BACKBONES.register() decorator or by calling BACKBONES.register(ResNet).",
+ "type": "comment"
+ },
+ "8442": {
+ "file_id": 624,
+ "content": " Usage: To build a module.\n .. code-block:: python\n backbone_name = \"ResNet\"\n b = BACKBONES.get(backbone_name)()\n \"\"\"\n def __init__(self, name):\n \"\"\"\n Args:\n name (str): the name of this registry\n \"\"\"\n self._name = name\n self._obj_map = {}\n def __contains__(self, key):\n return self._obj_map.get(key) is not None\n def _do_register(self, name, obj):\n assert (\n name not in self._obj_map\n ), \"An object named '{}' was already registered in '{}' registry!\".format(\n name, self._name)\n self._obj_map[name] = obj\n def register(self, obj=None, name=None):\n \"\"\"\n Register the given object under the the name `obj.__name__`.\n Can be used as either a decorator or not. See docstring of this class for usage.\n \"\"\"\n if obj is None:\n # used as a decorator\n def deco(func_or_class, name=name):\n if name is None:\n name = func_or_class.__name__",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py:36-70"
+ },
+ "8443": {
+ "file_id": 624,
+ "content": "This code is a registry class for storing and managing objects. It allows registering objects under their names or using decorators, and provides functions to check if an object with a given name exists in the registry.",
+ "type": "comment"
+ },
+ "8444": {
+ "file_id": 624,
+ "content": " self._do_register(name, func_or_class)\n return func_or_class\n return deco\n # used as a function call\n if name is None:\n name = obj.__name__\n self._do_register(name, obj)\n def get(self, name):\n \"\"\"Get the registry record.\n Args:\n name (str): The class name.\n Returns:\n ret: The class.\n \"\"\"\n ret = self._obj_map.get(name)\n if ret is None:\n raise KeyError(\n \"No object named '{}' found in '{}' registry!\".format(\n name, self._name))\n return ret",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/utils/registry.py:71-96"
+ },
+ "8445": {
+ "file_id": 624,
+ "content": "The code defines a class with methods for registering, getting and unregistering objects in a registry. The `_do_register` method is used to store the object's name and function or class into a dictionary. If no name is provided when calling the function, it defaults to the object's name. The `get` method retrieves an object from the registry using its name. If the object is not found, it raises a KeyError with an error message.",
+ "type": "comment"
+ },
+ "8446": {
+ "file_id": 625,
+ "content": "/paddlevideo/utils/save_load.py",
+ "type": "filepath"
+ },
+ "8447": {
+ "file_id": 625,
+ "content": "The code imports modules, transfers model parameters, adjusts positional embeddings, and provides save/load functions for Resnet18, VisionTransformer (TimeSformer), SwinTransformer3D models using PaddlePaddle library.",
+ "type": "summary"
+ },
+ "8448": {
+ "file_id": 625,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport os.path as osp\nimport time\nimport paddle\nimport paddle.nn.functional as F\nfrom paddlevideo.utils import get_logger, main_only\nfrom tqdm import tqdm\nimport numpy as np\nfrom scipy import ndimage\ndef pretrain_swin_param_trans(model, state_dicts):\n # delete classifier's params\n if 'head.fc' + '.weight' in state_dicts:\n del state_dicts['head.fc' + '.weight']\n if 'head.fc' + '.bias' in state_dicts:",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:1-30"
+ },
+ "8449": {
+ "file_id": 625,
+ "content": "This code is from the PaddleVideo library and it imports necessary modules, defines a function for transferring pre-trained Swin model parameters, and deletes the classifier's weights from state_dicts.",
+ "type": "comment"
+ },
+ "8450": {
+ "file_id": 625,
+ "content": " del state_dicts['head.fc' + '.bias']\n state_dicts = {\n k.replace('backbone.', ''): v\n for k, v in state_dicts.items()\n }\n if len(state_dicts) == len(model.state_dict()):\n print(\"Load 3D weights\")\n return state_dicts\n print(\"Load 2D weights\")\n relative_position_index_keys = [\n k for k in state_dicts.keys() if \"relative_position_index\" in k\n ]\n for k in relative_position_index_keys:\n del state_dicts[k]\n # delete attn_mask since we always re-init it\n attn_mask_keys = [k for k in state_dicts.keys() if \"attn_mask\" in k]\n for k in attn_mask_keys:\n del state_dicts[k]\n state_dicts['patch_embed.proj.weight'] = state_dicts[\n 'patch_embed.proj.weight'].unsqueeze(2).tile(\n [1, 1, model.patch_size[0], 1, 1]) / model.patch_size[0]\n # bicubic interpolate relative_position_bias_table if not match\n relative_position_bias_table_keys = [\n k for k in state_dicts.keys() if \"relative_position_bias_table\" in k\n ]",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:31-61"
+ },
+ "8451": {
+ "file_id": 625,
+ "content": "This code checks if the loaded state dictionaries match the model's state dictionaries and handles any inconsistencies. It removes unnecessary keys, adjusts certain weights, and bicubically interpolates relative position bias tables if they don't match to ensure proper loading of 2D or 3D weights.",
+ "type": "comment"
+ },
+ "8452": {
+ "file_id": 625,
+ "content": " total_len = len(relative_position_bias_table_keys)\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n for key in tqdm(relative_position_bias_table_keys,\n total=total_len,\n position=0):\n relative_position_bias_table_pretrained = state_dicts[key]\n relative_position_bias_table_current = model.state_dict()[key]\n L1, nH1 = relative_position_bias_table_pretrained.shape\n L2, nH2 = relative_position_bias_table_current.shape\n L2 = (2 * model.window_size[1] - 1) * (2 * model.window_size[2] - 1)\n wd = model.window_size[0]\n if nH1 != nH2:\n desc.set_description(f\"Error in loading {key}, skip\")\n else:\n if L1 != L2:\n S1 = int(L1**0.5)\n relative_position_bias_table_pretrained_resized = paddle.nn.functional.interpolate(\n relative_position_bias_table_pretrained.transpose(",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:62-82"
+ },
+ "8453": {
+ "file_id": 625,
+ "content": "Loading weights for relative position bias tables from pretrained and current model state dictionaries.",
+ "type": "comment"
+ },
+ "8454": {
+ "file_id": 625,
+ "content": " [1, 0]).reshape([1, nH1, S1, S1]),\n size=(2 * model.window_size[1] - 1,\n 2 * model.window_size[2] - 1),\n mode='bicubic')\n relative_position_bias_table_pretrained = relative_position_bias_table_pretrained_resized.reshape(\n [nH2, L2]).transpose([1, 0])\n desc.set_description(f\"Loading {key}\")\n state_dicts[key] = relative_position_bias_table_pretrained.tile(\n [2 * wd - 1, 1])\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n return state_dicts\ndef pretrain_vit_param_trans(model, state_dicts, num_patches, num_seg,\n attention_type):\n \"\"\"\n Convert ViT's pre-trained model parameters to a parameter dictionary that matches the existing model\n \"\"\"\n if 'head' + '.weight' in state_dicts:\n del state_dicts['head' + '.weight']",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:83-105"
+ },
+ "8455": {
+ "file_id": 625,
+ "content": "Function is loading pre-trained model parameters, resizing a table, and setting the description.\nThe code is performing model parameter transformation for ViT models, deleting unnecessary weights.",
+ "type": "comment"
+ },
+ "8456": {
+ "file_id": 625,
+ "content": " if 'head' + '.bias' in state_dicts:\n del state_dicts['head' + '.bias']\n total_len = len(model.state_dict())\n if num_patches + 1 != state_dicts['pos_embed'].shape[1]: # when\n pos_embed = state_dicts['pos_embed']\n cls_pos_embed = paddle.to_tensor(\n pos_embed[0, 0, :]).unsqueeze(0).unsqueeze(1)\n other_pos_embed = paddle.to_tensor(pos_embed[0, 1:, :])\n gs_new = int(np.sqrt(num_patches))\n gs_old = int(np.sqrt(other_pos_embed.shape[0]))\n zoom = (gs_new / gs_old, gs_new / gs_old, 1)\n other_pos_embed = paddle.reshape(other_pos_embed, [gs_old, gs_old, -1])\n other_pos_embed = ndimage.zoom(other_pos_embed, zoom, order=1)\n other_pos_embed = paddle.to_tensor(other_pos_embed)\n new_pos_embed = paddle.reshape(other_pos_embed, [1, num_patches, -1])\n new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed), axis=1)\n state_dicts['pos_embed'] = new_pos_embed\n time.sleep(0.01)\n if 'time_embed' in state_dicts and num_seg != state_dicts[",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:106-126"
+ },
+ "8457": {
+ "file_id": 625,
+ "content": "This code block checks the shape of the 'pos_embed' tensor and adjusts it based on the number of patches provided. It resizes the tensor using Paddle's ndimage.zoom function and then reconstructs the updated positional embedding for the model. This is necessary when the number of patches changes, ensuring the positional embeddings are consistent with the new patch count.",
+ "type": "comment"
+ },
+ "8458": {
+ "file_id": 625,
+ "content": " 'time_embed'].shape[1]:\n time_embed = state_dicts['time_embed'].transpose((0, 2, 1)).unsqueeze(0)\n new_time_embed = F.interpolate(time_embed,\n size=(time_embed.shape[-2], num_seg),\n mode='nearest')\n state_dicts['time_embed'] = new_time_embed.squeeze(0).transpose(\n (0, 2, 1))\n time.sleep(0.01)\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n if attention_type == 'divided_space_time':\n new_state_dicts = state_dicts.copy()\n for key in tqdm(state_dicts):\n if 'blocks' in key and 'attn' in key:\n desc.set_description(\"Loading %s\" % key)\n new_key = key.replace('attn', 'temporal_attn')\n if not new_key in state_dicts:\n new_state_dicts[new_key] = state_dicts[key]\n else:",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:127-147"
+ },
+ "8459": {
+ "file_id": 625,
+ "content": "This code block is part of a larger program that loads pre-trained model weights. It first checks if the shape of 'time_embed' matches a specific condition, and if not, it performs some transformations on it. Afterwards, it starts a progress bar with the description \"Loading weights\" to show the progress of loading these weights. If the attention type is 'divided_space_time', it makes a copy of state_dicts and iterates over its keys, replacing 'attn' keys with 'temporal_attn' if not already present.",
+ "type": "comment"
+ },
+ "8460": {
+ "file_id": 625,
+ "content": " new_state_dicts[new_key] = state_dicts[new_key]\n if 'blocks' in key and 'norm1' in key:\n desc.set_description(\"Loading %s\" % key)\n new_key = key.replace('norm1', 'temporal_norm1')\n if not new_key in state_dicts:\n new_state_dicts[new_key] = state_dicts[key]\n else:\n new_state_dicts[new_key] = state_dicts[new_key]\n time.sleep(0.01)\n elif attention_type == 'space_only': # tokenshift raw vit\n new_state_dicts = state_dicts.copy()\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n return new_state_dicts\ndef pretrain_resnet18_param_trans(model, loaded_dict):\n encoder_dict = model.encoder.state_dict()\n pose_encoder_dict = model.pose_encoder.state_dict()\n names = ['encoder.', 'encoder_day.', 'encoder_night.']\n for name in names:\n total_len = len(loaded_dict.items())",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:148-172"
+ },
+ "8461": {
+ "file_id": 625,
+ "content": "This code appears to be related to model weight loading and adaptation for a pre-trained ResNet18 in a specific context. It modifies the state_dicts of certain keys, like replacing 'norm1' with 'temporal_norm1', possibly to adapt the weights to fit the new model structure. The code also checks if a certain key exists and copies it if not, ensuring the new model has all necessary parameters. Finally, it updates the description for the loading process.",
+ "type": "comment"
+ },
+ "8462": {
+ "file_id": 625,
+ "content": " with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n for key, value in tqdm(loaded_dict.items(),\n total=total_len,\n position=0):\n key = str(name + key)\n if key in encoder_dict:\n encoder_dict[key] = value\n desc.set_description('Loading %s' % key)\n time.sleep(0.01)\n num_input_images = 2\n loaded_dict['conv1.weight'] = paddle.concat(\n [loaded_dict['conv1.weight']] * num_input_images, 1) / num_input_images\n total_len = len(loaded_dict.items())\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n for name, value in tqdm(loaded_dict.items(),\n total=total_len,\n position=0):\n name = str('encoder.' + name)",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:173-197"
+ },
+ "8463": {
+ "file_id": 625,
+ "content": "This code is loading weights from a dictionary, updating the encoder_dict if the key already exists. It also updates loaded_dict for a specific convolution layer based on the number of input images and uses tqdm to provide progress updates.",
+ "type": "comment"
+ },
+ "8464": {
+ "file_id": 625,
+ "content": " if name in pose_encoder_dict:\n pose_encoder_dict[name] = value\n desc.set_description('Loading %s' % key)\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n return encoder_dict, pose_encoder_dict\n#XXX(shipping): maybe need load N times because of different cards have different params.\n@main_only\ndef load_ckpt(model, weight_path, **kargs):\n \"\"\"\n 1. Load pre-trained model parameters\n 2. Extract and convert from the pre-trained model to the parameters\n required by the existing model\n 3. Load the converted parameters of the existing model\n \"\"\"\n #model.set_state_dict(state_dict)\n if not osp.isfile(weight_path):\n raise IOError(f'{weight_path} is not a checkpoint file')\n #state_dicts = load(weight_path)\n logger = get_logger(\"paddlevideo\")\n state_dicts = paddle.load(weight_path)\n if 'ResnetEncoder' in str(model):\n encoder_dict, pose_encoder_dict = pretrain_resnet18_param_trans(",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:198-226"
+ },
+ "8465": {
+ "file_id": 625,
+ "content": "This code loads pre-trained model parameters from a specified file path and converts them for use in the existing model. If the weight_path is not a valid checkpoint file, it raises an IOError. The code also utilizes Paddle's `paddle.load()` function to load state_dicts from the specified file path. It handles loading of Resnet18 parameters specifically with the `pretrain_resnet18_param_trans()` function.",
+ "type": "comment"
+ },
+ "8466": {
+ "file_id": 625,
+ "content": " model, state_dicts)\n model.encoder.load_dict(encoder_dict)\n model.pose_encoder.load_dict(pose_encoder_dict)\n tmp = model.state_dict()\n elif \"VisionTransformer\" in str(model): # For TimeSformer case\n tmp = pretrain_vit_param_trans(model, state_dicts, kargs['num_patches'],\n kargs['num_seg'],\n kargs['attention_type'])\n elif 'SwinTransformer3D' in str(model):\n tmp = pretrain_swin_param_trans(model, state_dicts)\n else:\n tmp = {}\n total_len = len(model.state_dict())\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n for item in tqdm(model.state_dict(), total=total_len, position=0):\n name = item\n desc.set_description('Loading %s' % name)\n if name not in state_dicts: # Convert from non-parallel model\n if str('backbone.' + name) in state_dicts:",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:227-248"
+ },
+ "8467": {
+ "file_id": 625,
+ "content": "This code is loading the model's weights and dictionary entries. It checks the type of the model and then either loads or transposes the parameters accordingly, handling cases such as VisionTransformer (TimeSformer) and SwinTransformer3D. For other models, it simply initializes an empty dictionary and starts loading each item from the state_dict in a tqdm progress bar.",
+ "type": "comment"
+ },
+ "8468": {
+ "file_id": 625,
+ "content": " tmp[name] = state_dicts['backbone.' + name]\n else: # Convert from parallel model\n tmp[name] = state_dicts[name]\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n model.set_state_dict(tmp)\ndef mkdir(dir):\n if not os.path.exists(dir):\n # avoid error when train with multiple gpus\n try:\n os.makedirs(dir)\n except:\n pass\ndef _extract_student_weights(all_params, student_prefix=\"Student.\"):\n s_params = {\n key[len(student_prefix):]: all_params[key]\n for key in all_params if student_prefix in key\n }\n return s_params\n@main_only\ndef save(obj, path, save_student_model=False):\n if save_student_model:\n s_params = _extract_student_weights(obj)\n student_path = path.replace(\".pdparams\", \"_student.pdparams\")\n if len(s_params) > 0:\n paddle.save(s_params, student_path)",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:249-282"
+ },
+ "8469": {
+ "file_id": 625,
+ "content": "This code saves a PaddlePaddle model's state dictionary and optionally the student model's state dictionary to separate files. It also has functionality for handling parallel models, converting them into separate state dictionaries. The `mkdir` function is used to create directories if they don't exist already. If the `save_student_model` flag is set to True, it will save both the main and student model weights in separate files.",
+ "type": "comment"
+ },
+ "8470": {
+ "file_id": 625,
+ "content": " paddle.save(obj, path)\ndef load(file_name):\n if not osp.isfile(file_name):\n raise IOError(f'{file_name} not exist')\n return paddle.load(file_name)",
+ "type": "code",
+ "location": "/paddlevideo/utils/save_load.py:283-289"
+ },
+ "8471": {
+ "file_id": 625,
+ "content": "This code defines two functions: \"save\" and \"load\". The \"save\" function uses the Paddle library to save an object (obj) at a specified path. The \"load\" function checks if a file exists, raises an IOError if it does not, and then loads the object from the file using the Paddle library's load function.",
+ "type": "comment"
+ },
+ "8472": {
+ "file_id": 626,
+ "content": "/paddlevideo/version.py",
+ "type": "filepath"
+ },
+ "8473": {
+ "file_id": 626,
+ "content": "This code contains the version information for PaddleVideo, licensed under the Apache License 2.0, and defines the current version as \"0.0.1\".",
+ "type": "summary"
+ },
+ "8474": {
+ "file_id": 626,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n__all__ = [\"paddlevideo_version\"]\npaddlevideo_version = \"0.0.1\"",
+ "type": "code",
+ "location": "/applications/EIVideo/EIVideo/paddlevideo/version.py:1-16"
+ },
+ "8475": {
+ "file_id": 626,
+ "content": "This code contains the version information for PaddleVideo, licensed under the Apache License 2.0, and defines the current version as \"0.0.1\".",
+ "type": "comment"
+ },
+ "8476": {
+ "file_id": 627,
+ "content": "/run.sh",
+ "type": "filepath"
+ },
+ "8477": {
+ "file_id": 627,
+ "content": "This script trains multiple deep learning models for various computer vision tasks with PaddlePaddle framework and demonstrates running BMN test, exporting models, inference using PaddleVideo toolkit, and provides training time calculation.",
+ "type": "summary"
+ },
+ "8478": {
+ "file_id": 627,
+ "content": "export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\n#export FLAGS_conv_workspace_size_limit=800 #MB\n#export FLAGS_cudnn_exhaustive_search=1\n#export FLAGS_cudnn_batchnorm_spatial_persistent=1\nstart_time=$(date +%s)\n# run pp-tsm training\n#python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptsm main.py --validate -c configs/recognition/pptsm/pptsm_k400_frames_uniform.yaml\n# run pp-tsm_v2 distillation training\npython3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptsm_v2 main.py --validate -c configs/recognition/pptsm/v2/pptsm_lcnet_k400_16frames_uniform_dml_distillation.yaml\n# run ava training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3\" --log_dir=logdir.ava_part main.py --validate -w paddle.init_param.pdparams -c configs/detection/ava/ava_part.yaml\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=logdir.ava_all.1203 main.py --validate -w paddle.init_param.pdparams -c configs/detection/ava/ava_all.yaml",
+ "type": "code",
+ "location": "/run.sh:1-18"
+ },
+ "8479": {
+ "file_id": 627,
+ "content": "This script sets the CUDA visible devices, launches distributed training for multiple models (pp-tsm, pp-tsm_v2, ava), and specifies log directories and configurations. It runs all at once using 8 GPUs and Python3.7.",
+ "type": "comment"
+ },
+ "8480": {
+ "file_id": 627,
+ "content": "# run adds training\n# python3.7 main.py --validate -c configs/estimation/adds/adds.yaml --seed 20\n# run tsm training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_tsm main.py --validate -c configs/recognition/tsm/tsm_k400_frames.yaml\n# run tsm amp training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_tsm main.py --amp --validate -c configs/recognition/tsm/tsm_k400_frames.yaml\n# run tsm amp training, nhwc\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_tsm main.py --amp --validate -c configs/recognition/tsm/tsm_k400_frames_nhwc.yaml\n# run tsn training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_tsn main.py --validate -c configs/recognition/tsn/tsn_k400_frames.yaml\n# run video-swin-transformer training\n# python3.7 -u -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_videoswin main.py --amp --validate -c configs/recognition/videoswin/videoswin_k400_videos.yaml",
+ "type": "code",
+ "location": "/run.sh:20-36"
+ },
+ "8481": {
+ "file_id": 627,
+ "content": "This code contains various command lines to run different video recognition training processes using PaddlePaddle framework on multiple GPUs. Each line specifies the model architecture, the configuration file, and the options like validation, amp (automatic mixed precision), and GPU allocation for each specific task.",
+ "type": "comment"
+ },
+ "8482": {
+ "file_id": 627,
+ "content": "# run slowfast training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_slowfast main.py --validate -c configs/recognition/slowfast/slowfast.yaml\n# run slowfast multi-grid training\n# python3.7 -B -m paddle.distributed.launch --selected_gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log-slowfast main.py --validate --multigrid -c configs/recognition/slowfast/slowfast_multigrid.yaml\n# run bmn training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3\" --log_dir=log_bmn main.py --validate -c configs/localization/bmn.yaml\n# run attention_lstm training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_attetion_lstm main.py --validate -c configs/recognition/attention_lstm/attention_lstm_youtube-8m.yaml\n# run pp-tsn training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptsn main.py --validate -c configs/recognition/pptsn/pptsn_k400_frames.yaml\n# run timesformer training\n# python3.7 -B -m paddle.",
+ "type": "code",
+ "location": "/run.sh:38-54"
+ },
+ "8483": {
+ "file_id": 627,
+ "content": "This code executes multiple deep learning model training scripts for various computer vision tasks such as recognition, localization, and more using PaddlePaddle framework. The training runs in distributed mode with GPU utilization to speed up the process.",
+ "type": "comment"
+ },
+ "8484": {
+ "file_id": 627,
+ "content": "distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_timesformer main.py --validate -c configs/recognition/timesformer/timesformer_k400_videos.yaml\n# run pp-timesformer training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptimesformer main.py --validate -c configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml\n# run st-gcn training\n# python3.7 main.py -c configs/recognition/stgcn/stgcn_fsd.yaml\n# run agcn training\n# python3.7 main.py -c configs/recognition/agcn/agcn_fsd.yaml\n# run actbert training\n# python3.7 main.py --validate -c configs/multimodal/actbert/actbert.yaml\n# run tsn dali training\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3\" --log_dir=log_tsn main.py --train_dali -c configs/recognition/tsn/tsn_dali.yaml\n# test.sh\n# just use `example` as example, please replace to real name.\n# python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_test main.py --test -c configs/example.yaml -w \"output/example/example_best.pdparams\"",
+ "type": "code",
+ "location": "/run.sh:54-74"
+ },
+ "8485": {
+ "file_id": 627,
+ "content": "This code executes multiple deep learning model training and testing scripts for various tasks. It launches distributed training with specific GPU configurations, sets log directories, and uses different configuration files depending on the task. The tasks include pp-timesformer, st-gcn, agcn, actbert, tsn dali training, and example test.",
+ "type": "comment"
+ },
+ "8486": {
+ "file_id": 627,
+ "content": "# NOTE: run bmn test, only support single card, bs=1\n# python3.7 main.py --test -c configs/localization/bmn.yaml -w output/BMN/BMN_epoch_00010.pdparams -o DATASET.batch_size=1\n# export_models script\n# just use `example` as example, please replace to real name.\n# python3.7 tools/export_model.py -c configs/example.yaml -p output/example/example_best.pdparams -o ./inference\n# predict script\n# just use `example` as example, please replace to real name.\n# python3.7 tools/predict.py -v example.avi --model_file \"./inference/example.pdmodel\" --params_file \"./inference/example.pdiparams\" --enable_benchmark=False --model=\"example\" --num_seg=8\nend_time=$(date +%s)\ncost_time=$[ $end_time-$start_time ]\necho \"Time to train is $(($cost_time/60))min $(($cost_time%60))s\"",
+ "type": "code",
+ "location": "/run.sh:76-89"
+ },
+ "8487": {
+ "file_id": 627,
+ "content": "This script demonstrates the process of running BMN test, exporting models, and performing inference using the PaddleVideo toolkit. It highlights the commands required for each step, including the necessary configuration files and output directories. The script also calculates and outputs the training time in minutes and seconds.",
+ "type": "comment"
+ },
+ "8488": {
+ "file_id": 628,
+ "content": "/setup.py",
+ "type": "filepath"
+ },
+ "8489": {
+ "file_id": 628,
+ "content": "The setup file installs necessary packages for PaddleVideo, supports Python 3.2-3.6, and includes a console script \"ppvideo\" with keywords: \"A treasure chest for video understanding powered by PaddlePaddle.\" The code specifies Python version 3.7 and categorizes the project as a utility for metadata description in setup.py file.",
+ "type": "summary"
+ },
+ "8490": {
+ "file_id": 628,
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom setuptools import setup\nfrom io import open\nwith open('requirements.txt', encoding=\"utf-8-sig\") as f:\n requirements = f.readlines()\ndef readme():\n with open('docs/en/quick_start.md', encoding=\"utf-8-sig\") as f:\n README = f.read()\n return README\nsetup(\n name='ppvideo', #name of .whl file\n packages=['ppvideo'], #install package name\n package_dir={'ppvideo': ''},\n include_package_data=",
+ "type": "code",
+ "location": "/setup.py:1-31"
+ },
+ "8491": {
+ "file_id": 628,
+ "content": "This code is a setup file for the PaddleVideo package using setuptools. It specifies the package name, installs required packages from requirements.txt, reads README content, and sets up directory structure.",
+ "type": "comment"
+ },
+ "8492": {
+ "file_id": 628,
+ "content": " True, #Accept all data files and directories matched by MANIFEST.in\n install_requires=requirements,\n entry_points={\"console_scripts\": [\"ppvideo= ppvideo.tools.wheel:main\"]},\n version='2.3.0',\n license='Apache License 2.0',\n description='Awesome Video toolkits based on PaddlePaddle ',\n long_description=readme(),\n long_description_content_type='text/markdown',\n url='https://github.com/PaddlePaddle/PaddleVideo',\n download_url='https://github.com/PaddlePaddle/PaddleVideo.git',\n keywords=[\n 'A treasure chest for video understanding powered by PaddlePaddle.'\n ],\n classifiers=[\n 'Intended Audience :: Developers', 'Operating System :: OS Independent',\n 'Natural Language :: Chinese (Simplified)',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.2',\n 'Programming Language :: Python :: 3.3',\n 'Programming Language :: Python :: 3.4',\n 'Programming Language :: Python :: 3.5',\n 'Programming Language :: Python :: 3.6',",
+ "type": "code",
+ "location": "/setup.py:32-53"
+ },
+ "8493": {
+ "file_id": 628,
+ "content": "This code is for setting up a Python package named \"PaddleVideo\" using setup.py. It specifies package details such as its name, version, requirements, description, license, and URL. The package is built with console script \"ppvideo\", and it supports Python 3.2-3.6. Keywords: \"A treasure chest for video understanding powered by PaddlePaddle.\"",
+ "type": "comment"
+ },
+ "8494": {
+ "file_id": 628,
+ "content": " 'Programming Language :: Python :: 3.7', 'Topic :: Utilities'\n ],\n)",
+ "type": "code",
+ "location": "/setup.py:54-56"
+ },
+ "8495": {
+ "file_id": 628,
+ "content": "The code is specifying Python version 3.7 and categorizing the project as a utility for metadata description in setup.py file.",
+ "type": "comment"
+ },
+ "8496": {
+ "file_id": 629,
+ "content": "/test_tipc/README.md",
+ "type": "filepath"
+ },
+ "8497": {
+ "file_id": 629,
+ "content": "The code provides TIPC support for PaddleVideo, offering tutorials on acceleration features, defining naming conventions for testing, ONNX conversion, deployment with Paddle Serving, offline quantized training/inference, and multi-machine multi-GPU training/inference.",
+ "type": "summary"
+ },
+ "8498": {
+ "file_id": 629,
+ "content": "# 飞桨训推一体认证（TIPC）\n## 1. 简介\n飞桨除了基本的模型训练和预测，还提供了支持多端多平台的高性能推理部署工具。本文档提供了PaddleVideo中所有模型的飞桨训推一体认证 (Training and Inference Pipeline Certification(TIPC)) 信息和测试工具，方便用户查阅每种模型的训练推理部署打通情况，并可以进行一键测试。\n\n
![](\"docs/guide.png\")
\n
训练预测 | 更多
训练方式 | 模型压缩 | 其他预测部署 |\n| :--- | :--- | :----: | :--------: | :---- | :---- | :---- |\n| PP-TSM |pptsm_k400_frames_uniform | 动作识别 | 支持 | 混合精度 | 离线量化 | Paddle Inference: C++ |\n| PP-TSN |pptsn_k400_videos | 动作识别 | 支持 | 混合精度 | - | Paddle Inference: C++ |\n| AGCN |agcn_fsd\t | 动作识别 | 支持 | 混合精度 | - | - |\n| STGCN |stgcn_fsd | 动作识别 | 支持 | 混合精度 | - | - |\n| TimeSformer |timesformer_k400_videos | 动作识别 | 支持 | 混合精度 | - | - |",
+ "type": "code",
+ "location": "/test_tipc/README.md:2-30"
+ },
+ "8499": {
+ "file_id": 629,
+ "content": "This code provides an introduction to the PaddleVideo training and inference pipeline certification (TIPC), including a summary of support status for various models and deployment methods. It also mentions that more details on specific acceleration features can be found in tutorials associated with each test tool.",
+ "type": "comment"
+ }
+}
\ No newline at end of file
diff --git a/docs/data/85.json b/docs/data/85.json
new file mode 100644
index 000000000..86a437e4b
--- /dev/null
+++ b/docs/data/85.json
@@ -0,0 +1,547 @@
+{
+ "8500": {
+ "file_id": 629,
+ "content": "| SlowFast |slowfast | 动作识别 | 支持 | 混合精度 | - | - |\n| TSM |tsm_k400_frames | 动作识别 | 支持 | 混合精度 | - | - |\n| TSN |tsn_k400_frames | 动作识别 |支持|混合精度|-|-|\n| AttentionLSTM |attention_lstm_youtube8m | 动作识别 | 支持 | 混合精度 | - | - |\n| BMN |bmn | 动作时间定位 | 支持 | 混合精度 | - | - |\n## 3. 测试工具简介\n### 目录介绍\n```shell\ntest_tipc/\n├── configs/ # 配置文件目录\n│ ├── PP-TSM/\n│ │ ├── train_infer_python.txt # PP-TSM在Linux上进行python训练预测（基础训练预测）的配置文件\n│ │ ├── serving_infer_cpp.txt # PP-TSM在Linux上进行cpp serving测试的配置文件\n│ │ ├── train_amp_infer_python.txt # PP-TSM在Linux上进行python训练预测（混合精度训练预测）的配置文件\n│ │ ├── serving_infer_python.txt # PP-TSM在Linux上进行python serving预测的配置文件\n│ │ └── train_ptq_infer_python.txt # PP-TSM在Linux上进行离线量化推理测试的配置文件\n│ ├── PP-TSN/\n│ │ ├── train_infer_python.txt # PP-TSN在Linux上进行python训练预测（基础训练预测）的配置文件\n│ │ ├── paddle2onnx_infer_python.txt # PP-TSN在Linux上进行Paddle2ONNX预测（基础训练预测）的配置文件\n│ │ ├── serving_infer_cpp.txt # PP-TSN在Linux上进行cpp serving测试的配置文件\n│ │ └── train_amp_infer_python.txt # PP-TSN在Linux上进行python训练预测（混合精度训练预测）的配置文件",
+ "type": "code",
+ "location": "/test_tipc/README.md:31-55"
+ },
+ "8501": {
+ "file_id": 629,
+ "content": "This code snippet introduces the test tool for PaddleVideo, providing an overview of supported models and their respective configurations, as well as the directory structure containing these configuration files.",
+ "type": "comment"
+ },
+ "8502": {
+ "file_id": 629,
+ "content": "│ ├── ...\n│ └── ...\n├── results/ # 预先保存的预测结果，用于和实际预测结果进行精度比对\n│ ├── PP-TSM/\n│ │\t├── python_ppvideo_PP-TSM_results_fp16.txt # 预存的PP-TSM识别识别模型python预测fp16精度的结果\n│ │\t└── python_ppvideo_PP-TSM_results_fp32.txt # 预存的PP-TSM识别识别模型python预测fp32精度的结果\n│ ├── PP-TSN/\n│ │\t├── python_ppvideo_PP-TSN_results_fp32.txt # 预存的PP-TSN识别识别模型python预测fp16精度的结果\n│ │\t└── python_ppvideo_PP-TSN_results_fp32.txt # 预存的PP-TSN识别识别模型python预测fp32精度的结果\n│ ├── PP-TSN_CPP/\n│ │\t├── python_ppvideo_PP-TSN_results_fp32.txt # 预存的PP-TSN识别识别模型C++预测fp16精度的结果\n│ │\t└── python_ppvideo_PP-TSN_results_fp32.txt # 预存的PP-TSN识别识别模型C++预测fp32精度的结果\n│ ├── ...\n│ └── ...\n├── prepare.sh # 完成test_*.sh运行所需要的数据和模型下载\n├── docs/ # 详细的TIPC各种功能文档\n├── test_train_inference_python.sh # 测试python训练预测的主程序\n├── test_inference_cpp.sh # 测试C++预测的主程序\n├── test_paddle2onnx.sh # 测试paddle2onnx转换与推理的主程序\n├── compare_results.py # 用于对比log中的预测结果与results中的预存结果精度误差是否在限定范围内\n└── README.md # 介绍文档",
+ "type": "code",
+ "location": "/test_tipc/README.md:56-76"
+ },
+ "8503": {
+ "file_id": 629,
+ "content": "This code represents the directory structure of a PaddleVideo test_tipc project. It includes pre-stored prediction results for various models in the 'results' folder, which are used to compare and verify the precision of the actual predictions. The scripts 'prepare.sh', 'test_train_inference_python.sh', 'test_inference_cpp.sh', and 'compare_results.py' are provided for testing, training, inference using Python or C++, as well as comparing the results with pre-stored data to calculate precision errors. The 'docs' folder contains detailed documentation on TIPC features, while 'test_paddle2onnx.sh' is used to test Paddle to ONNX conversion and inference.",
+ "type": "comment"
+ },
+ "8504": {
+ "file_id": 629,
+ "content": "```\n### 测试流程概述\n使用本工具，可以测试不同功能的支持情况，以及预测结果是否对齐，测试流程概括如下：\n\n
![](\"docs/Video_TIPC.png\")
\n
\n
![\"图片\"](\"https://user-images.githubusercontent.com/46156734/148925774-a04b641c-6a71-43ed-a7c0-d4b66e8d6e8a.png\")
",
+ "type": "code",
+ "location": "/applications/EIVideo/README.md:1-15"
+ },
+ "929": {
+ "file_id": 86,
+ "content": "EIVideo is an interactive intelligent video annotation tool, available for Windows systems starting from Win7. It has downloadable packages for both EIVideo and QEIVideo users, with options to choose pre-releases or the latest stable version. The tool features a user-friendly interface and is actively maintained under the QPT-Family organization on GitHub, with an open license for use.",
+ "type": "comment"
+ },
+ "930": {
+ "file_id": 86,
+ "content": "
\n
![\"图片\"](\"https://ai-studio-static-online.cdn.bcebos.com/f792bac0dd3b4f44ade7d744b58e908e2a85ed8718b541cfb6b2ce9fc8ad4374\")
\n
![\"图片\"](\"https://user-images.githubusercontent.com/46156734/148927601-791362c0-0286-4fb9-b9d1-c193f7485de1.png\")
\n当您选择使用QEIVideo作为图形化界面时，即可视为同意使用“可能会存在大量体验不佳”的EAP产品。",
+ "type": "code",
+ "location": "/applications/EIVideo/README.md:16-49"
+ },
+ "931": {
+ "file_id": 86,
+ "content": "EIVideo: Interactive intelligent video annotation toolbox, based on the Baidu Paddle MA-Net interactive video segmentation model. Can complete full video annotation with simple frame tagging. Improves video segmentation quality through multiple interactions with the video.",
+ "type": "comment"
+ },
+ "932": {
+ "file_id": 86,
+ "content": "同样，您可选择借助基于[PaddleVideo](https://github.com/PaddlePaddle/PaddleVideo) 实现的\n交互式视频标注模型[EIVideo](https://github.com/QPT-Family/EIVideo/EIVideo) 进行二次开发，在此之上也可完成您需要的自定义图形化界面，后续也将提供二次开发指南。\n ![\"图片\"](\"https://user-images.githubusercontent.com/46156734/148928046-b1490080-52f0-4a15-b7ff-11d54b135039.png\")
\n> 如果您愿意参与到EIVideo或QEIVideo的建设中来，欢迎您与PMC取得联系 -> WX:GT_ZhangAcer \n## 使用方式\n### 安装&运行\n#### QPT包 - 适合无Python基础用户\n自动化配置相关Python环境，但仅支持Windows7/10/11操作系统，且不对盗版Windows7做任何适配。 \n下载地址：暂未上传\n> 自动化部署工具由[QPT - 自动封装工具](https://github.com/QPT-Family/QPT) 支持 \n#### 标准Python包 - 适合普通Python开发者\n* 国际方式：\n ```shell\n python -m pip install eivideo\n python qeivideo\n ```\n* 国内推荐：\n ```shell\n python -m pip install eivideo -i https://mirrors.bfsu.edu.cn/pypi/web/simple\n python qeivideo\n ```\n> 上述命令仅适用于常规情况，若您安装了多个Python或修改了相关开发工具与配置，请自行修改相关命令使其符合您的开发环境。\n#### 开发版本 - 适合高阶开发者进行开发/社区贡献\n* 国际方式：\n ```shell\n git clone https://github.com/QPT-Family/EIVideo.git\n python -m pip install -r requirements.txt\n ```",
+ "type": "code",
+ "location": "/applications/EIVideo/README.md:51-85"
+ },
+ "933": {
+ "file_id": 86,
+ "content": "Code is introducing the user to EIVideo, a customizable interactive video annotation model based on PaddleVideo, with instructions for installation and usage.",
+ "type": "comment"
+ },
+ "934": {
+ "file_id": 86,
+ "content": "* 国内推荐：\n ```shell\n # 请勿用于Push！！！\n git clone https://hub.fastgit.org/QPT-Family/EIVideo.git\n python -m pip install -r requirements.txt -i https://mirrors.bfsu.edu.cn/pypi/web/simple\n ```\n* 运行程序\n ```shell\n # 进入工作目录\n cd 此处填写EIVideo所在的目录的绝对路径，且该目录下拥有EIVideo与QEIVideo两文件夹。\n # 运行\n python QEIVideo/start.py\n # 如运行时无法找到对应包，可选择下述方式添加环境变量来调整索引次序后执行python\n # Windows\n set PYTHONPATH=$pwd:$PYTHONPATH\n # Linux\n export PYTHONPATH=$pwd:$PYTHONPATH\n ```\n> 上述命令仅适用于常规情况，若您安装了多个Python或修改了相关开发工具与配置，请自行修改相关命令使其符合您的开发环境。\n## (Q)EIVideo产品规划安排 \n> 由于QEIVideo由飞桨开源社区学生爱好者构成，所以在项目的产出过程中将会以学习为主进行开源贡献，如您原因与我们一同建设，我们也将非常欢迎~\n ![\"图片\"](\"https://user-images.githubusercontent.com/46156734/148928475-b5b340b7-241d-4ddc-8155-70d98c6384a9.png\")
\n- [x] EIVideo与Demo版QEIVideo发布0.1.0Alpha版本\n- [ ] 完善QEIVideo，丰富基础标注功能，于Q1升级至1.0Alpha版本\n- [ ] 回归QEIVideo稳定性，于Q2完成1.0正式版本发版\n- [ ] 增加视频目标检测、分类任务的交互式标注功能。\n### 开源协议\n本项目使用GNU LESSER GENERAL PUBLIC LICENSE(LGPL)开源协议。 \n> 因所使用的模型与数据集等原因，本项目中任一代码、参数均不可直接进行商用，如需商用请与我们取得联系。",
+ "type": "code",
+ "location": "/applications/EIVideo/README.md:86-119"
+ },
+ "935": {
+ "file_id": 86,
+ "content": "This code provides instructions for cloning the EIVideo repository, installing necessary dependencies, and running the QEIVideo application. It also mentions that these commands are suitable for regular cases, and users might need to modify them according to their specific development environment. The code discusses the product roadmap of (Q)EIVideo, including planned features and versions. It also specifies the open-source license used for this project and clarifies that the code and parameters cannot be directly used for commercial purposes without prior consent from the developers.",
+ "type": "comment"
+ },
+ "936": {
+ "file_id": 86,
+ "content": "### 引用来源\n1. EIVideo模型以及相关源码、论文与项目 - [PaddleVideo](https://github.com/PaddlePaddle/PaddleVideo)\n2. 部分表情包来源 - [甘城なつき](https://www.pixiv.net/users/3036679)",
+ "type": "code",
+ "location": "/applications/EIVideo/README.md:121-123"
+ },
+ "937": {
+ "file_id": 86,
+ "content": "This code block provides the reference sources for the EIVideo model and its related resources, as well as mentioning the origin of some emoji used in the project.",
+ "type": "comment"
+ },
+ "938": {
+ "file_id": 87,
+ "content": "/applications/EIVideo/resources/QT/demo.ui",
+ "type": "filepath"
+ },
+ "939": {
+ "file_id": 87,
+ "content": "The code creates a Qt application UI with video display, QGraphicsView and push button, along with complex design elements including buttons, sliders, panels, tabs, labels, and progress bars.",
+ "type": "summary"
+ },
+ "940": {
+ "file_id": 87,
+ "content": "\n\n MainWindow\n \n \n \n 0\n 0\n 800\n 486\n \n \n \n \n 800\n 486\n \n \n \n \n 800\n 486\n \n \n \n MainWindow\n \n \n \n \n \n 20\n 20\n 761\n 361\n \n \n \n QFrame::StyledPanel\n \n \n QFrame::Raised\n \n ",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/QT/demo.ui:1-44"
+ },
+ "941": {
+ "file_id": 87,
+ "content": "This code defines a user interface for a main window with a central widget, a frame for displaying video content, and a QGraphicsView to render the video. The window has a fixed size of 800x486 pixels.",
+ "type": "comment"
+ },
+ "942": {
+ "file_id": 87,
+ "content": " \n \n 0\n 0\n 761\n 321\n \n \n \n \n \n \n 0\n 320\n 761\n 41\n \n \n \n QFrame::StyledPanel\n \n \n QFrame::Raised\n \n \n \n \n -1\n -1\n 761\n 41\n \n \n \n - \n \n \n 打开视频\n \n \n
\n - ",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/QT/demo.ui:45-86"
+ },
+ "943": {
+ "file_id": 87,
+ "content": "The code represents a UI layout design for a user interface with a frame, a horizontal layout widget containing a push button labeled \"打开视频\", and possibly other UI elements. The frame is styled as raised panel, has a specified geometry, and the push button serves to open a video file.",
+ "type": "comment"
+ },
+ "944": {
+ "file_id": 87,
+ "content": " \n \n 保存标注\n \n \n
\n - \n \n \n Qt::Horizontal\n \n \n
\n - \n \n \n 选择目标\n \n \n
\n - \n \n \n 清空目标\n \n \n
\n - \n \n \n 开始推理\n \n \n
\n \n \n \n \n ",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/QT/demo.ui:87-125"
+ },
+ "945": {
+ "file_id": 87,
+ "content": "This code defines a user interface layout with various widgets, including QPushButtons and QSlider. The buttons have text labels in Chinese for \"保存标注\", \"选择目标\", \"清空目标\", and \"开始推理\". The layout is nested within other widgets to create a complex UI design.",
+ "type": "comment"
+ },
+ "946": {
+ "file_id": 87,
+ "content": " \n 0\n 10\n 751\n 301\n \n \n \n QFrame::StyledPanel\n \n \n QFrame::Raised\n \n \n \n \n \n \n 20\n 380\n 761\n 81\n \n \n \n 0\n \n \n \n 状态\n \n \n \n true\n \n \n \n 10\n 30\n 71\n 21\n \n \n ",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/QT/demo.ui:126-169"
+ },
+ "947": {
+ "file_id": 87,
+ "content": "The code defines a user interface layout with a panel, tab widget, and label. The panel has dimensions, frame shape, and shadow properties set. The tab widget contains a single enabled tab named \"状态\" (Chinese for \"Status\") and has a label inside it with specific geometry and text settings.",
+ "type": "comment"
+ },
+ "948": {
+ "file_id": 87,
+ "content": " 当前状态：\n \n \n \n \n true\n \n \n \n 80\n 30\n 81\n 21\n \n \n \n -------------\n \n \n \n \n \n 170\n 32\n 521\n 21\n \n \n \n 24\n \n \n \n \n true\n \n \n \n 680\n 30\n 60\n 21\n ",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/QT/demo.ui:170-212"
+ },
+ "949": {
+ "file_id": 87,
+ "content": "This code snippet represents the UI layout for a user interface, using QT framework. It includes labels, progress bars and their respective properties such as position, size, text and value. The labels display current status, act information, and potentially other relevant data. The progress bar shows progress with a specific value and is likely used to represent the completion of certain tasks or actions.",
+ "type": "comment"
+ },
+ "950": {
+ "file_id": 87,
+ "content": " \n \n Qt::LeftToRight\n \n \n 12%\n \n \n Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter\n \n \n \n \n \n 属性配置\n \n \n \n \n \n \n \n \n",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/QT/demo.ui:213-236"
+ },
+ "951": {
+ "file_id": 87,
+ "content": "This code represents the user interface layout for a Qt application. It includes various widgets such as labels, buttons and tabs arranged in a specific order with their respective properties and alignment set. The code also specifies the title of each tab.",
+ "type": "comment"
+ },
+ "952": {
+ "file_id": 88,
+ "content": "/applications/EIVideo/resources/cmd",
+ "type": "filepath"
+ },
+ "953": {
+ "file_id": 88,
+ "content": "Updating PaddleVideo's EIVideo on GitHub: pushing and pulling development branches, splitting and rejoining code.",
+ "type": "summary"
+ },
+ "954": {
+ "file_id": 88,
+ "content": "# 更新PaddleVideo上的EIVideo\ngit subtree push --prefix=applications/EIVideo/ https://github.com/QPT-Family/EIVideo 开发分支\ngit subtree pull --prefix=applications/EIVideo/ https://github.com/QPT-Family/EIVideo 开发分支 --squash\ngit subtree split --rejoin --prefix=applications/EIVideo/ --branch 开发分支",
+ "type": "code",
+ "location": "/applications/EIVideo/resources/cmd:1-4"
+ },
+ "955": {
+ "file_id": 88,
+ "content": "Updating PaddleVideo's EIVideo on GitHub: pushing and pulling development branches, splitting and rejoining code.",
+ "type": "comment"
+ },
+ "956": {
+ "file_id": 89,
+ "content": "/applications/FightRecognition/README.md",
+ "type": "filepath"
+ },
+ "957": {
+ "file_id": 89,
+ "content": "The README guides using PaddleVideo's Fight Recognition model for detecting fight and non-fight videos across four datasets. It includes data preparation, training, evaluation, exporting, quickstart guidance, and GPU usage control.",
+ "type": "summary"
+ },
+ "958": {
+ "file_id": 89,
+ "content": "# 打架识别模型\n## 内容\n- [1 快速开始](#快速开始)\n- [2 数据准备](#数据准备)\n - [2.1 数据集下载](#数据集下载)\n - [2.2 视频抽帧](#视频抽帧)\n - [2.3 训练集和验证集划分](#训练集和验证集划分)\n - [2.4 视频裁剪](#视频裁剪)\n- [3 模型训练](#模型训练)\n- [4 模型评估](#模型评估)\n- [5 模型导出](#模型导出)\n实时行人分析工具[PP-Human](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.4/deploy/pphuman)中集成了视频分类的打架识别模块。本文档介绍如何基于[PaddleVideo](https://github.com/PaddlePaddle/PaddleVideo/)，完成打架识别模型的训练流程。\n目前打架识别模型使用的是[PP-TSM](https://github.com/PaddlePaddle/PaddleVideo/blob/63c88a435e98c6fcaf353429d2df6cc24b8113ba/docs/zh-CN/model_zoo/recognition/pp-tsm.md)，并在PP-TSM视频分类模型训练流程的基础上修改适配，完成模型训练。\n请先参考[使用说明](https://github.com/XYZ-916/PaddleVideo/blob/develop/docs/zh-CN/usage.md)了解PaddleVideo模型库的使用。\n\n## 1 快速开始\n打架识别静态图模型获取[https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.zip](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.zip)。\n打架识别[demo](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/fight_demo.mp4)。\n首先需要将下载好的静态图模型解压并放到`inference`目录下，然后执行下面的命令即可直接判断一个给定的视频中是否存在打架行为：",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:1-29"
+ },
+ "959": {
+ "file_id": 89,
+ "content": "This README provides an overview of the Fight Recognition model using PaddleVideo, including sections on quick start, data preparation, model training, evaluation, and model export. The PP-TSM model is used for fight recognition and can be adapted from the existing PP-TSM video classification model training process. Quickstart instructions and download links are provided, along with information on where to find additional usage guidance.",
+ "type": "comment"
+ },
+ "960": {
+ "file_id": 89,
+ "content": "```\ncd ${PaddleVideo_root}\npython tools/predict.py --input_file fight.avi \\\n --config pptsm_fight_frames_dense.yaml \\\n --model_file inference/ppTSM/ppTSM.pdmodel \\\n --params_file inference/ppTSM/ppTSM.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\n\n## 2 数据准备\nPP-TSM是一个基于视频片段进行预测的模型。在PaddleVideo中，训练数据为`.mp4`、`.avi`等格式视频或者是抽帧后的视频帧序列，标签则可以是`.txt`格式存储的文件。\n\n### 2.1 数据集下载\n本项目基于6个公开的打架、暴力行为相关数据集合并后的数据进行模型训练。公开数据集具体信息如下：\n| 数据集 | 下载连接 | 简介 | 标注 | 数量 | 时长 |\n| ---- | ---- | ---------- | ---- | ---- | ---------- |\n| Surveillance Camera Fight Dataset| https://github.com/sayibet/fight-detection-surv-dataset | 裁剪视频，监控视角 | 视频级别 | 打架：150；非打架：150 | 2s |\n| A Dataset for Automatic Violence Detection in Videos | https://github.com/airtlab/A-Dataset-for-Automatic-Violence-Detection-in-Videos | 裁剪视频，室内自行录制 | 视频级别 | 暴力行为：115个场景，2个机位，共230 ；非暴力行为：60个场景，2个机位，共120 | 几秒钟 |",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:31-55"
+ },
+ "961": {
+ "file_id": 89,
+ "content": "This code is executing a Python script named \"predict.py\" in PaddleVideo's root directory, to predict fight events from a video file named 'fight.avi'. It uses the pre-trained pptsm_fight_frames_dense model and sets GPU usage and TensorRT as False.",
+ "type": "comment"
+ },
+ "962": {
+ "file_id": 89,
+ "content": "| Hockey Fight Detection Dataset | https://www.kaggle.com/datasets/yassershrief/hockey-fight-vidoes?resource=download | 裁剪视频，非真实场景 | 视频级别 | 打架：500；非打架：500 | 2s |\n| Video Fight Detection Dataset | https://www.kaggle.com/datasets/naveenk903/movies-fight-detection-dataset | 裁剪视频，非真实场景 | 视频级别 | 打架：100；非打架：101 | 2s |\n| Real Life Violence Situations Dataset | https://www.kaggle.com/datasets/mohamedmustafa/real-life-violence-situations-dataset | 裁剪视频，非真实场景 | 视频级别 | 暴力行为：1000；非暴力行为：1000 | 几秒钟 |\n| UBI Abnormal Event Detection Dataset| http://socia-lab.di.ubi.pt/EventDetection/ | 未裁剪视频，监控视角 | 帧级别 | 打架：216；非打架：784；裁剪后二次标注：打架1976，非打架1630 | 原视频几秒到几分钟不等，裁剪后2s |\n打架（暴力行为）视频3956个，非打架（非暴力行为）视频3501个，共7457个视频，每个视频几秒钟。\n\n### 2.2 视频抽帧\n为了加快训练速度，将视频进行抽帧。\n```bash\ncd ${PaddleVideo_root}\npython data/ucf101/extract_rawframes.py dataset/ rawframes/ --level 2 --ext mp4\n```\n其中，视频存放在`dataset`目录下，打架（暴力）视频存放在`dataset/fight`中；非打架（非暴力）视频存放在`dataset/nofight`中。`rawframes`目录存放抽取的视频帧。\n\n### 2.3 训练集和验证集划分",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:56-75"
+ },
+ "963": {
+ "file_id": 89,
+ "content": "Code comments:\n- Hockey Fight Detection Dataset: URL, clipped videos, non-realistic scenarios, video level, 500 fight and 500 non-fight videos, 2s duration.\n- Video Fight Detection Dataset: URL, clipped videos, non-realistic scenarios, video level, 100 fights and 101 non-fights, 2s duration.\n- Real Life Violence Situations Dataset: URL, clipped videos, non-realistic scenarios, video level, 1000 fights and 1000 non-fights, a few seconds duration.\n- UBI Abnormal Event Detection Dataset: URL, unclipped videos, surveillance angle, frame level, 216 fights, 784 non-fights, 7,840 frames total, original video durations varying from a few seconds to a few minutes.\n- Extracting rawframes for faster training by running a script in PaddleVideo_root.\n- Split dataset into fight and non-fight videos stored in fight and nofight directories respectively.",
+ "type": "comment"
+ },
+ "964": {
+ "file_id": 89,
+ "content": "本项目验证集1500条，来自Surveillance Camera Fight Dataset、A Dataset for Automatic Violence Detection in Videos、UBI Abnormal Event Detection Dataset三个数据集。\n也可根据下面的代码将数据按照0.8:0.2的比例划分成训练集和测试集：\n```python\nimport os\nimport glob\nimport random\nimport fnmatch\nimport re\nclass_id = {\n \"nofight\":0,\n \"fight\":1\n}\ndef get_list(path,key_func=lambda x: x[-11:], rgb_prefix='img_', level=1):\n if level == 1:\n frame_folders = glob.glob(os.path.join(path, '*'))\n elif level == 2:\n frame_folders = glob.glob(os.path.join(path, '*', '*'))\n else:\n raise ValueError('level can be only 1 or 2')\n def count_files(directory):\n lst = os.listdir(directory)\n cnt = len(fnmatch.filter(lst, rgb_prefix + '*'))\n return cnt\n # check RGB\n video_dict = {}\n for f in frame_folders:\n cnt = count_files(f)\n k = key_func(f)\n if level==2:\n k = k.split(\"/\")[0]\n video_dict[f]=str(cnt)+\" \"+str(class_id[k])\n return video_dict\ndef fight_splits(video_dict, train_percent=0.8):",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:77-118"
+ },
+ "965": {
+ "file_id": 89,
+ "content": "The code reads data from three datasets: Surveillance Camera Fight Dataset, A Dataset for Automatic Violence Detection in Videos, and UBI Abnormal Event Detection Dataset. It also allows for splitting the data into training and testing sets with an 80:20 ratio. The 'get_list' function retrieves the list of files and counts them, while the 'fight_splits' function takes the video dictionary and train percent as inputs to split the data into training and testing sets.",
+ "type": "comment"
+ },
+ "966": {
+ "file_id": 89,
+ "content": " videos = list(video_dict.keys())\n train_num = int(len(videos)*train_percent)\n train_list = []\n val_list = []\n random.shuffle(videos)\n for i in range(train_num):\n train_list.append(videos[i]+\" \"+str(video_dict[videos[i]]))\n for i in range(train_num,len(videos)):\n val_list.append(videos[i]+\" \"+str(video_dict[videos[i]]))\n print(\"train:\",len(train_list),\",val:\",len(val_list))\n with open(\"fight_train_list.txt\",\"w\") as f:\n for item in train_list:\n f.write(item+\"\\n\")\n with open(\"fight_val_list.txt\",\"w\") as f:\n for item in val_list:\n f.write(item+\"\\n\")\nframe_dir = \"rawframes\"\nlevel = 2\ntrain_percent = 0.8\nif level == 2:\n def key_func(x):\n return '/'.join(x.split('/')[-2:])\nelse:\n def key_func(x):\n return x.split('/')[-1]\nvideo_dict = get_list(frame_dir, key_func=key_func, level=level) \nprint(\"number:\",len(video_dict))\nfight_splits(video_dict, train_percent)\n```\n最终生成fight_train_list.txt和fight_val_list.txt两个文件。打架的标签为1，非打架的标签为0。",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:119-160"
+ },
+ "967": {
+ "file_id": 89,
+ "content": "This code generates two lists, one for training and one for validation, based on a provided video dictionary. It then shuffles the list of videos and splits them into train and val lists. The code also defines a key function depending on the level parameter. Finally, it prints the lengths of both lists, writes them to separate files \"fight_train_list.txt\" and \"fight_val_list.txt\", and calls the fight_splits() function with the video dictionary and train percentage as parameters. These two files will contain the labels for training and validation sets, where fight (label 1) and non-fight (label 0) videos are listed separately.",
+ "type": "comment"
+ },
+ "968": {
+ "file_id": 89,
+ "content": "\n### 2.4 视频裁剪\n对于未裁剪的视频，需要先进行裁剪才能用于模型训练，这个给出视频裁剪的函数`cut_video`，输入为视频路径，裁剪的起始帧和结束帧以及裁剪后的视频保存路径。\n```python\nimport cv2\ndef cut_video(video_path, frameToStart, frametoStop, saved_video_path):\n cap = cv2.VideoCapture(video_path)\n FPS = cap.get(cv2.CAP_PROP_FPS)\n #print(\"FPS:\",FPS)\n TOTAL_FRAME = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) # 获取视频总帧数\n #print(\"TOTAL_FRAME:\",TOTAL_FRAME)\n size = (cap.get(cv2.CAP_PROP_FRAME_WIDTH), cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n #print(\"size:\",size)\n videoWriter =cv2.VideoWriter(saved_video_path,apiPreference = 0,fourcc = cv2.VideoWriter_fourcc(*'mp4v'),fps=FPS,\n frameSize=(int(size[0]),int(size[1])))\n COUNT = 0\n while True:\n success, frame = cap.read()\n if success:\n COUNT += 1\n if COUNT <= frametoStop and COUNT > frameToStart: # 选取起始帧\n videoWriter.write(frame)\n else:\n print(\"cap.read failed!\")\n break\n if COUNT > frametoStop:",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:162-192"
+ },
+ "969": {
+ "file_id": 89,
+ "content": "The code defines a function `cut_video` which takes a video path, start and stop frame numbers, and a saved video path. It uses OpenCV to read the input video, determine its FPS, total frames, and size. The function then creates a new VideoWriter object with the specified output file name, fourcc codec, and same FPS as the input video. It writes only the frames between the start and stop frame numbers to the new video file.",
+ "type": "comment"
+ },
+ "970": {
+ "file_id": 89,
+ "content": " break\n cap.release()\n videoWriter.release()\n print(saved_video_path)\n```\n\n## 3 模型训练\n下载预训练模型：\n```bash\nwget https://videotag.bj.bcebos.com/PaddleVideo/PretrainModel/ResNet50_vd_ssld_v2_pretrained.pdparams\n```\n模型训练：\n```bash\n# 单卡训练\ncd ${PaddleVideo_root}\npython main.py --validate -c pptsm_fight_frames_dense.yaml\n```\n```bash\ncd ${PaddleVideo_root}\n# 多卡训练\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython -B -m paddle.distributed.launch --gpus=“0,1,2,3” \\\n --log_dir=log_pptsm_dense main.py --validate \\\n -c pptsm_fight_frames_dense.yaml\n```\n\n## 4 模型评估\n训练好的模型下载：[https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM_fight.pdparams)\n模型评估：\n```bash\ncd ${PaddleVideo_root}\npython main.py --test -c pptsm_fight_frames_dense.yaml \\\n -w ppTSM_fight_best.pdparams\n```\n其中`ppTSM_fight_best.pdparams`为训练好的模型。\n\n## 5 模型导出\n导出inference模型：\n```bash\ncd ${PaddleVideo_root}\npython tools/export_model.py -c pptsm_fight_frames_dense.yaml \\",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:193-245"
+ },
+ "971": {
+ "file_id": 89,
+ "content": "This code represents the final part of the model training process. The first line is a break statement which implies the end of a loop or condition block. Following that, it releases the `cap` and `videoWriter` objects, suggesting they were used for capturing and writing video data respectively. The last line prints out the saved video path.",
+ "type": "comment"
+ },
+ "972": {
+ "file_id": 89,
+ "content": " -p ppTSM_fight_best.pdparams \\\n -o inference/ppTSM\n```",
+ "type": "code",
+ "location": "/applications/FightRecognition/README.md:246-248"
+ },
+ "973": {
+ "file_id": 89,
+ "content": "This code is loading a pre-trained model, \"ppTSM_fight_best.pdparams\", and saving the inference output to the \"inference/ppTSM\" directory.",
+ "type": "comment"
+ },
+ "974": {
+ "file_id": 90,
+ "content": "/applications/FigureSkating/README.md",
+ "type": "filepath"
+ },
+ "975": {
+ "file_id": 90,
+ "content": "This code provides a guide for using OpenPose to process figure skating action data by converting videos into bone point data, suitable for model training and prediction in the PaddleVideo framework.",
+ "type": "summary"
+ },
+ "976": {
+ "file_id": 90,
+ "content": "# 花样滑冰动作识别\n---\n## 内容\n- [视频数据处理方法](#视频数据处理方法)\n- [模型训练预测方法](#模型训练预测方法)\n\n
![](\"Alex.gif\")
\n
![](\"../../../images/tsn_architecture.png\")
"
+ },
+ {
+ "comment": "This code provides instructions for training the Temporal Segment Networks model on the Kinetics-400 dataset. It explains how to download and add pre-trained ResNet50 weights as initialization parameters, and specifies where to find more information about data preparation and model configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/tsn.md\":20-47",
+ "content": "
\n
![](\"images/BasketballAction_demo.gif\")
\n
\n
![](\"./images/SlowFast_FasterRCNN.png\")
\n
\n![](\"../../../images/SlowFast.png\")
\nSlowFast Overview\n
\n## Data\nWe use Kinetics-400 to train this model\uff0cdata preparation please refer to [Kinetics-400 dataset](../../dataset/k400.md).\n## Train\nYou can start training by\uff1a\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\npython -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_slowfast main.py --validate -c configs/recognition/slowfast/slowfast.yaml"
+ },
+ {
+ "comment": "This code implements Multigrid training strategy to speed up SlowFast model training, which is time-consuming. The provided training script and performance evaluation show that using the multigrid method reduces the training time by 2.89x compared to normal training. For more details, refer to the accelerate documentation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/slowfast.md\":38-57",
+ "content": "```\n- Training would be efficent using our code. The training speed is 2x faster than the original implementation. Details can refer to [benchmark](https://github.com/PaddlePaddle/PaddleVideo/blob/main/docs/en/benchmark.md).\n### Speed up training\nIt's time consuming to train SlowFast model. So we implement [Multigrid training stragety](https://arxiv.org/abs/1912.00998) to speed up training. Training script:\n```bash\npython -B -m paddle.distributed.launch --selected_gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log-slowfast main.py --validate --multigrid -c configs/recognition/slowfast/slowfast_multigrid.yaml\n```\nPerformance evaluation:\n| training stragety | time cost of one epoch/min | total training time/min | speed-up |\n| :------ | :-----: | :------: |:------: |\n| Multigrid | 27.25 | 9758 (6.7 days) | 2.89x |\n| Normal | 78.76 | 15438 (10.7days) | base |\nFor more details, please refer to [accelerate doc](https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/tutorials/accelerate.md#%E8%AE%AD%E7%BB%83%E7%AD%96%E7%95%A5%E5%8A%A0%E9%80%9F)."
+ },
+ {
+ "comment": "This code provides instructions for testing the SlowFast model in PaddleVideo. It uses the distributed launch command to run on multiple GPUs, specifying the log directory and the model configuration file slowfast.yaml. The test accuracy for two configurations is also shown, with a note that Acc1 may be lower due to missing data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/slowfast.md\":60-78",
+ "content": "## Test\nYou can start testing by\uff1a\n```bash\npython -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_slowfast_test main.py --test -c configs/recognition/slowfast/slowfast.yaml -w output/SlowFast/SlowFast_epoch_000196.pdparams\n```\n- Args `-w` is used to specifiy the model path\uff0cyou can download our model in [SlowFast.pdparams](https://videotag.bj.bcebos.com/PaddleVideo/SlowFast/SlowFast.pdparams).\nTest accuracy in Kinetics-400:\n| Configs | Acc1 | Acc5 | Weights |\n| :---: | :---: | :---: | :---: |\n| [slowfast.yaml](../../../../configs/recognition/slowfast/slowfast.yaml) | 74.35 | 91.33 | [slowfast_4x16.pdparams](https://videotag.bj.bcebos.com/PaddleVideo/SlowFast/SlowFast.pdparams) |\n| [slowfast_multigrid.yaml](../../../../configs/recognition/slowfast/slowfast_multigrid.yaml) | 75.84 | 92.33 | [slowfast_8x8.pdparams](https://videotag.bj.bcebos.com/PaddleVideo/SlowFast/SlowFast_8*8.pdparams) |\n- Acc1 may be lower than that released in papaer, as ~5% data of kinetics-400 is missing. Experiments have verified that if training with the same data, we can get the same accuracy."
+ },
+ {
+ "comment": "This code provides instructions for exporting and using the SlowFast model in PaddleVideo. The first command generates the architecture file (SlowFast.pdmodel) and parameter file (SlowFast.pdiparams). The second command demonstrates how to run inference with these files on an input video, specifying the model configuration and enabling GPU usage if available. It outputs the top-1 class and score for the predicted results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/slowfast.md\":81-111",
+ "content": "## Inference\n### export inference model\n To get model architecture file `SlowFast.pdmodel` and parameters file `SlowFast.pdiparams`, use:\n```bash\npython3.7 tools/export_model.py -c configs/recognition/slowfast/slowfast.yaml \\\n -p data/SlowFast.pdparams \\\n -o inference/SlowFast\n```\n- Args usage please refer to [Model Inference](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0/docs/zh-CN/start.md#2-%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86).\n### infer\n```bash\npython3.7 tools/predict.py --input_file data/example.avi \\\n --config configs/recognition/slowfast/slowfast.yaml \\\n --model_file inference/SlowFast/SlowFast.pdmodel \\\n --params_file inference/SlowFast/SlowFast.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nexample of logs:\n```\nCurrent video file: data/example.avi\n top-1 class: 5\n top-1 score: 1.0"
+ },
+ {
+ "comment": "This code retrieves the class name from a given class ID using a map file and predicts the top1 result for a video named \"example.avi\". The reference provided is related to the SlowFast Networks for Video Recognition paper, which likely explains how this functionality works in detail.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/slowfast.md\":112-119",
+ "content": "```\nwe can get the class name using class id and map file `data/k400/Kinetics-400_label_list.txt`. The top1 prediction of `data/example.avi` is `archery`.\n## Reference\n- [SlowFast Networks for Video Recognition](https://arxiv.org/abs/1812.03982), Feichtenhofer C, Fan H, Malik J, et al."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1242c244-2702-4893-9d20-f2b36989dc42.json b/docs/doc/1242c244-2702-4893-9d20-f2b36989dc42.json
new file mode 100644
index 000000000..79a0805eb
--- /dev/null
+++ b/docs/doc/1242c244-2702-4893-9d20-f2b36989dc42.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The script prepares PaddleVideo's UR FALL dataset keypoints, normalizing them and handling inconsistencies for training. It also prepares a dataset for PPHuman, reading annotations, extracting data, and saving for training.",
+ "details": [
+ {
+ "comment": "This script converts keypoint results of UR FALL dataset into a format suitable for training by PaddleVideo. It normalizes keypoints using bounding boxes and adjusts the shape to be compatible with the PaddleVideo framework. The function also handles cases where the number of frames is more or less than 100.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/PPHuman/datasets/prepare_dataset.py\":0-33",
+ "content": "import os\nimport json\nimport numpy as np\nimport pickle\n\"\"\"\n This python script is used to convert keypoint results of UR FALL dataset\n for training by PaddleVideo\n\"\"\"\ndef self_norm(kpt, bbox):\n # kpt: (2, T, 17, 1), bbox: (T, 4)\n tl = bbox[:, 0:2]\n wh = bbox[:, 2:]\n tl = np.expand_dims(np.transpose(tl, (1, 0)), (2, 3))\n wh = np.expand_dims(np.transpose(wh, (1, 0)), (2, 3))\n res = (kpt - tl) / wh\n res *= np.expand_dims(np.array([[384.], [512.]]), (2, 3))\n return res\ndef convert_to_ppvideo(all_kpts, all_scores, all_bbox):\n # shape of all_kpts is (T, 17, 2)\n keypoint = np.expand_dims(np.transpose(all_kpts, [2, 0, 1]),\n -1) #(2, T, 17, 1)\n keypoint = self_norm(keypoint, all_bbox)\n scores = all_scores\n if keypoint.shape[1] > 100:\n frame_start = (keypoint.shape[1] - 100) // 2\n keypoint = keypoint[:, frame_start:frame_start + 100:2, :, :]\n scores = all_scores[frame_start:frame_start + 100:2, :, :]\n elif keypoint.shape[1] < 100:"
+ },
+ {
+ "comment": "The function `prepare_dataset` receives keypoint and scores as inputs. If the length of either is not divisible by 2, it pads them with zeros to maintain consistency. The else block simply takes every other value in both arrays. The `decode_json_path` function loads a JSON file, sorts its contents, extracts bounding boxes, keypoints, and scores from each entry, ignoring cases where there is more than one bounding box, and appends the processed data to separate lists for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/PPHuman/datasets/prepare_dataset.py\":34-68",
+ "content": " keypoint = np.concatenate([\n keypoint,\n np.zeros((2, 100 - keypoint.shape[1], 17, 1), dtype=keypoint.dtype)\n ], 1)[:, ::2, :, :]\n scores = np.concatenate([\n all_scores,\n np.zeros((100 - all_scores.shape[0], 17, 1), dtype=keypoint.dtype)\n ], 0)[::2, :, :]\n else:\n keypoint = keypoint[:, ::2, :, :]\n scores = scores[::2, :, :]\n return keypoint, scores\ndef decode_json_path(json_path):\n content = json.load(open(json_path))\n content = sorted(content, key=lambda x: x[0])\n all_kpts = []\n all_score = []\n all_bbox = []\n for annos in content:\n bboxes = annos[1]\n kpts = annos[2][0]\n frame_id = annos[0]\n if len(bboxes) != 1:\n continue\n kpt_res = []\n kpt_score = []\n for kpt in kpts[0]:\n x, y, score = kpt\n kpt_res.append([x, y])\n kpt_score.append([score])\n all_kpts.append(np.array(kpt_res))\n all_score.append(np.array(kpt_score))"
+ },
+ {
+ "comment": "This code prepares a dataset for PaddleVideo's PPHuman application. It reads annotations from \"annotations\" folder, extracts keypoints, labels, and scores, then saves them into numpy arrays and pickle file for training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/PPHuman/datasets/prepare_dataset.py\":69-97",
+ "content": " all_bbox.append([\n bboxes[0][0], bboxes[0][1], bboxes[0][2] - bboxes[0][0],\n bboxes[0][3] - bboxes[0][1]\n ])\n all_kpts_np = np.array(all_kpts)\n all_score_np = np.array(all_score)\n all_bbox_np = np.array(all_bbox)\n video_anno, scores = convert_to_ppvideo(all_kpts_np, all_score_np,\n all_bbox_np)\n return video_anno, scores\nif __name__ == '__main__':\n all_keypoints = []\n all_labels = [[], []]\n all_scores = []\n for i, path in enumerate(os.listdir(\"annotations\")):\n video_anno, score = decode_json_path(os.path.join(\"annotations\", path))\n all_keypoints.append(video_anno)\n all_labels[0].append(str(i))\n all_labels[1].append(0) #label 0 means falling\n all_scores.append(score)\n all_data = np.stack(all_keypoints, 0)\n all_score_data = np.stack(all_scores, 0)\n np.save(f\"train_data.npy\", all_data)\n pickle.dump(all_labels, open(f\"train_label.pkl\", \"wb\"))\n np.save(\"kptscore_data.npy\", all_score_data)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/124984cd-1046-4edd-b77a-75171501d444.json b/docs/doc/124984cd-1046-4edd-b77a-75171501d444.json
new file mode 100644
index 000000000..6315f3e67
--- /dev/null
+++ b/docs/doc/124984cd-1046-4edd-b77a-75171501d444.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code file is an __init__.py for EIVideo application, authored by Acer Zhang on Jan 6th. It sets root path and defines constants for temporary image and JSON file paths. The join_root_path function helps construct full paths from given partial paths.",
+ "details": [
+ {
+ "comment": "This code file is an __init__.py for EIVideo application, authored by Acer Zhang on Jan 6th. It sets root path and defines constants for temporary image and JSON file paths. The join_root_path function helps construct full paths from given partial paths.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/__init__.py\":0-15",
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport os\nfrom EIVideo.version import __version__\nEI_VIDEO_ROOT = os.path.abspath(os.path.dirname(__file__))\nTEMP_IMG_SAVE_PATH = \"./temp.png\"\nTEMP_JSON_SAVE_PATH = \"./save.json\"\nTEMP_JSON_FINAL_PATH = \"./final.json\"\ndef join_root_path(path: str):\n return os.path.join(EI_VIDEO_ROOT, path)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/127e4c33-f150-44d4-b70f-14f9e61c18cc.json b/docs/doc/127e4c33-f150-44d4-b70f-14f9e61c18cc.json
new file mode 100644
index 000000000..9b22fb7a2
--- /dev/null
+++ b/docs/doc/127e4c33-f150-44d4-b70f-14f9e61c18cc.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a Paddle's nn.Layer Decoder class with convolutional layers, BatchNorm, and ReLU activation functions for Manet architecture decoding. It imports the 'zero_' function to initialize all model biases to 0.",
+ "details": [
+ {
+ "comment": "This code defines a Decoder class using Paddle's nn.Layer, which takes in the number of classes and backbone type as parameters. It initializes the convolutional layers for feature extraction, BatchNorm layers for normalization, and ReLU activation functions. The last_conv sequence contains multiple Conv2D, BatchNorm, and ReLU layers for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/backbones/decoder_manet.py\":0-29",
+ "content": "import paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom EIVideo.paddlevideo.utils.manet_utils import kaiming_normal_\nclass Decoder(nn.Layer):\n def __init__(self, num_classes, backbone, BatchNorm):\n super(Decoder, self).__init__()\n if backbone == 'resnet' or backbone == 'drn' or backbone == 'resnet_edge':\n low_level_inplanes = 256\n elif backbone == 'xception':\n low_level_inplanes = 128\n elif backbone == 'mobilenet':\n low_level_inplanes = 24\n else:\n raise NotImplementedError\n self.conv1 = nn.Conv2D(low_level_inplanes, 48, 1, bias_attr=False)\n self.bn1 = BatchNorm(48)\n self.relu = nn.ReLU(True)\n self.last_conv = nn.Sequential(\n nn.Conv2D(304,\n 256,\n kernel_size=3,\n stride=1,\n padding=1,\n bias_attr=False), BatchNorm(256), nn.ReLU(True),\n nn.Sequential(),\n nn.Conv2D(256,"
+ },
+ {
+ "comment": "This code defines a decoder block for the Manet architecture. It includes a 2D convolution layer, batch normalization, and ReLU activation. The forward function performs interpolation on input feature maps and concatenates them with low-level features before passing through a final convolution. The _init_weight function initializes the weights of the block using Kaiming initialization for convolutions and fills batch norm with a constant value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/backbones/decoder_manet.py\":30-58",
+ "content": " 256,\n kernel_size=3,\n stride=1,\n padding=1,\n bias_attr=False), BatchNorm(256), nn.ReLU(True),\n nn.Sequential())\n self._init_weight()\n def forward(self, x, low_level_feat):\n low_level_feat = self.conv1(low_level_feat)\n low_level_feat = self.bn1(low_level_feat)\n low_level_feat = self.relu(low_level_feat)\n x = F.interpolate(x,\n size=low_level_feat.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat((x, low_level_feat), axis=1)\n x = self.last_conv(x)\n return x\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight)\n elif isinstance(m, nn.BatchNorm2D):\n from EIVideo.paddlevideo.utils.manet_utils import fill_\n fill_(m.weight, 1)"
+ },
+ {
+ "comment": "This code imports the function 'zero_' from EIVideo.paddlevideo.utils.manet_utils and then defines a build_decoder function that returns an instance of Decoder class with provided parameters (num_classes, backbone, BatchNorm). The zero_(m.bias) line initializes all the bias in the model (m) to 0 using the imported 'zero_' function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/backbones/decoder_manet.py\":59-64",
+ "content": " from EIVideo.paddlevideo.utils.manet_utils import zero_\n zero_(m.bias)\ndef build_decoder(num_classes, backbone, BatchNorm):\n return Decoder(num_classes, backbone, BatchNorm)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/12eb8ee3-6d8e-4bf8-aa1b-1a6465f46372.json b/docs/doc/12eb8ee3-6d8e-4bf8-aa1b-1a6465f46372.json
new file mode 100644
index 000000000..6787541df
--- /dev/null
+++ b/docs/doc/12eb8ee3-6d8e-4bf8-aa1b-1a6465f46372.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is from PaddleVideo's EIVideo module and includes util functions for distributed computing. It defines a function get_dist_info() to retrieve the current rank and world size, and main_only() is a decorator that only runs the wrapped function if the rank is 0 (used in distributed environments).",
+ "details": [
+ {
+ "comment": "This code is from PaddleVideo's EIVideo module and includes util functions for distributed computing. It defines a function get_dist_info() to retrieve the current rank and world size, and main_only() is a decorator that only runs the wrapped function if the rank is 0 (used in distributed environments).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/dist_utils.py\":0-29",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport functools\nimport paddle\nimport paddle.distributed as dist\ndef get_dist_info():\n world_size = dist.get_world_size()\n rank = dist.get_rank()\n return rank, world_size\ndef main_only(func):\n @functools.wraps(func)\n def wrapper(*args, **kwargs):\n rank, _ = get_dist_info()\n if rank == 0:\n return func(*args, **kwargs)\n return wrapper"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/133c60ab-9ffd-445e-9459-52ac1d4dd941.json b/docs/doc/133c60ab-9ffd-445e-9459-52ac1d4dd941.json
new file mode 100644
index 000000000..0aef3f9f0
--- /dev/null
+++ b/docs/doc/133c60ab-9ffd-445e-9459-52ac1d4dd941.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The given code contains a list of unique URLs for MP4 video files from the \"EuroCup2016\" dataset in the \"FootballAction\" application, which can be used for training or testing purposes.",
+ "details": [
+ {
+ "comment": "This code contains a list of URLs to mp4 video files from the \"EuroCup2016\" dataset in the \"FootballAction\" application. These videos may be used for training or testing purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/datasets/EuroCup2016/url.list\":0-25",
+ "content": "mp4/63e51df254d2402fac703b6c4fdb4ea9.mp4\nmp4/76b5f7ee28d942988c6b224bfac136bd.mp4\nmp4/250b88724acf40dbb6d7e8ccb400ef38.mp4\nmp4/c9516c903de3416c97dae91a59e968d7.mp4\nmp4/e1982c90cdd74abaacc4d0692070b400.mp4\nmp4/1be705a8f67648da8ec4b4296fa80895.mp4\nmp4/de23c0b2be3a4eb1990c5c657061fb29.mp4\nmp4/2754615de6e64c4fb95ce1a8095dc1c1.mp4\nmp4/299fe30d8f3b4a45b89313fe31f9f3c0.mp4\nmp4/6cc7db52c5ef4e70b401a5e00d8dd67a.mp4\nmp4/22e89747689e4f7e83e3620620c93269.mp4\nmp4/2ceb6c549fc64305a06a75acb355642b.mp4\nmp4/719b0a4bcb1f461eabb152298406b861.mp4\nmp4/259856b769044b4d8dc94076deb356bf.mp4\nmp4/d0bd3eab1e794f0f9501c353a6d37827.mp4\nmp4/19eb47cc736240d6b2dd930ab69da839.mp4\nmp4/4435b708af6d48519a6b726144147d51.mp4\nmp4/ea16ad2a020643529e257bd6cb11b3c3.mp4\nmp4/eeebffbd4ec74222a9c2d0775d79b689.mp4\nmp4/8cfb4e605af44055b1576c37eb0e3209.mp4\nmp4/6bca62b57cc449c6935f0b17f28d06be.mp4\nmp4/70cfc31e520840b2afca458f93a01ce4.mp4\nmp4/6496960935e845578e391a5916739752.mp4\nmp4/d6d25403a4bb4784aecff5f21fd00dc5.mp4\nmp4/3e23d452a082403391f8abfb87bf2fb4.mp4\nmp4/4c5d9d9af4f044c4a68d134061dc264f.mp4"
+ },
+ {
+ "comment": "This code contains a list of URLs pointing to MP4 video files from the \"EuroCup2016\" dataset in the \"FootballAction\" application of the PaddleVideo library. The URLs are unique identifiers for each video file, allowing for easy access and retrieval.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/datasets/EuroCup2016/url.list\":26-48",
+ "content": "mp4/6994844c64b44c26b935cee9604bef0a.mp4\nmp4/d6322cb95f6a4402ac80432b561abd5d.mp4\nmp4/2c8b5587083a4784a51622e4fec87ccd.mp4\nmp4/5faa60d70ed141de8560110e840f2048.mp4\nmp4/45d08bc5cb0f424f9ed9d7874eb561cd.mp4\nmp4/6630aaf0e32146088d0b624e9288f071.mp4\nmp4/f2edbee29c1b4966b3a410260f78fbe3.mp4\nmp4/f24116fdd6a54214991db32f7dddef67.mp4\nmp4/0265731a0c6f4a9398c88db8e3d4a3bc.mp4\nmp4/02d2de09997f4215b06e3b00ff0502a0.mp4\nmp4/9c231896c56a43f291a5e190949f4333.mp4\nmp4/4afbbf9afcd44dfea45b044117cccb48.mp4\nmp4/745db97a080d4f44b450dc17a2bcf069.mp4\nmp4/5933d0ce17854483b81a318d7d45a34e.mp4\nmp4/d2cfef2da9f84237a6950c7f6659655c.mp4\nmp4/5572686cb90f440988ded956a60e555d.mp4\nmp4/8962ac5a332346e180c79d701ae0a175.mp4\nmp4/f6e64ee9b13a4088b24c45c257894c1e.mp4\nmp4/f6ed2b612b3d43baa0726be8b14ebe7c.mp4\nmp4/8ab7b0cba5744eb3b6fb10003dfda383.mp4\nmp4/1f0a0698e38d493988fe42a50f7e8723.mp4\nmp4/737fdb054ca141f2a45013c1740dd0a0.mp4\nmp4/bab63a9bcf204e4b99c4a887a01bfd60.mp4"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1425e044-d978-4014-9936-3d6c863fb682.json b/docs/doc/1425e044-d978-4014-9936-3d6c863fb682.json
new file mode 100644
index 000000000..999a36270
--- /dev/null
+++ b/docs/doc/1425e044-d978-4014-9936-3d6c863fb682.json
@@ -0,0 +1,195 @@
+{
+ "summary": "The model employs PaddlePaddle for object matching, k-nearest neighbor search, SpatialCorrelationSampler for pairwise distances, and a neural network with separable convolutional layers for semantic segmentation. The Ma-Net's int_seghead updates global and local maps for sequence processing and performs tensor operations for video object segmentation.",
+ "details": [
+ {
+ "comment": "This code snippet is from a PaddlePaddle-based video object detection model. It defines functions for calculating pairwise distances between embeddings and initializes some global variables. The model is designed to take reference and query embeddings as input, compute pairwise squared L2 distances, and returns the flattened tensor of distances. This distance calculation is likely used in the matching process of objects in the video frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":0-41",
+ "content": "import os\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport sys\nsys.path.append(\"..\")\nfrom config import cfg\nimport time\nimport paddle.nn.functional as F\nfrom utils.api import int_, float_, long_\nfrom utils.api import kaiming_normal_\n#############################################################GLOBAL_DIST_MAP\nMODEL_UNFOLD = True\nWRONG_LABEL_PADDING_DISTANCE = 1e20\ndef _pairwise_distances(x, y, ys=None):\n \"\"\"Computes pairwise squared l2 distances between tensors x and y.\n Args:\n x: Tensor of shape [n, feature_dim].\n y: Tensor of shape [m, feature_dim].\n Returns:\n Float32 distances tensor of shape [n, m].\n \"\"\"\n xs = paddle.sum(x * x, 1)\n xs = xs.unsqueeze(1)\n if ys is None:\n ys = paddle.sum(y * y, 1)\n ys = ys.unsqueeze(0)\n else:\n ys = ys\n d = xs + ys - 2. * paddle.matmul(x, paddle.t(y))\n return d, ys\n##################\ndef _flattened_pairwise_distances(reference_embeddings, query_embeddings, ys):\n \"\"\"Calculates flattened tensor of pairwise distances between ref and query."
+ },
+ {
+ "comment": "The code calculates the distance between reference and query embeddings, performing pairwise distances calculations using the _pairwise_distances function. The result is a distance tensor with shape [reference_embeddings.size / embedding_dim, query_embeddings.size / embedding_dim]. This function also includes _nn_features_per_object_for_chunk which extracts features for each object using nearest neighbor attention.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":42-64",
+ "content": " Args:\n reference_embeddings: Tensor of shape [..., embedding_dim],\n the embedding vectors for the reference frame\n query_embeddings: Tensor of shape [n_query_images, height, width,\n embedding_dim], the embedding vectors for the query frames.\n Returns:\n A distance tensor of shape [reference_embeddings.size / embedding_dim,\n query_embeddings.size / embedding_dim]\n \"\"\"\n embedding_dim = query_embeddings.shape[-1]\n reference_embeddings = reference_embeddings.reshape([-1, embedding_dim])\n first_dim = -1\n query_embeddings = query_embeddings.reshape([first_dim, embedding_dim])\n dists, ys = _pairwise_distances(query_embeddings, reference_embeddings, ys)\n return dists, ys\ndef _nn_features_per_object_for_chunk(reference_embeddings, query_embeddings,\n wrong_label_mask, k_nearest_neighbors,\n ys):\n \"\"\"Extracts features for each object using nearest neighbor attention.\n Args:\n reference_embeddings: Tensor of shape [n_chunk, embedding_dim],"
+ },
+ {
+ "comment": "This code calculates pairwise distances between reference and query embedding vectors, selects the k-nearest neighbors, and returns the nearest neighbor features. It takes into account a wrong_label_mask and padding distance, which helps handle incorrect labels and avoid noisy data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":65-87",
+ "content": " the embedding vectors for the reference frame.\n query_embeddings: Tensor of shape [m_chunk, embedding_dim], the embedding\n vectors for the query frames.\n wrong_label_mask:\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m_chunk, n_objects, feature_dim].\n \"\"\"\n # reference_embeddings_key = reference_embeddings\n # query_embeddings_key = query_embeddings\n dists, ys = _flattened_pairwise_distances(reference_embeddings,\n query_embeddings, ys)\n dists = (paddle.unsqueeze(dists, 1) +\n paddle.unsqueeze(float_(wrong_label_mask), 0) *\n WRONG_LABEL_PADDING_DISTANCE)\n if k_nearest_neighbors == 1:\n features = paddle.min(dists, 2, keepdim=True)\n else:\n dists, _ = paddle.topk(-dists, k=k_nearest_neighbors, axis=2)\n dists = -dists\n valid_mask = (dists < WRONG_LABEL_PADDING_DISTANCE)"
+ },
+ {
+ "comment": "The code calculates the nearest neighbor features for each object in chunks to save memory. It starts by masking and averaging distances between reference and query embeddings, then selects relevant indices from flattened arrays to calculate nearest neighbors for objects. The function takes `reference_embeddings_flat`, `query_embeddings_flat`, `reference_labels_flat`, `ref_obj_ids`, `k_nearest_neighbors`, and `n_chunks` as input and returns the features and labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":88-117",
+ "content": " masked_dists = dists * valid_mask.float()\n pad_dist = paddle.max(masked_dists, axis=2, keepdim=True)[0].tile(\n (1, 1, masked_dists.shape[-1]))\n dists = paddle.where(valid_mask, dists, pad_dist)\n # take mean of distances\n features = paddle.mean(dists, axis=2, keepdim=True)\n return features, ys\n###\ndef _selected_pixel(ref_labels_flat, ref_emb_flat):\n index_list = paddle.arange(len(ref_labels_flat))\n index_list = index_list\n index_ = paddle.masked_select(index_list, ref_labels_flat != -1)\n index_ = long_(index_)\n ref_labels_flat = paddle.index_select(ref_labels_flat, index_, 0)\n ref_emb_flat = paddle.index_select(ref_emb_flat, index_, 0)\n return ref_labels_flat, ref_emb_flat\n###\ndef _nearest_neighbor_features_per_object_in_chunks(\n reference_embeddings_flat, query_embeddings_flat, reference_labels_flat,\n ref_obj_ids, k_nearest_neighbors, n_chunks):\n \"\"\"Calculates the nearest neighbor features per object in chunks to save mem."
+ },
+ {
+ "comment": "This code performs k-nearest neighbor search using chunking to save memory. It takes embedding vectors for reference and query frames, their class labels, object ids, the number of nearest neighbors, and the number of chunks as input. It calculates the chunk size based on the number of query frames and the specified number of chunks. If TEST_MODE is enabled, it selects some pixels from the input. Then, it checks if the reference labels are equal to unsqueezed object ids for each query frame and creates a mask for wrong labels. It returns nearest neighbor features of shape [m, n_objects, feature_dim].",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":118-140",
+ "content": " Uses chunking to bound the memory use.\n Args:\n reference_embeddings_flat: Tensor of shape [n, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings_flat: Tensor of shape [m, embedding_dim], the embedding\n vectors for the query frames.\n reference_labels_flat: Tensor of shape [n], the class labels of the\n reference frame.\n ref_obj_ids: int tensor of unique object ids in the reference labels.\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m, n_objects, feature_dim].\n \"\"\"\n chunk_size = int_(\n np.ceil((float_(query_embeddings_flat.shape[0]) / n_chunks).numpy()))\n if cfg.TEST_MODE:\n reference_labels_flat, reference_embeddings_flat = _selected_pixel(\n reference_labels_flat, reference_embeddings_flat)\n wrong_label_mask = (reference_labels_flat != paddle.unsqueeze("
+ },
+ {
+ "comment": "This code calculates nearest neighbor features for each object across multiple chunks. It splits the query embeddings into different chunks, then computes the features for each chunk individually. If there is only one chunk, it returns the features directly. Otherwise, it concatenates all the computed features along axis 0 and returns them as nearest neighbor features.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":141-168",
+ "content": " ref_obj_ids, 1))\n all_features = []\n for n in range(n_chunks):\n if n == 0:\n ys = None\n if n_chunks == 1:\n query_embeddings_flat_chunk = query_embeddings_flat\n else:\n chunk_start = n * chunk_size\n chunk_end = (n + 1) * chunk_size\n query_embeddings_flat_chunk = query_embeddings_flat[\n chunk_start:chunk_end]\n features, ys = _nn_features_per_object_for_chunk(\n reference_embeddings_flat, query_embeddings_flat_chunk,\n wrong_label_mask, k_nearest_neighbors, ys)\n all_features.append(features)\n if n_chunks == 1:\n nn_features = all_features[0]\n else:\n nn_features = paddle.concat(all_features, axis=0)\n return nn_features\ndef nearest_neighbor_features_per_object(reference_embeddings,\n query_embeddings,\n reference_labels,\n k_nearest_neighbors,\n gt_ids=None,"
+ },
+ {
+ "comment": "This function calculates the distance between nearest neighbors in reference_embeddings and query_embeddings for each object. It uses the provided reference_labels to determine objects, subsamples if max_neighbors_per_object is specified, and considers k_nearest_neighbors. The gt_ids are used for determining unique ground truth ids in the first frame.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":169-185",
+ "content": " n_chunks=100):\n \"\"\"Calculates the distance to the nearest neighbor per object.\n For every pixel of query_embeddings calculate the distance to the\n nearest neighbor in the (possibly subsampled) reference_embeddings per object.\n Args:\n reference_embeddings: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings: Tensor of shape [n_query_images, height, width,\n embedding_dim], the embedding vectors for the query frames.\n reference_labels: Tensor of shape [height, width, 1], the class labels of\n the reference frame.\n max_neighbors_per_object: Integer, the maximum number of candidates\n for the nearest neighbor query per object after subsampling,\n or 0 for no subsampling.\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n gt_ids: Int tensor of shape [n_objs] of the sorted unique ground truth\n ids in the first frame. If None, it will be derived from"
+ },
+ {
+ "comment": "This function calculates the nearest neighbor features for query images using reference embeddings and labels. It first asserts that the shape of reference embeddings matches the shape of reference labels. Then, it flattens the reference labels and checks if gt_ids (ground truth ids) are provided. If not, it finds unique object ids in the reference labels, creates a tensor with those ids, and converts them to integer type. Else, it converts the given gt_ids to integers. The function reshapes the query and reference embeddings, calculates embedding dimensions, and returns the nearest neighbor features and gt_ids.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":186-210",
+ "content": " reference_labels.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [n_query_images, height, width, n_objects, feature_dim].\n gt_ids: An int32 tensor of the unique sorted object ids present\n in the reference labels.\n \"\"\"\n assert (reference_embeddings.shape[:2] == reference_labels.shape[:2])\n h, w, _ = query_embeddings.shape\n reference_labels_flat = reference_labels.reshape([-1])\n if gt_ids is None:\n ref_obj_ids = paddle.unique(reference_labels_flat)[-1]\n ref_obj_ids = np.arange(0, ref_obj_ids + 1)\n gt_ids = paddle.to_tensor(ref_obj_ids)\n gt_ids = int_(gt_ids)\n else:\n gt_ids = int_(paddle.arange(0, gt_ids + 1))\n embedding_dim = query_embeddings.shape[-1]\n query_embeddings_flat = query_embeddings.reshape([-1, embedding_dim])\n reference_embeddings_flat = reference_embeddings.reshape(\n [-1, embedding_dim])"
+ },
+ {
+ "comment": "This code chunk performs nearest neighbor feature extraction for each object in the image, reshapes it, and then returns it along with gt_ids. The local_pairwise_distances function computes pairwise squared l2 distances using a local search window. It is used to compare features between different points in an optimized manner, considering a maximum distance per dimension.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":211-234",
+ "content": " nn_features = _nearest_neighbor_features_per_object_in_chunks(\n reference_embeddings_flat, query_embeddings_flat, reference_labels_flat,\n gt_ids, k_nearest_neighbors, n_chunks)\n nn_features_dim = nn_features.shape[-1]\n nn_features = nn_features.reshape(\n [1, h, w, gt_ids.shape[0], nn_features_dim])\n return nn_features.cuda(), gt_ids\n########################################################################LOCAL_DIST_MAP\ndef local_pairwise_distances(x, y, max_distance=9):\n \"\"\"Computes pairwise squared l2 distances using a local search window.\n Optimized implementation using correlation_cost.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].\n y: Float32 tensor of shape [height, width, feature_dim].\n max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n Returns:\n Float32 distances tensor of shape\n [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n if cfg.MODEL_LOCAL_DOWNSAMPLE:"
+ },
+ {
+ "comment": "This code is performing cross-correlation between two input tensors and applying a boundary condition to the resulting tensor. It then applies sigmoid activation, resizes the tensor to original dimensions, and transposes it back to the original shape before unsqueezing the last dimension.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":235-260",
+ "content": " #####\n ori_h, ori_w, _ = x.shape\n x = x.transpose([2, 0, 1]).unsqueeze(0)\n x = F.avg_pool2d(x, (2, 2), (2, 2))\n y = y.transpose([2, 0, 1]).unsqueeze(0)\n y = F.avg_pool2d(y, (2, 2), (2, 2))\n x = x.squeeze(0).transpose([1, 2, 0])\n y = y.squeeze(0).transpose([1, 2, 0])\n corr = cross_correlate(x, y, max_distance=max_distance)\n xs = paddle.sum(x * x, 2, keepdim=True)\n ys = paddle.sum(y * y, 2, keepdim=True)\n ones_ys = paddle.ones_like(ys)\n ys = cross_correlate(ones_ys, ys, max_distance=max_distance)\n d = xs + ys - 2 * corr\n # Boundary should be set to Inf.\n tmp = paddle.zeros_like(d)\n boundary = paddle.equal(\n cross_correlate(ones_ys, ones_ys, max_distance=max_distance), 0)\n d = paddle.where(boundary, tmp.fill_(float_('inf')), d)\n d = (paddle.nn.functional.sigmoid(d) - 0.5) * 2\n d = d.transpose([2, 0, 1]).unsqueeze(0)\n d = F.interpolate(d,\n size=(ori_h, ori_w),"
+ },
+ {
+ "comment": "This code calculates the pairwise squared l2 distances between two tensors using either correlation or cross-correlation method, depending on the mode. In correlation mode, it uses bilinear interpolation and aligns corners. Otherwise, it uses cross-correlate function with a max_distance parameter. It also handles boundary cases by setting values to infinity where necessary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":261-286",
+ "content": " mode='bilinear',\n align_corners=True)\n d = d.squeeze(0).transpose([1, 2, 0])\n else:\n corr = cross_correlate(x, y, max_distance=max_distance)\n xs = paddle.sum(x * x, 2, keepdim=True)\n ys = paddle.sum(y * y, 2, keepdim=True)\n ones_ys = paddle.ones_like(ys)\n ys = cross_correlate(ones_ys, ys, max_distance=max_distance)\n d = xs + ys - 2 * corr\n # Boundary should be set to Inf.\n tmp = paddle.zeros_like(d)\n boundary = paddle.equal(\n cross_correlate(ones_ys, ones_ys, max_distance=max_distance), 0)\n d = paddle.where(boundary, tmp.fill_(float_('inf')), d)\n return d\ndef local_pairwise_distances2(x, y, max_distance=9):\n \"\"\"Computes pairwise squared l2 distances using a local search window.\n Naive implementation using map_fn.\n Used as a slow fallback for when correlation_cost is not available.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].\n y: Float32 tensor of shape [height, width, feature_dim]."
+ },
+ {
+ "comment": "This code section performs local downsampling on the input tensors x and y. It first transposes the tensors and applies average pooling with a 2x2 kernel to reduce their size. Then, it pads the result of y with a large value, calculates offsets using unfolding, subtracts them from x, and sums squared differences across channels. The result is a distances tensor of shape [height, width, (2 * max_distance + 1) ** 2].",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":287-311",
+ "content": " max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n Returns:\n Float32 distances tensor of shape\n [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n if cfg.MODEL_LOCAL_DOWNSAMPLE:\n ori_h, ori_w, _ = x.shape\n x = paddle.transpose(x, [2, 0, 1]).unsqueeze(0)\n x = F.avg_pool2d(x, (2, 2), (2, 2))\n y = paddle.transpose(y, [2, 0, 1]).unsqueeze(0)\n y = F.avg_pool2d(y, (2, 2), (2, 2))\n _, channels, height, width = x.shape\n padding_val = 1e20\n padded_y = F.pad(\n y, (max_distance, max_distance, max_distance, max_distance),\n mode='constant',\n value=padding_val)\n offset_y = F.unfold(padded_y, kernel_sizes=[height, width]).reshape(\n [1, channels, height, width, -1])\n x = x.reshape([1, channels, height, width, 1])\n minus = x - offset_y\n dists = paddle.sum(paddle.multiply(minus, minus),\n axis=1).reshape([1, height, width,"
+ },
+ {
+ "comment": "This code calculates the distance between a set of 2D points and another point, in a sliding window manner. It handles two cases: when the first point set has been divided into smaller blocks for faster computation, and when it hasn't. The result is stored in dists as a list of distance matrices.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":312-335",
+ "content": " -1]).transpose([0, 3, 1, 2])\n dists = (paddle.nn.functional.sigmoid(dists) - 0.5) * 2\n dists = F.interpolate(dists,\n size=[ori_h, ori_w],\n mode='bilinear',\n align_corners=True)\n dists = dists.squeeze(0).transpose([1, 2, 0])\n else:\n padding_val = 1e20\n padded_y = nn.functional.pad(\n y, (0, 0, max_distance, max_distance, max_distance, max_distance),\n mode='constant',\n value=padding_val)\n height, width, _ = x.shape\n dists = []\n for y_start in range(2 * max_distance + 1):\n y_end = y_start + height\n y_slice = padded_y[y_start:y_end]\n for x_start in range(2 * max_distance + 1):\n x_end = x_start + width\n offset_y = y_slice[:, x_start:x_end]\n dist = paddle.sum(paddle.pow((x - offset_y), 2), dim=2)\n dists.append(dist)"
+ },
+ {
+ "comment": "This code defines the SpatialCorrelationSampler class and a function called cross_correlate. The cross_correlate function takes two tensors, x and y, of shape [height, width, feature_dim] as inputs. It computes the cross correlation of these tensors using an optimized implementation from the SpatialCorrelationSampler class. The output tensor has a shape of [height, width, (2 * max_distance + 1) ** 2].",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":336-364",
+ "content": " dists = paddle.stack(dists, dim=2)\n return dists\nclass SpatialCorrelationSampler:\n pass\ndef cross_correlate(x, y, max_distance=9):\n \"\"\"Efficiently computes the cross correlation of x and y.\n Optimized implementation using correlation_cost.\n Note that we do not normalize by the feature dimension.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].\n y: Float32 tensor of shape [height, width, feature_dim].\n max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n Returns:\n Float32 tensor of shape [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n corr_op = SpatialCorrelationSampler(kernel_size=1,\n patch_size=2 * max_distance + 1,\n stride=1,\n dilation_patch=1,\n padding=0)\n xs = x.transpose(2, 0, 1)\n xs = paddle.unsqueeze(xs, 0)"
+ },
+ {
+ "comment": "This code is part of the IntVOS model and defines a function that computes nearest neighbor features, allowing only local matches. It takes previous frame embedding, query embedding, previous frame labels, ground truth IDs and maximum distance as input. It transposes and reshapes the tensors before returning the computed correlations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":365-391",
+ "content": " ys = y.transpose(2, 0, 1)\n ys = paddle.unsqueeze(ys, 0)\n corr = corr_op(xs, ys)\n bs, _, _, hh, ww = corr.shape\n corr = corr.reshape([bs, -1, hh, ww])\n corr = paddle.squeeze(corr, 0)\n corr = corr.transpose(1, 2, 0)\n return corr\ndef local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding,\n query_embedding,\n prev_frame_labels,\n gt_ids,\n max_distance=12):\n \"\"\"Computes nearest neighbor features while only allowing local matches.\n Args:\n prev_frame_embedding: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the last frame.\n query_embedding: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the query frames.\n prev_frame_labels: Tensor of shape [height, width, 1], the class labels of\n the previous frame.\n gt_ids: Int Tensor of shape [n_objs] of the sorted unique ground truth\n ids in the first frame.\n max_distance: Integer, the maximum distance allowed for local matching."
+ },
+ {
+ "comment": "This function calculates the nearest neighbor features using local pairwise distances. If MODEL_UNFOLD is set, it pads and unfolds the labels for offset masks generation. Else, it directly creates masks by comparing prev_frame_labels to gt_ids.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":392-420",
+ "content": " Returns:\n nn_features: A float32 np.array of nearest neighbor features of shape\n [1, height, width, n_objects, 1].\n \"\"\"\n d = local_pairwise_distances2(query_embedding,\n prev_frame_embedding,\n max_distance=max_distance)\n height, width = prev_frame_embedding.shape[:2]\n if MODEL_UNFOLD:\n labels = float_(prev_frame_labels).transpose([2, 0, 1]).unsqueeze(0)\n padded_labels = F.pad(labels, (\n 2 * max_distance,\n 2 * max_distance,\n 2 * max_distance,\n 2 * max_distance,\n ))\n offset_labels = F.unfold(padded_labels,\n kernel_sizes=[height, width],\n strides=[2, 2]).reshape([height, width, -1, 1])\n offset_masks = paddle.equal(\n offset_labels,\n float_(gt_ids).unsqueeze(0).unsqueeze(0).unsqueeze(0))\n else:\n masks = paddle.equal(prev_frame_labels,\n gt_ids.unsqueeze(0).unsqueeze(0))"
+ },
+ {
+ "comment": "This code applies padding to masks and creates offset masks by slicing the padded masks. It then constructs a 3D tensor of offset masks using Paddle's stack function. It also tiles the input data 'd' along the gt_ids dimension and creates a padding tensor. The code then computes the minimum distance between the tiled input and the masked data, resulting in distances tensor. Finally, it reshapes the distances tensor to have a specific shape and returns it.\nThe _res_block class is a layer that takes an input dimension (in_dim) and an output dimension (out_dim).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":421-453",
+ "content": " padded_masks = nn.functional.pad(masks, (\n 0,\n 0,\n max_distance,\n max_distance,\n max_distance,\n max_distance,\n ))\n offset_masks = []\n for y_start in range(2 * max_distance + 1):\n y_end = y_start + height\n masks_slice = padded_masks[y_start:y_end]\n for x_start in range(2 * max_distance + 1):\n x_end = x_start + width\n offset_mask = masks_slice[:, x_start:x_end]\n offset_masks.append(offset_mask)\n offset_masks = paddle.stack(offset_masks, axis=2)\n d_tiled = d.unsqueeze(-1).tile((1, 1, 1, gt_ids.shape[0]))\n pad = paddle.ones_like(d_tiled)\n d_masked = paddle.where(offset_masks, d_tiled, pad)\n dists = paddle.min(d_masked, axis=2)\n dists = dists.reshape([1, height, width, gt_ids.shape[0], 1])\n return dists\n##############################################################\n#################\nclass _res_block(nn.Layer):\n def __init__(self, in_dim, out_dim):"
+ },
+ {
+ "comment": "This code defines a Residual Block and an Instance Segmentation Head for the Ma-Net model. The Residual Block consists of two 3x3 convolutions, batch normalization, and ReLU activations, while the IntSegHead layer takes in a specific input dimension for instance segmentation tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":454-485",
+ "content": " super(_res_block, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,\n out_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.relu1 = nn.ReLU()\n self.bn1 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg.TRAIN_BN_MOM)\n self.conv2 = nn.Conv2D(out_dim,\n out_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.relu2 = nn.ReLU()\n self.bn2 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg.TRAIN_BN_MOM)\n def forward(self, x):\n res = x\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu2(x)\n x += res\n return x\n####################\nclass IntSegHead(nn.Layer):\n def __init__(self,\n in_dim=(cfg.MODEL_SEMANTIC_EMBEDDING_DIM + 3),"
+ },
+ {
+ "comment": "This code defines a neural network class called \"IntSegHead\" for segmentation tasks. It consists of multiple convolutional and batch normalization layers, followed by ReLU activations. The output is passed through another convolutional layer before being fed into the final convolutional layer to produce the result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":486-512",
+ "content": " emb_dim=cfg.MODEL_HEAD_EMBEDDING_DIM):\n super(IntSegHead, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,\n emb_dim,\n kernel_size=7,\n stride=1,\n padding=3)\n self.bn1 = paddle.nn.BatchNorm2D(emb_dim, momentum=cfg.TRAIN_BN_MOM)\n self.relu1 = nn.ReLU(True)\n self.res1 = _res_block(emb_dim, emb_dim)\n self.res2 = _res_block(emb_dim, emb_dim)\n self.conv2 = nn.Conv2D(256, emb_dim, kernel_size=3, stride=1, padding=1)\n self.bn2 = paddle.nn.BatchNorm2D(emb_dim, momentum=cfg.TRAIN_BN_MOM)\n self.relu2 = nn.ReLU(True)\n self.conv3 = nn.Conv2D(emb_dim, 1, 1, 1)\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.res1(x)\n x = self.res2(x)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu2(x)\n x = self.conv3(x)\n return x"
+ },
+ {
+ "comment": "This code defines a custom layer \"_split_separable_conv2d\" that performs separable convolution using two consecutive 2D convolutions. It consists of two 2D convolutions separated by Batch Normalization and ReLU activation functions. The first convolution is followed by Batch Normalization and ReLU, while the second convolution is also followed by another Batch Normalization and ReLU. Weights are initialized using Kaiming Normal initialization for both convolutions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":515-536",
+ "content": "class _split_separable_conv2d(nn.Layer):\n def __init__(self, in_dim, out_dim, kernel_size=7):\n super(_split_separable_conv2d, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,\n in_dim,\n kernel_size=kernel_size,\n stride=1,\n padding=int((kernel_size - 1) / 2),\n groups=in_dim)\n self.relu1 = nn.ReLU(True)\n self.bn1 = paddle.nn.BatchNorm2D(in_dim, momentum=cfg.TRAIN_BN_MOM)\n self.conv2 = nn.Conv2D(in_dim, out_dim, kernel_size=1, stride=1)\n self.relu2 = nn.ReLU(True)\n self.bn2 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg.TRAIN_BN_MOM)\n kaiming_normal_(self.conv1.weight, mode='fan_out', nonlinearity='relu')\n kaiming_normal_(self.conv2.weight, mode='fan_out', nonlinearity='relu')\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.conv2(x)"
+ },
+ {
+ "comment": "The code defines a DynamicSegHead class with four split separable convolutional layers, followed by a 1x1 convolution. It also initializes an IntVOS class that takes in the configuration and feature extractor as parameters. The classes are used for semantic segmentation tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":537-570",
+ "content": " x = self.bn2(x)\n x = self.relu2(x)\n return x\nclass DynamicSegHead(nn.Layer):\n def __init__(self,\n in_dim=(cfg.MODEL_SEMANTIC_EMBEDDING_DIM + 3),\n embed_dim=cfg.MODEL_HEAD_EMBEDDING_DIM,\n kernel_size=1):\n super(DynamicSegHead, self).__init__()\n self.layer1 = _split_separable_conv2d(in_dim, embed_dim)\n self.layer2 = _split_separable_conv2d(embed_dim, embed_dim)\n self.layer3 = _split_separable_conv2d(embed_dim, embed_dim)\n self.layer4 = _split_separable_conv2d(embed_dim, embed_dim)\n self.conv = nn.Conv2D(embed_dim, 1, 1, 1)\n kaiming_normal_(self.conv.weight, mode='fan_out', nonlinearity='relu')\n def forward(self, x):\n x = self.layer1(x)\n x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n x = self.conv(x)\n return x\n##################\n###############\nclass IntVOS(nn.Layer):\n def __init__(self, cfg, feature_extracter):\n super(IntVOS, self).__init__()"
+ },
+ {
+ "comment": "The code initializes components for a network architecture. It creates feature extractors, convolutional layers, batch normalization layers, and ReLU activation functions to process and extract semantic features from input data. These features will be used for tasks such as object detection or image classification.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":571-588",
+ "content": " self.feature_extracter = feature_extracter ##embedding extractor\n self.feature_extracter.cls_conv = nn.Sequential()\n self.feature_extracter.upsample4 = nn.Sequential()\n self.semantic_embedding = None\n self.seperate_conv = nn.Conv2D(cfg.MODEL_ASPP_OUTDIM,\n cfg.MODEL_ASPP_OUTDIM,\n kernel_size=3,\n stride=1,\n padding=1,\n groups=cfg.MODEL_ASPP_OUTDIM)\n self.bn1 = paddle.nn.BatchNorm2D(cfg.MODEL_ASPP_OUTDIM,\n momentum=cfg.TRAIN_BN_MOM)\n self.relu1 = nn.ReLU(True)\n self.embedding_conv = nn.Conv2D(cfg.MODEL_ASPP_OUTDIM,\n cfg.MODEL_SEMANTIC_EMBEDDING_DIM, 1, 1)\n self.relu2 = nn.ReLU(True)\n self.bn2 = paddle.nn.BatchNorm2D(cfg.MODEL_SEMANTIC_EMBEDDING_DIM,\n momentum=cfg.TRAIN_BN_MOM)"
+ },
+ {
+ "comment": "The code initializes the network's semantic embedding layer, consisting of a sequence of convolutional layers, and applies Kaiming initialization to the weights. It also creates a dynamic segmentation head (seghead) for propagation and an interaction segmentation head based on the configuration flag MODEL_USEIntSeg. The function defines the forward pass for the network, taking in various inputs such as image data, reference labels, and masks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":589-615",
+ "content": " self.semantic_embedding = nn.Sequential(*[\n self.seperate_conv, self.bn1, self.relu1, self.embedding_conv,\n self.bn2, self.relu2\n ])\n for m in self.semantic_embedding:\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')\n self.dynamic_seghead = DynamicSegHead() # propagation segm head\n if cfg.MODEL_USEIntSeg:\n self.inter_seghead = IntSegHead(\n in_dim=cfg.MODEL_SEMANTIC_EMBEDDING_DIM + 3)\n else:\n self.inter_seghead = DynamicSegHead(\n in_dim=cfg.MODEL_SEMANTIC_EMBEDDING_DIM +\n 2) # interaction segm head\n def forward(self,\n x=None,\n ref_scribble_label=None,\n previous_frame_mask=None,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,"
+ },
+ {
+ "comment": "This code splits the input feature into three parts, then if `global_map_tmp_dic` is None, it passes these parts and other parameters to `prop_seghead()`, which returns a dictionary. If `global_map_tmp_dic` is not None, it also passes `global_map_tmp_dic` as an additional parameter before calling `prop_seghead()`. The function then returns the returned dictionary and updates `global_map_tmp_dic`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":616-639",
+ "content": " global_map_tmp_dic=None,\n local_map_dics=None,\n interaction_num=None,\n start_annotated_frame=None,\n frame_num=None):\n x = self.extract_feature(x)\n # print('extract_feature:', x.mean().item())\n ref_frame_embedding, previous_frame_embedding, current_frame_embedding = paddle.split(\n x, num_or_sections=3, axis=0)\n if global_map_tmp_dic is None:\n dic = self.prop_seghead(\n ref_frame_embedding, previous_frame_embedding,\n current_frame_embedding, ref_scribble_label,\n previous_frame_mask, normalize_nearest_neighbor_distances,\n use_local_map, seq_names, gt_ids, k_nearest_neighbors,\n global_map_tmp_dic, local_map_dics, interaction_num,\n start_annotated_frame, frame_num, self.dynamic_seghead)\n return dic\n else:\n dic, global_map_tmp_dic = self.prop_seghead(\n ref_frame_embedding, previous_frame_embedding,"
+ },
+ {
+ "comment": "The code defines a function that takes various inputs including frame embeddings, scribble labels, and masked frames. It performs feature extraction using a predefined feature extractor and semantic embedding. The function then returns the extracted features and temporary global map dictionaries.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":640-663",
+ "content": " current_frame_embedding, ref_scribble_label,\n previous_frame_mask, normalize_nearest_neighbor_distances,\n use_local_map, seq_names, gt_ids, k_nearest_neighbors,\n global_map_tmp_dic, local_map_dics, interaction_num,\n start_annotated_frame, frame_num, self.dynamic_seghead)\n return dic, global_map_tmp_dic\n def extract_feature(self, x):\n x = self.feature_extracter(x)\n x = self.semantic_embedding(x)\n return x\n def prop_seghead(self,\n ref_frame_embedding=None,\n previous_frame_embedding=None,\n current_frame_embedding=None,\n ref_scribble_label=None,\n previous_frame_mask=None,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,\n global_map_tmp_dic=None,"
+ },
+ {
+ "comment": "This code defines a function that takes various inputs and returns feature_embedding, global_match_map, local_match_map, and previous_frame_mask. It performs interpolation on the ref_scribble_label and previous_frame_mask using nearest mode to resize them to the same size as current_frame_embedding.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":664-684",
+ "content": " local_map_dics=None,\n interaction_num=None,\n start_annotated_frame=None,\n frame_num=None,\n dynamic_seghead=None):\n \"\"\"return: feature_embedding,global_match_map,local_match_map,previous_frame_mask\"\"\"\n ###############\n global_map_tmp_dic = global_map_tmp_dic\n dic_tmp = {}\n bs, c, h, w = current_frame_embedding.shape\n if cfg.TEST_MODE:\n scale_ref_scribble_label = float_(ref_scribble_label)\n else:\n scale_ref_scribble_label = paddle.nn.functional.interpolate(\n float_(ref_scribble_label), size=(h, w), mode='nearest')\n scale_ref_scribble_label = int_(scale_ref_scribble_label)\n scale_previous_frame_label = paddle.nn.functional.interpolate(\n float_(previous_frame_mask), size=(h, w), mode='nearest')\n # print(scale_previous_frame_label.sum()) # xx\n # print(previous_frame_mask.sum().item()) # xx"
+ },
+ {
+ "comment": "In this code snippet, we see the process of extracting nearest neighbor features per object for each batch of frames. The frames are transposed and labeled before finding the k_nearest_neighbors. These operations are performed within a loop for every frame in the batch (bs).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":685-703",
+ "content": " scale_previous_frame_label = int_(scale_previous_frame_label)\n # print(scale_previous_frame_label.sum().item()) # xx\n for n in range(bs):\n seq_current_frame_embedding = current_frame_embedding[n]\n seq_ref_frame_embedding = ref_frame_embedding[n]\n seq_prev_frame_embedding = previous_frame_embedding[n]\n seq_ref_frame_embedding = seq_ref_frame_embedding.transpose(\n [1, 2, 0])\n seq_current_frame_embedding = seq_current_frame_embedding.transpose(\n [1, 2, 0])\n seq_ref_scribble_label = scale_ref_scribble_label[n].transpose(\n [1, 2, 0])\n #########Global Map\n nn_features_n, ref_obj_ids = nearest_neighbor_features_per_object(\n reference_embeddings=seq_ref_frame_embedding,\n query_embeddings=seq_current_frame_embedding,\n reference_labels=seq_ref_scribble_label,\n k_nearest_neighbors=k_nearest_neighbors,"
+ },
+ {
+ "comment": "This code segment checks if the current sequence name exists in the global map temporary dictionary. If it does not exist, a paddle.ones_like(nn_features_n) is created and assigned to the dictionary with shape [104, 1, 1, 1, 1]. Then, the code performs a where operation using nn_features_n, comparing it to the global map value for the current sequence name and frame number. If nn_features_n is less than or equal to the global map value, it remains unchanged; otherwise, the global map value overwrites nn_features_n. The last line transposes seq_prev_frame_embedding before continuing with the next chunk of code.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":704-724",
+ "content": " gt_ids=gt_ids[n],\n n_chunks=10)\n if normalize_nearest_neighbor_distances:\n nn_features_n = (paddle.nn.functional.sigmoid(nn_features_n) -\n 0.5) * 2\n if global_map_tmp_dic is not None: ###when testing, use global map memory\n if seq_names[n] not in global_map_tmp_dic:\n global_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).tile([104, 1, 1, 1, 1])\n nn_features_n = paddle.where(\n nn_features_n <=\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0),\n nn_features_n,\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0))\n global_map_tmp_dic[seq_names[n]][\n frame_num[n]] = nn_features_n.detach()[0]\n #########################Local dist map\n seq_prev_frame_embedding = seq_prev_frame_embedding.transpose("
+ },
+ {
+ "comment": "The code is performing nearest neighbor feature extraction for previous frames in a video sequence. It checks if the use_local_map flag is set, and depending on its value, either uses local_previous_frame_nearest_neighbor_features_per_object function or nearest_neighbor_features_per_object function to extract features. If use_local_map is true, it takes previous frame embedding, current frame embedding, previous frame labels, reference object IDs and max distance as inputs. Otherwise, it takes previous frame embeddings, current frame embeddings, previous frame labels, k-nearest neighbors, gt_ids (for current iteration), and number of chunks as inputs. The code then assigns the extracted features to prev_frame_nn_features_n variable.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":725-744",
+ "content": " [1, 2, 0])\n seq_previous_frame_label = scale_previous_frame_label[n].transpose(\n [1, 2, 0])\n if use_local_map:\n prev_frame_nn_features_n = local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding=seq_prev_frame_embedding,\n query_embedding=seq_current_frame_embedding,\n prev_frame_labels=seq_previous_frame_label,\n gt_ids=ref_obj_ids,\n max_distance=cfg.MODEL_MAX_LOCAL_DISTANCE)\n else:\n prev_frame_nn_features_n, _ = nearest_neighbor_features_per_object(\n reference_embeddings=seq_prev_frame_embedding,\n query_embeddings=seq_current_frame_embedding,\n reference_labels=seq_previous_frame_label,\n k_nearest_neighbors=k_nearest_neighbors,\n gt_ids=gt_ids[n],\n n_chunks=20)\n prev_frame_nn_features_n = ("
+ },
+ {
+ "comment": "This code segment is checking if the current sequence name is present in the local map dictionaries for distance and temporary maps. If it's not, it creates new entries with zeros initialized. The local map distance value is then updated based on the frame number and interaction number, using the absolute difference from a start annotated frame to determine the distance. This could be used in a video sequence processing context where local map dictionaries store temporary and distance maps for different sequences.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":745-763",
+ "content": " paddle.nn.functional.sigmoid(prev_frame_nn_features_n) -\n 0.5) * 2\n# print(prev_frame_nn_features_n.mean().item(), prev_frame_nn_features_n.shape, interaction_num) # o\n#############\n if local_map_dics is not None: ##When testing, use local map memory\n local_map_tmp_dic, local_map_dist_dic = local_map_dics\n if seq_names[n] not in local_map_dist_dic:\n print(seq_names[n], 'not in local_map_dist_dic')\n local_map_dist_dic[seq_names[n]] = paddle.zeros(104, 9)\n if seq_names[n] not in local_map_tmp_dic:\n print(seq_names[n], 'not in local_map_tmp_dic')\n local_map_tmp_dic[seq_names[n]] = paddle.zeros_like(\n prev_frame_nn_features_n).unsqueeze(0).tile(\n [104, 9, 1, 1, 1, 1])\n local_map_dist_dic[seq_names[n]][\n frame_num[n], interaction_num -\n 1] = 1.0 / (abs(frame_num[n] - start_annotated_frame)"
+ },
+ {
+ "comment": "This code block appears to be part of a larger function. It seems to store and update the features of previous frames for a given sequence, based on the interaction number and frame number. If the current interaction's distance is greater than the previous one, it updates the previous frame features. The code uses dictionaries to store these features, with the frame number and interaction number as keys. The detach() function seems to remove the feature tensor from the computation graph for memory efficiency, while unsqueeze(0) reshapes the tensor to have a batch dimension.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":764-780",
+ "content": " ) # bugs fixed.\n local_map_tmp_dic[seq_names[n]][\n frame_num[n],\n interaction_num - 1] = prev_frame_nn_features_n.squeeze(\n 0).detach() # bugs fixed.\n if interaction_num == 1:\n prev_frame_nn_features_n = local_map_tmp_dic[seq_names[n]][\n frame_num[n]][interaction_num - 1]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n else:\n if local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num - 1] > \\\n local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num - 2]:\n prev_frame_nn_features_n = local_map_tmp_dic[\n seq_names[n]][frame_num[n]][interaction_num - 1]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)"
+ },
+ {
+ "comment": "This code snippet is part of a neural network model for video object detection. It deals with handling previous frame features, categorizing frames based on the reference object IDs, and concatenating different tensor inputs together. The code checks if the current frame's label matches the reference object ID, unsqueezes and tiles the tensors accordingly, transposes them, and finally concatenates these transformed tensors using `paddle.concat`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":781-802",
+ "content": " else:\n prev_frame_nn_features_n = local_map_tmp_dic[\n seq_names[n]][frame_num[n]][interaction_num - 2]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n local_map_dics = (local_map_tmp_dic, local_map_dist_dic)\n to_cat_previous_frame = (\n float_(seq_previous_frame_label) == float_(ref_obj_ids)\n ) # float comparision?\n to_cat_current_frame_embedding = current_frame_embedding[\n n].unsqueeze(0).tile((ref_obj_ids.shape[0], 1, 1, 1))\n to_cat_nn_feature_n = nn_features_n.squeeze(0).transpose(\n [2, 3, 0, 1])\n to_cat_previous_frame = float_(\n to_cat_previous_frame.unsqueeze(-1).transpose([2, 3, 0, 1]))\n to_cat_prev_frame_nn_feature_n = prev_frame_nn_features_n.squeeze(\n 0).transpose([2, 3, 0, 1])\n to_cat = paddle.concat("
+ },
+ {
+ "comment": "This function, int_seghead, takes various inputs such as reference frame embedding, scribble label, previous round label etc. It normalizes nearest neighbor distances if specified and returns the dictionary temporary (dic_tmp) containing predicted results for each sequence, along with optional global map temporary dictionary (global_map_tmp_dic) and local map dictionaries (local_map_dics). The interaction number (interaction_num), frame number (frame_num) and list of sequence names (seq_names) are also used.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":803-828",
+ "content": " (to_cat_current_frame_embedding, to_cat_nn_feature_n,\n to_cat_prev_frame_nn_feature_n, to_cat_previous_frame), 1)\n pred_ = dynamic_seghead(to_cat)\n pred_ = pred_.transpose([1, 0, 2, 3])\n dic_tmp[seq_names[n]] = pred_\n if global_map_tmp_dic is None:\n return dic_tmp\n else:\n if local_map_dics is None:\n return dic_tmp, global_map_tmp_dic\n else:\n return dic_tmp, global_map_tmp_dic, local_map_dics\n def int_seghead(self,\n ref_frame_embedding=None,\n ref_scribble_label=None,\n prev_round_label=None,\n normalize_nearest_neighbor_distances=True,\n global_map_tmp_dic=None,\n local_map_dics=None,\n interaction_num=None,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,\n frame_num=None,"
+ },
+ {
+ "comment": "This code segment is part of the Ma-Net network in PaddleVideo. It interpolates the reference scribble label and previous round label images, assigns ground truth IDs, and performs local distance map calculations on a sequence of frames for a batch of videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":829-852",
+ "content": " first_inter=True):\n dic_tmp = {}\n bs, c, h, w = ref_frame_embedding.shape\n scale_ref_scribble_label = paddle.nn.functional.interpolate(\n float_(ref_scribble_label), size=(h, w), mode='nearest')\n scale_ref_scribble_label = int_(scale_ref_scribble_label)\n if not first_inter:\n scale_prev_round_label = paddle.nn.functional.interpolate(\n float_(prev_round_label), size=(h, w), mode='nearest')\n scale_prev_round_label = int_(scale_prev_round_label)\n n_chunks = 500\n for n in range(bs):\n gt_id = paddle.arange(0, gt_ids[n] + 1)\n gt_id = int_(gt_id)\n seq_ref_frame_embedding = ref_frame_embedding[n]\n ########################Local dist map\n seq_ref_frame_embedding = paddle.transpose(seq_ref_frame_embedding,\n [1, 2, 0])\n seq_ref_scribble_label = paddle.transpose(\n scale_ref_scribble_label[n], [1, 2, 0])"
+ },
+ {
+ "comment": "This code segment is updating the global and local maps for a given sequence of frames. It first calculates the nearest neighbor features (nn_features_n) using the previous frame embedding, query embedding, previous frame labels, and ground truth IDs. Then it checks if this current sequence name exists in the global map temporary dictionary (global_map_tmp_dic). If not, it initializes a one-tensor for that sequence and tiles it to match the shape of nn_features_n. It then applies a where statement to compare nn_features_n with the global map tensor, selecting either nn_features_n or the global map tensor depending on which is smaller. Finally, it updates the global map temporary dictionary entry for this sequence at the current frame number with the selected tensor from the where statement.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":853-876",
+ "content": " nn_features_n = local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding=seq_ref_frame_embedding,\n query_embedding=seq_ref_frame_embedding,\n prev_frame_labels=seq_ref_scribble_label,\n gt_ids=gt_id,\n max_distance=cfg.MODEL_MAX_LOCAL_DISTANCE)\n #######\n ######################Global map update\n if seq_names[n] not in global_map_tmp_dic:\n global_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).tile([104, 1, 1, 1, 1])\n nn_features_n_ = paddle.where(\n nn_features_n <=\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0),\n nn_features_n,\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0))\n ###\n ###\n global_map_tmp_dic[seq_names[n]][\n frame_num[n]] = nn_features_n_.detach()[0]\n ##################Local map update"
+ },
+ {
+ "comment": "The code checks if a dictionary of local maps is provided. If so, it retrieves the temporary and distance dictionaries from it. It then updates these dictionaries for the current sequence name (seq_names[n]), adding a 0 to a specific element if the sequence name is not already in the distance dictionary. Finally, it creates embedding tensors for frame and feature comparison and prepares them for concatenation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":877-896",
+ "content": " if local_map_dics is not None:\n local_map_tmp_dic, local_map_dist_dic = local_map_dics\n if seq_names[n] not in local_map_dist_dic:\n local_map_dist_dic[seq_names[n]] = paddle.zeros([104, 9])\n if seq_names[n] not in local_map_tmp_dic:\n local_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).unsqueeze(0).tile([104, 9, 1, 1, 1, 1])\n local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num -\n 1] = 0\n local_map_dics = (local_map_tmp_dic, local_map_dist_dic)\n ##################\n to_cat_current_frame_embedding = ref_frame_embedding[n].unsqueeze(\n 0).tile((gt_id.shape[0], 1, 1, 1))\n to_cat_nn_feature_n = nn_features_n.squeeze(0).transpose(\n [2, 3, 0, 1])\n to_cat_scribble_mask_to_cat = (\n float_(seq_ref_scribble_label) == float_(gt_id)"
+ },
+ {
+ "comment": "This code is performing a series of operations on tensors to create the 'to_cat' tensor for use in the model. It checks if it's the first iteration and adjusts the previous round label accordingly. Then, it concatenates three different tensor inputs along the 1st axis (channel dimension) and passes the result through a segmentation head network to get the final prediction 'pred_'. This code seems to be part of a larger neural network for video object segmentation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":897-920",
+ "content": " ) # float comparision?\n to_cat_scribble_mask_to_cat = float_(\n to_cat_scribble_mask_to_cat.unsqueeze(-1).transpose(\n [2, 3, 0, 1]))\n if not first_inter:\n seq_prev_round_label = scale_prev_round_label[n].transpose(\n [1, 2, 0])\n to_cat_prev_round_to_cat = (\n float_(seq_prev_round_label) == float_(gt_id)\n ) # float comparision?\n to_cat_prev_round_to_cat = float_(\n to_cat_prev_round_to_cat.unsqueeze(-1).transpose(\n [2, 3, 0, 1]))\n else:\n to_cat_prev_round_to_cat = paddle.zeros_like(\n to_cat_scribble_mask_to_cat)\n to_cat_prev_round_to_cat[0] = 1.\n to_cat = paddle.concat(\n (to_cat_current_frame_embedding, to_cat_scribble_mask_to_cat,\n to_cat_prev_round_to_cat), 1)\n pred_ = self.inter_seghead(to_cat)"
+ },
+ {
+ "comment": "This code is transposing the tensor 'pred_' and storing it in 'dic_tmp' with corresponding sequence name as key. It then checks if 'local\\_map\\_dics' is None, and returns 'dic\\_tmp' or returns both 'dic\\_tmp' and 'local\\_map\\_dics'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/IntVOS.py\":921-926",
+ "content": " pred_ = pred_.transpose([1, 0, 2, 3])\n dic_tmp[seq_names[n]] = pred_\n if local_map_dics is None:\n return dic_tmp\n else:\n return dic_tmp, local_map_dics"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/14314a35-bbab-42ff-9a94-5625a4383107.json b/docs/doc/14314a35-bbab-42ff-9a94-5625a4383107.json
new file mode 100644
index 000000000..557a1f20e
--- /dev/null
+++ b/docs/doc/14314a35-bbab-42ff-9a94-5625a4383107.json
@@ -0,0 +1,65 @@
+{
+ "summary": "Video classification tasks involve recognizing actions through RGB images and skeleton data. Concepts include temporal action localization, dense-captioning events, popular datasets, feature extraction, motion representation, and classification using deep learning methods since 2014.",
+ "details": [
+ {
+ "comment": "Introduction to video classification (action recognition) with various applications in different fields, including online platforms and offline sectors like security, transportation, and quality inspection. Tasks include classification/recognition and detection, further subdivided by combining different scenes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":0-17",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../zh-CN/tutorials/summarize.md) | English\n# Introduction for video classification(action recognition)\n## Wide range of application scenarios\nVideo classification has a wide range of applications in many fields, such as online video platforms such as short videos, offline such as security, transportation, quality inspection and other fields\u3002\n## Multiple subtasks\nSimilar to image tasks, video tasks can also be divided into two categories: **classification (recognition) and detection**, and these two types of tasks can be specifically subdivided by combining different scenes\uff1a\n+ Task1\uff1aTrimmed Action Recognition. Users input a trimmed video,which contains only single action,then a video tag will be output by model as depicted in fig below:\n\n
\n Action Classification\n
\n In terms of the data modality used, classification tasks can be further subdivided into classification based on si"
+ },
+ {
+ "comment": "This code is describing different types of classification tasks in video analysis. It covers multi-modality data, RGB images, human skeleton data, and various perspectives such as first-person, third-person, and multiple perspectives. Additionally, it mentions untrimmed videos, temporal action proposals, and ROI extraction in image detection tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":17-31",
+ "content": "ngle modality data, classification based on multi-modality data, classification based on RGB images and classification based on human skeleton, etc, as shown in the figure below:\n \n
\n multi-modality\n
\nIn terms of the perspective of video, it can also be divided into first-person action recognition, \nthird-person action recognition, single perspective action recognition and multi-perspective fusion action recognition. \nUsers who are interested in these fields can refer to relevant literatures.\n+ Task2\uff1aUntrimmed Video Classification. \nUnlike trimmed videos, untrimmed videos often contain multiple actions and have a long time span. \nThere are a lot of movements that we may need not paying attention to. Through the global analysis of the input long video, and then make a soft classify to mutiple categories.\n+ Task3\uff1aTemporal Action Proposal. It is similar to the ROI extraction in the image detection task. "
+ },
+ {
+ "comment": "Task 4: Temporal Action Localization - find video segments with possible actions, classify them.\nTask 5: Dense-Captioning Events - describe untrimmed videos' actions in temporal dimension.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":32-48",
+ "content": "The task is to find the video clips that may contain action in a long video with a lot of actions.\n+ Task4\uff1aTemporal Action Localization. Compared with the temporal action proposal task as mentioned above, \ntemporal action localization task is more consistent with detection task in the field of imgae, \nit requires not only to find the video segments with possible actions from the video but also to classify them,\nas shown in the figure below\n \n
\n Action Detection\n
\n+ Task5\uff1aDense-Captioning Events. The reason why it is called dense captioning events is mainly \nbecause that this task requires video action description on the basis of temporal action localization \n(detection). That is to say, the task needs to locate the actions in a **untrimmed** video,in **temporal \ndimension** and describe the behavior of the **whole video** after obtaining many video segments which contain actions.\n## Introduction of datasets"
+ },
+ {
+ "comment": "The code provides a brief overview of popular video action recognition datasets, such as KTH and UCF101. It mentions that the datasets are essential for training and validating models, but overfitting may occur with larger 3D networks on smaller datasets like KTH.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":50-71",
+ "content": "### Classification datasets\nThe training and validation of the model cannot be done without comprehensive, \nlarge and well annotated datasets. With the deepening of research on video action recognition, \nmore and more datasets are applied to the research in this field. \nTypical datasets are as follows:\n+ KTH[1](#1)\nKTH dataset is an early small action recognition dataset, \nincluding 599 videos of 6 types of actions (walking, jumping, running, punching, waving and clapping). \nThe background is relatively still, except for the zoom in and out of the camera, \nthe camera movement is relatively slight. Since this data set is relatively small, \nit is easy to overfit when training heavy 3D networks, \nso most current researches are not based on this it.\n+ UCF10[2](#2)\nUCF101 is a medium-size dataset in which most videos are from YouTube. \nIt contains 13,320 videos with 101 types of actions. \nEach type of action is performed by 25 people, each of whom performs 4-7 sets of actions. \nThe UCF101 and HMDB51 datasets used to be the benchmarks to evaluate the effectiveness of action "
+ },
+ {
+ "comment": "HMDB51 is a dataset proposed by Brown University in 2011, consisting of movie and online video sources. It contains 6849 samples across 51 classes with at least 101 samples each. Kinetics is the largest action recognition dataset, created by Google's DeepMind team in 2017. It uses YouTube videos, now expanded to 600k videos in 700 categories. The categories are divided into human, human and animal, and human and human interaction. Kinetics can train deep networks like 3D-RESNET up to 152 layers without overfitting, solving the issue of small training datasets.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":72-86",
+ "content": "recognition model for a long time before the Kinetics dataset was released.\n+ HMDB51[3](#3)\nBrown University's proposed dataset named HMDB51 was released in 2011. \nMost of the videos come from movies, \nbut some come from public databases and online video libraries such as YouTube. \nThe datasets contains 6849 samples divided into 51 classes, \neach of which contains at least 101 samples.\n+ Kinetics[4](#4)\nKinetics is the most important large-scale action recognition dataset, which was proposed by Google's DeepMind team in 2017. The video data also comes from YouTube, with 400 categories (now expanded to 700 categories) and more than 300,000 videos (now expanded to 600,000 videos), each lasting about 10 seconds. \nThe action categories are mainly divided into three categories: \"human\", \"human and animal\", \"human and human interaction\". Kinetics can train 3D-RESNET up to 152 layers without over-fitting, \nwhich solves the problem that the previous training dataset is too small to train deep 3D network. "
+ },
+ {
+ "comment": "Kinetics is the benchmark for action recognition, replacing UCF101 and HMDB51. Most studies use this dataset for evaluation and pre-training. SomethingV1 has 108,499 annotated videos with 174 kinds of actions, requiring strong temporal modeling ability. Other datasets include Charades (complex action recognition), Breakfast Action, Sports 1M, THUMOS 2014 (action detection).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":87-103",
+ "content": "Kinetics has replaced UCF101 and HMDB51 as the benchmark in the field of action recognition. \nAt present, most studies use this dataset for evaluation and pre-training.\n+ Something-Something[5](#5)\nSomethingV1 contains 108,499 annotated videos (V2 has expanded to 220,847), each of which last two to six seconds. These videos contain 174 kinds of actions. Different from the previous dataset, \nthe identification of this data set requires stronger time information, \nso this dataset has a very important reference value in testing the temporal modeling ability of the model.\nIn addition to the above datasets, there are Charades[6](#6) dataset for complex Action recognition, Breakfast Action[7](#7), and Sports 1M[8](#8).\n### Detection datasets\n+ THUMOS 2014\nThis dataset is from THUMOS Challenge 2014, Its training set is UCF101, validation set and test set include 1010 and 1574 undivided video clips respectively. In the action detection task, only 20 kinds of unsegmented videos of actions were labeled with sequential action fragments, "
+ },
+ {
+ "comment": "The code describes two datasets, Mexaction2 and ActivityNet. Mexaction2 has horse riding and bullfighting actions, split into training, validation, and test sets. It includes YouTube clips, UCF101 horseback riding videos, and an unsegmented 77-hour INA video with low marked action proportions. ActivityNet is the largest database, including classification and detection tasks, but only provides YouTube links without direct downloads.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":104-120",
+ "content": "including 200 validation sets (3007 action fragments) and 213 test sets (3358 action fragments).\n+ MEXaction2\nThe Mexaction2 dataset contains two types of action: horse riding and bullfighting. \nThe dataset consists of three parts: YouTube videos, horseback riding videos in UCF101, and INA videos. \nYouTube clips and horseback riding videos in UCF101 are short segmented video clips that are used as training sets. \nThe INA video is a long unsegmented video with a total length of 77 hours, \nand it is divided into three parts: training, validation and test. \nThere are 1336 action segments in the training set, 310 in the validation set and 329 in the test set. \nMoreover, the Mexaction2 dataset is characterized by very long unsegmented video lengths, \nand marked action segments only account for a very low proportion of the total video length.\n+ ActivityNet\nAt present the largest database, also contains two tasks of classification and detection. \nThis dataset only provides a YouTube link to the video, not a direct download of the video, "
+ },
+ {
+ "comment": "The code discusses the process of feature extraction, motion representation and classification in action recognition. It highlights two stages - manual feature-based method and deep learning-based method. It mentions DTP and IDT as typical motion descriptors used before deep-learning was applied. The code also shows a framework diagram for action recognition.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":121-137",
+ "content": "so you also need to use the YouTube download tool in Python to automatically download the videos. \nThe dataset contains 200 action categories, 20,000 (training + verification + test set) videos, \nand a total of about 700 hours of video.\n## Introduction of classic models\nAs shown in the figure, \nthe action recognition framework mainly includes three steps: \nfeature extraction, motion representation and classification. \nHow to extract spatiotemporal features of video is the core problem of action recognition and video classification.\n \n
\nFramework of action recognition\n
\nAccording to different methods, action recognition (video classification) methods can be generally summarized into two stages: \nmanual feature-based method and deep learning-based method. \nTypical motion descriptors in the manual feature-based method stage include DTP and IDT, \nwhich are also the most excellent motion descriptors accepted by most researchers before deep-learning is applied in this field. "
+ },
+ {
+ "comment": "The code discusses the application of deep learning methods in video classification since 2014, highlighting their effectiveness beyond manual motion design. It mentions various classic network structures proposed by researchers for representing motion characteristics and includes images to illustrate these models. The code also introduces PaddleVideo's inclusion of such models like TSN, TSM, slowfast, etc., and anticipates future analysis of these classical models and papers in the field. Additionally, it references an ActivityNet competition for further context.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":138-153",
+ "content": "Interinterested readers may refer to the relevant references at the end of this paper. \nSince 2014, deep learning methods have been gradually applied to the field of video classification. \nAt present, deep learning-based methods have become a hotspot of research in both academic and the practice, and the effect is far beyond the motion features of manual design. \nSince 2014, many classic network structures have been put forward by the researchers regarding the problem of how to represent motion characteristics, \nas shown in the figure below:\n \n
\nClassic Models\n
\nAt present,Paddlevideo has contained several classic models such as:TSN[9](#9),TSM[10](#10),slowfast[11](#11),et al.In the future,\nwe will analyze the classic models and papers in these fields. Please look forward to it\n## Introduction of competetion\n+ [ActivityNet](http://activity-net.org/challenges/2020/challenge.html)"
+ },
+ {
+ "comment": "This code snippet provides information about the ActivityNet competition, which is a large-scale action recognition event held annually since 2016. It focuses on identifying everyday activities from user-generated YouTube videos and has become the most influential in the field of action recognition. The code also includes references to relevant research papers for further reading.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":155-172",
+ "content": "ActivityNet is a large-scale action recognition competition. Since 2016, \nit has been held simultaneously with CVPR every year. Up to this year, \nit has been held for 4 consecutive sessions. It focuses on identifying everyday, high-level, goal-oriented activities from \nuser-generated videos taken from the Internet video portal YouTube. \nAt present, ActivityNet competition has become the most influential competition in the field of action recognition.\n## Reference\n\n[1] Schuldt C, Laptev I, Caputo B.Recognizing Human Actions: A Local SVM Approach Proceedings of International Conference on Pattern Recognition. Piscataway, NJ: IEEE, 2004:23-26\n
\n
\n\n[2] Soomro K, Zamir A R, Shah M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv:1212.0402,2012.\n
\n
\n\n[3] Kuehne H, Jhuang H, Garrote E, et al. HMDB: a large video database for human motion recognition Proceedings of IEEE International Conference on Computer Vision. Piscataway, NJ: IEEE, 2011:2556-2563."
+ },
+ {
+ "comment": "This code provides references to various research papers related to action recognition and video classification, such as the \"Quo Vadis, Action Recognition?\" paper by Carreira and Zisserman, and the \"Hollywood in Homes\" paper by Sigurdsson et al. These references are from well-known conferences like IEEE Conference on Computer Vision and Pattern Recognition (CVPR) and arXiv preprints.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":173-192",
+ "content": "
\n
\n\n[4] Carreira J , Zisserman A . Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017:6299-6308.\n
\n
\n\n[5] Goyal R, Kahou S E, Michalski V. The \u201csomething something\u201d video database for learning and evaluating visual common sense. arXiv:1706.04261,2017.\n
\n
\n\n[6] Sigurdsson G A , Varol G\u00fcl, Wang Xiaolong, et al. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. arXiv: 604.01753,2016\n
\n
\n\n[7] Kuehne H, Arslan A, Serre T. The Language of Actions Recovering the Syntax and Semantics of Goal-Directed Human Activities Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2014.\n
\n
\n\n[8] Karpathy A , Toderici G , Shetty S , et al. Large-Scale Video Classification with Convolutional Neural Networks Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2014:1725-1732."
+ },
+ {
+ "comment": "The code represents a list of references for papers related to video recognition. Each reference has an identifier (id) and the corresponding paper details like authors, title, and publication information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/summarize.md\":193-205",
+ "content": "
\n
\n\n[9] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoo Tang,and Luc Van Gool. Temporal segment networks for action recognition in videos? In Proceedings of the European Conference on Computer Vision,pages 20\u201336. Springer, 2016.\n
\n
\n\n[10] Lin Ji , Gan Chuang , Han Song . TSM: Temporal Shift Module for Efficient Video Understanding. arXiv:1811.08383,2018.\n
\n
\n\n[11] Feichtenhofer C , Fan Haoqi , Malik J , et al. SlowFast Networks for Video Recognition. arXiv:1812.03982,2018.\n
"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/15ebb946-02e0-453d-a13e-f3597abefd86.json b/docs/doc/15ebb946-02e0-453d-a13e-f3597abefd86.json
new file mode 100644
index 000000000..fcba42590
--- /dev/null
+++ b/docs/doc/15ebb946-02e0-453d-a13e-f3597abefd86.json
@@ -0,0 +1,55 @@
+{
+ "summary": "The AveragePrecisionCalculator calculates average precision in video object detection tasks, using a priority queue and providing methods for non-interpolated average precision. It also includes sorting, recall & precision computation, data shuffling, and prediction normalization.",
+ "details": [
+ {
+ "comment": "This code calculates the interpolated average precision for an entire list or top-n ranked items, following the definition provided in the given reference. It can be used as a static function call to calculate average precision for short ranked lists in memory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":0-22",
+ "content": "# Copyright 2020 Google Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Calculate or keep track of the interpolated average precision.\nIt provides an interface for calculating interpolated average precision for an\nentire list or the top-n ranked items. For the definition of the\n(non-)interpolated average precision:\nhttp://trec.nist.gov/pubs/trec15/appendices/CE.MEASURES06.pdf\nExample usages:\n1) Use it as a static function call to directly calculate average precision for\na short ranked list in the memory."
+ },
+ {
+ "comment": "The code defines an AveragePrecisionCalculator class that calculates average precision based on a ranked list. The calculator can handle long lists that cannot fit in memory or partial predictions observed over time (such as from Tensorflow). It uses the accumulate method to process parts of the ranked list and peek_interpolated_ap_at_n to calculate the interpolated average precision at a specific recall level.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":24-54",
+ "content": "```\nimport random\np = np.array([random.random() for _ in xrange(10)])\na = np.array([random.choice([0, 1]) for _ in xrange(10)])\nap = average_precision_calculator.AveragePrecisionCalculator.ap(p, a)\n```\n2) Use it as an object for long ranked list that cannot be stored in memory or\nthe case where partial predictions can be observed at a time (Tensorflow\npredictions). In this case, we first call the function accumulate many times\nto process parts of the ranked list. After processing all the parts, we call\npeek_interpolated_ap_at_n.\n```\np1 = np.array([random.random() for _ in xrange(5)])\na1 = np.array([random.choice([0, 1]) for _ in xrange(5)])\np2 = np.array([random.random() for _ in xrange(5)])\na2 = np.array([random.choice([0, 1]) for _ in xrange(5)])\n# interpolated average precision at 10 using 1000 break points\ncalculator = average_precision_calculator.AveragePrecisionCalculator(10)\ncalculator.accumulate(p1, a1)\ncalculator.accumulate(p2, a2)\nap3 = calculator.peek_ap_at_n()\n```\n\"\"\"\nimport heapq\nimport random\nimport numbers"
+ },
+ {
+ "comment": "AveragePrecisionCalculator is a class used for calculating the average precision and average precision at n. It constructs an object to calculate average precision for single label, with optional top_n parameter for average precision at n. If top_n is not positive integer or None, a ValueError is raised. The class maintains heap of (prediction, actual) pairs and total positives seen. Heap size can be queried using the heap_size property.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":56-85",
+ "content": "import numpy\nclass AveragePrecisionCalculator(object):\n \"\"\"Calculate the average precision and average precision at n.\"\"\"\n def __init__(self, top_n=None):\n \"\"\"Construct an AveragePrecisionCalculator to calculate average precision.\n This class is used to calculate the average precision for a single label.\n Args:\n top_n: A positive Integer specifying the average precision at n, or\n None to use all provided data points.\n Raises:\n ValueError: An error occurred when the top_n is not a positive integer.\n \"\"\"\n if not ((isinstance(top_n, int) and top_n >= 0) or top_n is None):\n raise ValueError(\"top_n must be a positive integer or None.\")\n self._top_n = top_n # average precision at n\n self._total_positives = 0 # total number of positives have seen\n self._heap = [] # max heap of (prediction, actual)\n @property\n def heap_size(self):\n \"\"\"Gets the heap size maintained in the class.\"\"\"\n return len(self._heap)\n @property"
+ },
+ {
+ "comment": "This code defines a class that calculates the average precision in video object detection tasks. It provides methods to accumulate positive samples and return the total number of positives. The accumulate method takes prediction scores, ground truth labels, and optional num_positives parameter for accurate tracking when inputs are incomplete.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":86-107",
+ "content": " def num_accumulated_positives(self):\n \"\"\"Gets the number of positive samples that have been accumulated.\"\"\"\n return self._total_positives\n def accumulate(self, predictions, actuals, num_positives=None):\n \"\"\"Accumulate the predictions and their ground truth labels.\n After the function call, we may call peek_ap_at_n to actually calculate\n the average precision.\n Note predictions and actuals must have the same shape.\n Args:\n predictions: a list storing the prediction scores.\n actuals: a list storing the ground truth labels. Any value\n larger than 0 will be treated as positives, otherwise as negatives.\n num_positives = If the 'predictions' and 'actuals' inputs aren't complete,\n then it's possible some true positives were missed in them. In that case,\n you can provide 'num_positives' in order to accurately track recall.\n Raises:\n ValueError: An error occurred when the format of the input is not the\n numpy 1-D array or the shape of predictions and actuals does not match."
+ },
+ {
+ "comment": "This code checks if the length of predictions and actuals match. It also ensures that num_positives is a nonzero number, then adds positives to total_positives. The code uses heapq to push and pop elements in a priority queue based on top_n, ensuring correctness and efficiency.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":108-133",
+ "content": " \"\"\"\n if len(predictions) != len(actuals):\n raise ValueError(\n \"the shape of predictions and actuals does not match.\")\n if not num_positives is None:\n if not isinstance(num_positives,\n numbers.Number) or num_positives < 0:\n raise ValueError(\n \"'num_positives' was provided but it wan't a nonzero number.\"\n )\n if not num_positives is None:\n self._total_positives += num_positives\n else:\n self._total_positives += numpy.size(numpy.where(actuals > 0))\n topk = self._top_n\n heap = self._heap\n for i in range(numpy.size(predictions)):\n if topk is None or len(heap) < topk:\n heapq.heappush(heap, (predictions[i], actuals[i]))\n else:\n if predictions[i] > heap[0][0]: # heap[0] is the smallest\n heapq.heappop(heap)\n heapq.heappush(heap, (predictions[i], actuals[i]))"
+ },
+ {
+ "comment": "This code defines a class that calculates non-interpolated average precision. It has methods to clear accumulated predictions, peek the non-interpolated average precision at n, and calculate non-interpolated average precision from prediction and actual scores. The class uses numpy arrays and requires positive labels to be greater than 0 and negative labels as 0.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":135-165",
+ "content": " def clear(self):\n \"\"\"Clear the accumulated predictions.\"\"\"\n self._heap = []\n self._total_positives = 0\n def peek_ap_at_n(self):\n \"\"\"Peek the non-interpolated average precision at n.\n Returns:\n The non-interpolated average precision at n (default 0).\n If n is larger than the length of the ranked list,\n the average precision will be returned.\n \"\"\"\n if self.heap_size <= 0:\n return 0\n predlists = numpy.array(list(zip(*self._heap)))\n ap = self.ap_at_n(predlists[0],\n predlists[1],\n n=self._top_n,\n total_num_positives=self._total_positives)\n return ap\n @staticmethod\n def ap(predictions, actuals):\n \"\"\"Calculate the non-interpolated average precision.\n Args:\n predictions: a numpy 1-D array storing the sparse prediction scores.\n actuals: a numpy 1-D array storing the ground truth labels. Any value\n larger than 0 will be treated as positives, otherwise as negatives."
+ },
+ {
+ "comment": "This code calculates the non-interpolated average precision at a specified number 'n' from given predictions and actuals. It raises a ValueError if the input format is not a numpy 1D array, or the shape of predictions and actuals does not match.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":167-191",
+ "content": " Returns:\n The non-interpolated average precision at n.\n If n is larger than the length of the ranked list,\n the average precision will be returned.\n Raises:\n ValueError: An error occurred when the format of the input is not the\n numpy 1-D array or the shape of predictions and actuals does not match.\n \"\"\"\n return AveragePrecisionCalculator.ap_at_n(predictions, actuals, n=None)\n @staticmethod\n def ap_at_n(predictions, actuals, n=20, total_num_positives=None):\n \"\"\"Calculate the non-interpolated average precision.\n Args:\n predictions: a numpy 1-D array storing the sparse prediction scores.\n actuals: a numpy 1-D array storing the ground truth labels. Any value\n larger than 0 will be treated as positives, otherwise as negatives.\n n: the top n items to be considered in ap@n.\n total_num_positives : (optionally) you can specify the number of total\n positive\n in the list. If specified, it will be used in calculation.\n Returns:"
+ },
+ {
+ "comment": "This function calculates the non-interpolated average precision at a given rank 'n'. It checks if the lengths of predictions and actuals match, ensures 'n' is an integer greater than zero, and shuffles the lists to avoid overestimation. If any errors occur, it raises a ValueError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":192-219",
+ "content": " The non-interpolated average precision at n.\n If n is larger than the length of the ranked list,\n the average precision will be returned.\n Raises:\n ValueError: An error occurred when\n 1) the format of the input is not the numpy 1-D array;\n 2) the shape of predictions and actuals does not match;\n 3) the input n is not a positive integer.\n \"\"\"\n if len(predictions) != len(actuals):\n raise ValueError(\n \"the shape of predictions and actuals does not match.\")\n if n is not None:\n if not isinstance(n, int) or n <= 0:\n raise ValueError(\"n must be 'None' or a positive integer.\"\n \" It was '%s'.\" % n)\n ap = 0.0\n predictions = numpy.array(predictions)\n actuals = numpy.array(actuals)\n # add a shuffler to avoid overestimating the ap\n predictions, actuals = AveragePrecisionCalculator._shuffle(\n predictions, actuals)\n sortidx = sorted(range(len(predictions)),"
+ },
+ {
+ "comment": "This function calculates the average precision (AP) of a set of predictions and actuals. It first sorts the predictions by value, then calculates recall and precision to compute AP. If a total number of positives is provided, it uses that instead of counting non-zero actuals. The function also includes helper methods for shuffling the data and normalizing predictions with an epsilon value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":220-255",
+ "content": " key=lambda k: predictions[k],\n reverse=True)\n if total_num_positives is None:\n numpos = numpy.size(numpy.where(actuals > 0))\n else:\n numpos = total_num_positives\n if numpos == 0:\n return 0\n if n is not None:\n numpos = min(numpos, n)\n delta_recall = 1.0 / numpos\n poscount = 0.0\n # calculate the ap\n r = len(sortidx)\n if n is not None:\n r = min(r, n)\n for i in range(r):\n if actuals[sortidx[i]] > 0:\n poscount += 1\n ap += poscount / (i + 1) * delta_recall\n return ap\n @staticmethod\n def _shuffle(predictions, actuals):\n random.seed(0)\n suffidx = random.sample(range(len(predictions)), len(predictions))\n predictions = predictions[suffidx]\n actuals = actuals[suffidx]\n return predictions, actuals\n @staticmethod\n def _zero_one_normalize(predictions, epsilon=1e-7):"
+ },
+ {
+ "comment": "This function normalizes the predictions to a range of 0.0-1.0, ensuring that the rank in the original list remains unchanged and does not affect the average precision calculation. It prevents division by zero using a small epsilon value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/average_precision_calculator.py\":256-273",
+ "content": " \"\"\"Normalize the predictions to the range between 0.0 and 1.0.\n For some predictions like SVM predictions, we need to normalize them before\n calculate the interpolated average precision. The normalization will not\n change the rank in the original list and thus won't change the average\n precision.\n Args:\n predictions: a numpy 1-D array storing the sparse prediction scores.\n epsilon: a small constant to avoid denominator being zero.\n Returns:\n The normalized prediction.\n \"\"\"\n denominator = numpy.max(predictions) - numpy.min(predictions)\n ret = (predictions - numpy.min(predictions)) / numpy.max(\n denominator, epsilon)\n return ret"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/164872e3-44dd-4f30-91e5-cb0d20319c74.json b/docs/doc/164872e3-44dd-4f30-91e5-cb0d20319c74.json
new file mode 100644
index 000000000..7c7d98157
--- /dev/null
+++ b/docs/doc/164872e3-44dd-4f30-91e5-cb0d20319c74.json
@@ -0,0 +1,95 @@
+{
+ "summary": "The code presents a CTRGCN backbone for video models, initializes a CTRGC model with batch normalization layers and NTUGraph class, defines a neural network model with TCN_GCN_unit, and includes a final layer 10 (l10) to process input and return output.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries, defines a convolution initialization function and a batch normalization initialization function. It also sets up scale values for the batch normalization function and registers backbone models in the registry.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":0-30",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\ndef conv_init(conv):\n if conv.weight is not None:\n weight_init_(conv.weight, 'kaiming_normal_', mode='fan_in')\n if conv.bias is not None:\n nn.initializer.Constant(value=0.0)(conv.bias)\ndef bn_init(bn, scale):\n nn.initializer.Constant(value=float(scale))(bn.weight)"
+ },
+ {
+ "comment": "Defines a CTRGC class, a type of convolutional neural network layer. It has two reductions: rel_reduction (defaults to 8) and mid_reduction (defaults to 1). Depending on the input channels, it assigns different channel numbers for rel_channels (always 8 if in_channels is 3 or 9; otherwise based on rel_reduction). It also initializes a Conv2D layer with the assigned channel numbers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":31-65",
+ "content": " nn.initializer.Constant(value=0.0)(bn.bias)\ndef einsum(x1, x3):\n \"\"\"paddle.einsum only support in dynamic graph mode.\n x1 : n c u v\n x2 : n c t v\n \"\"\"\n n, c, u, v1 = x1.shape\n n, c, t, v3 = x3.shape\n assert (v1 == v3), \"Args of einsum not match!\"\n x1 = paddle.transpose(x1, perm=[0, 1, 3, 2]) # n c v u\n y = paddle.matmul(x3, x1)\n # out: n c t u\n return y\nclass CTRGC(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n rel_reduction=8,\n mid_reduction=1):\n super(CTRGC, self).__init__()\n self.in_channels = in_channels\n self.out_channels = out_channels\n if in_channels == 3 or in_channels == 9:\n self.rel_channels = 8\n self.mid_channels = 16\n else:\n self.rel_channels = in_channels // rel_reduction\n self.mid_channels = in_channels // mid_reduction\n self.conv1 = nn.Conv2D(self.in_channels,\n self.rel_channels,"
+ },
+ {
+ "comment": "This code defines a Convolutional Temporal Relational Graph Convolutional Network (CTRGCN) backbone for a video model. It initializes weights and performs forward pass calculations. It uses convolution layers, tanh activation function, and optionally includes an additional input A.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":66-92",
+ "content": " kernel_size=1)\n self.conv2 = nn.Conv2D(self.in_channels,\n self.rel_channels,\n kernel_size=1)\n self.conv3 = nn.Conv2D(self.in_channels,\n self.out_channels,\n kernel_size=1)\n self.conv4 = nn.Conv2D(self.rel_channels,\n self.out_channels,\n kernel_size=1)\n self.tanh = nn.Tanh()\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n conv_init(m)\n elif isinstance(m, nn.BatchNorm2D):\n bn_init(m, 1)\n def forward(self, x, A=None, alpha=1):\n x1, x2, x3 = self.conv1(x).mean(-2), self.conv2(x).mean(-2), self.conv3(\n x)\n x1 = self.tanh(x1.unsqueeze(-1) - x2.unsqueeze(-2))\n x1 = self.conv4(x1) * alpha + (\n A.unsqueeze(0).unsqueeze(0) if A is not None else 0) # N,C,V,V"
+ },
+ {
+ "comment": "Code snippet defines a class TemporalConv, which is a 2D convolutional layer for temporal data. It inherits from the paddle.nn.Layer and includes an instance of nn.Conv2D and nn.BatchNorm2D layers. The MultiScale_TemporalConv class is also defined but its implementation is missing, suggesting it extends TemporalConv with multiple temporal convolution blocks for multi-scale processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":93-126",
+ "content": " # We only support 'paddle.einsum()' in dynamic graph mode, if use in infer model please implement self.\n # x1 = paddle.einsum('ncuv,nctv->nctu', x1, x3)\n x1 = einsum(x1, x3)\n return x1\nclass TemporalConv(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n dilation=1):\n super(TemporalConv, self).__init__()\n pad = (kernel_size + (kernel_size - 1) * (dilation - 1) - 1) // 2\n self.conv = nn.Conv2D(in_channels,\n out_channels,\n kernel_size=(kernel_size, 1),\n padding=(pad, 0),\n stride=(stride, 1),\n dilation=(dilation, 1))\n self.bn = nn.BatchNorm2D(out_channels)\n def forward(self, x):\n x = self.conv(x)\n x = self.bn(x)\n return x\nclass MultiScale_TemporalConv(nn.Layer):\n def __init__(self,"
+ },
+ {
+ "comment": "This code defines a MultiScale_TemporalConv layer with multiple branches of temporal convolution. The number of branches is determined by the dilations, and out channels should be multiples of the number of branches for correct operation. Each branch has its own kernel size, and there are Conv2D layers followed by BatchNorm2D for each branch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":127-154",
+ "content": " in_channels,\n out_channels,\n kernel_size=3,\n stride=1,\n dilations=[1, 2, 3, 4],\n residual=True,\n residual_kernel_size=1):\n super(MultiScale_TemporalConv, self).__init__()\n assert out_channels % (\n len(dilations) +\n 2) == 0, '# out channels should be multiples of # branches'\n # Multiple branches of temporal convolution\n self.num_branches = len(dilations) + 2\n branch_channels = out_channels // self.num_branches\n if type(kernel_size) == list:\n assert len(kernel_size) == len(dilations)\n else:\n kernel_size = [kernel_size] * len(dilations)\n # Temporal Convolution branches\n self.branches = nn.LayerList([\n nn.Sequential(\n nn.Conv2D(in_channels,\n branch_channels,\n kernel_size=1,\n padding=0),\n nn.BatchNorm2D(branch_channels),"
+ },
+ {
+ "comment": "This code defines a Conv-Temporal RGN backbone model for video analysis. It consists of multiple branches with various convolutional and pooling layers, including TemporalConv and MaxPool2D operations. The branches are appended to the model and initialized with respective settings such as kernel size, dilation rate, etc.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":155-181",
+ "content": " nn.ReLU(),\n TemporalConv(branch_channels,\n branch_channels,\n kernel_size=ks,\n stride=stride,\n dilation=dilation),\n ) for ks, dilation in zip(kernel_size, dilations)\n ])\n # Additional Max & 1x1 branch\n self.branches.append(\n nn.Sequential(\n nn.Conv2D(in_channels,\n branch_channels,\n kernel_size=1,\n padding=0), nn.BatchNorm2D(branch_channels),\n nn.ReLU(),\n nn.MaxPool2D(kernel_size=(3, 1),\n stride=(stride, 1),\n padding=(1, 0)), nn.BatchNorm2D(branch_channels)))\n self.branches.append(\n nn.Sequential(\n nn.Conv2D(in_channels,\n branch_channels,\n kernel_size=1,\n padding=0,"
+ },
+ {
+ "comment": "This code defines a class for a Conv-Temporal Residual Group Convolutional Network (CTRGCN) backbone. The class contains a constructor that sets up the architecture, an initialization function to set the weights, and a forward pass function for feeding data into the model. It performs residual connections using temporal convolutions and has batch normalization layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":182-210",
+ "content": " stride=(stride, 1)), nn.BatchNorm2D(branch_channels)))\n # Residual connection\n if not residual:\n self.residual = lambda x: 0\n elif (in_channels == out_channels) and (stride == 1):\n self.residual = lambda x: x\n else:\n self.residual = TemporalConv(in_channels,\n out_channels,\n kernel_size=residual_kernel_size,\n stride=stride)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n # initialize\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n conv_init(m)\n elif isinstance(m, nn.BatchNorm2D):\n weight_init_(m.weight, 'Normal', std=0.02, mean=1.0)\n nn.initializer.Constant(value=0.0)(m.bias)\n def forward(self, x):\n # Input dim: (N,C,T,V)\n res = self.residual(x)\n branch_outs = []\n for tempconv in self.branches:"
+ },
+ {
+ "comment": "This code defines two classes: \"unit_tcn\" and \"unit_gcn\". The \"unit_tcn\" class is a Temporal Convolutional Network unit that performs temporal convolution with batch normalization and ReLU activation. The \"unit_gcn\" class is a Graph Convolutional Network unit that takes input channels, output channels, adjacency matrix A, coefficient embedding, adaptive flag, and residual flag as parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":211-249",
+ "content": " out = tempconv(x)\n branch_outs.append(out)\n out = paddle.concat(branch_outs, axis=1)\n out += res\n return out\nclass unit_tcn(nn.Layer):\n def __init__(self, in_channels, out_channels, kernel_size=9, stride=1):\n super(unit_tcn, self).__init__()\n pad = int((kernel_size - 1) / 2)\n self.conv = nn.Conv2D(in_channels,\n out_channels,\n kernel_size=(kernel_size, 1),\n padding=(pad, 0),\n stride=(stride, 1))\n self.bn = nn.BatchNorm2D(out_channels)\n self.relu = nn.ReLU()\n conv_init(self.conv)\n bn_init(self.bn, 1)\n def forward(self, x):\n x = self.bn(self.conv(x))\n return x\nclass unit_gcn(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n A,\n coff_embedding=4,\n adaptive=True,\n residual=True):\n super(unit_gcn, self).__init__()"
+ },
+ {
+ "comment": "This code initializes a CTRGC model with specified input and output channels. It also includes optional residual connection, batch normalization, and adaptive parameterization. The number of subsets is determined by the shape of A. If adaptive is set to True, it creates a trainable parameter for the subset of weights.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":250-275",
+ "content": " inter_channels = out_channels // coff_embedding\n self.inter_c = inter_channels\n self.out_c = out_channels\n self.in_c = in_channels\n self.adaptive = adaptive\n self.num_subset = A.shape[0]\n self.convs = nn.LayerList()\n for i in range(self.num_subset):\n self.convs.append(CTRGC(in_channels, out_channels))\n if residual:\n if in_channels != out_channels:\n self.down = nn.Sequential(\n nn.Conv2D(in_channels, out_channels, 1),\n nn.BatchNorm2D(out_channels))\n else:\n self.down = lambda x: x\n else:\n self.down = lambda x: 0\n if self.adaptive:\n pa_param = paddle.ParamAttr(\n initializer=paddle.nn.initializer.Assign(A.astype(np.float32)))\n self.PA = paddle.create_parameter(shape=A.shape,\n dtype='float32',\n attr=pa_param)"
+ },
+ {
+ "comment": "This code initializes the parameters A and alpha, sets up batch normalization (bn) layers with Softmax and ReLU activation functions, initializes weights using conv_init and bn_init functions, and defines a forward pass that adapts A based on the adaptive flag.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":276-305",
+ "content": " else:\n A_tensor = paddle.to_tensor(A, dtype=\"float32\")\n self.A = paddle.create_parameter(\n shape=A_tensor.shape,\n dtype='float32',\n default_initializer=paddle.nn.initializer.Assign(A_tensor))\n self.A.stop_gradient = True\n alpha_tensor = paddle.to_tensor(np.zeros(1), dtype=\"float32\")\n self.alpha = paddle.create_parameter(\n shape=alpha_tensor.shape,\n dtype='float32',\n default_initializer=paddle.nn.initializer.Assign(alpha_tensor))\n self.bn = nn.BatchNorm2D(out_channels)\n self.soft = nn.Softmax(-2)\n self.relu = nn.ReLU()\n def init_weights(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n conv_init(m)\n elif isinstance(m, nn.BatchNorm2D):\n bn_init(m, 1)\n bn_init(self.bn, 1e-6)\n def forward(self, x):\n y = None\n if self.adaptive:\n A = self.PA\n else:\n A = self.A.cuda(x.get_device())"
+ },
+ {
+ "comment": "This code defines a TCN_GCN_unit class, which is a combination of Graph Convolutional Network (GCN) and Temporal Convolution units. The unit takes input channels, output channels, adjacency matrix A, stride, residual connection, adaptive flag, kernel size, and dilations as parameters. It initializes the GCN and TemporalConv layers, followed by a ReLU activation function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":306-334",
+ "content": " for i in range(self.num_subset):\n z = self.convs[i](x, A[i], self.alpha)\n y = z + y if y is not None else z\n y = self.bn(y)\n y += self.down(x)\n y = self.relu(y)\n return y\nclass TCN_GCN_unit(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n A,\n stride=1,\n residual=True,\n adaptive=True,\n kernel_size=5,\n dilations=[1, 2]):\n super(TCN_GCN_unit, self).__init__()\n self.gcn1 = unit_gcn(in_channels, out_channels, A, adaptive=adaptive)\n self.tcn1 = MultiScale_TemporalConv(out_channels,\n out_channels,\n kernel_size=kernel_size,\n stride=stride,\n dilations=dilations,\n residual=False)\n self.relu = nn.ReLU()"
+ },
+ {
+ "comment": "The code defines a `CTRGCN` class with a `forward` method and an `NTUDGraph` class. The `forward` method takes input `x`, applies `relu` activation and adds the residual output of a `unit_tcn` layer or simply passes through if specified conditions are met. The `NTUDGraph` initializes with a fixed number of nodes, self-links, and inward connections.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":335-362",
+ "content": " if not residual:\n self.residual = lambda x: 0\n elif (in_channels == out_channels) and (stride == 1):\n self.residual = lambda x: x\n else:\n self.residual = unit_tcn(in_channels,\n out_channels,\n kernel_size=1,\n stride=stride)\n def forward(self, x):\n y = self.relu(self.tcn1(self.gcn1(x)) + self.residual(x))\n return y\nclass NTUDGraph:\n def __init__(self, labeling_mode='spatial'):\n num_node = 25\n self_link = [(i, i) for i in range(num_node)]\n inward_ori_index = [(1, 2), (2, 21), (3, 21), (4, 3), (5, 21), (6, 5),\n (7, 6), (8, 7), (9, 21), (10, 9), (11, 10),\n (12, 11), (13, 1), (14, 13), (15, 14), (16, 15),\n (17, 1), (18, 17), (19, 18), (20, 19), (22, 23),\n (23, 8), (24, 25), (25, 12)]\n inward = [(i - 1, j - 1) for (i, j) in inward_ori_index]"
+ },
+ {
+ "comment": "Function `get_adjacency_matrix` generates adjacency matrices for the model. The function takes a parameter `labeling_mode`, which is optional. It initializes a set of variables: `inward`, `outward`, and `neighbor`. These variables store the connections between nodes in both directions. Then, it calls other helper functions to generate normalized adjacency matrices for self-links, inward edges, outward edges, and finally returns an array containing all these matrices. This is useful for inputting into a model that requires specific formatted input data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":363-396",
+ "content": " outward = [(j, i) for (i, j) in inward]\n neighbor = inward + outward\n self.num_node = num_node\n self.self_link = self_link\n self.inward = inward\n self.outward = outward\n self.neighbor = neighbor\n self.A = self.get_adjacency_matrix(labeling_mode)\n def edge2mat(self, link, num_node):\n A = np.zeros((num_node, num_node))\n for i, j in link:\n A[j, i] = 1\n return A\n def normalize_digraph(self, A):\n Dl = np.sum(A, 0)\n h, w = A.shape\n Dn = np.zeros((w, w))\n for i in range(w):\n if Dl[i] > 0:\n Dn[i, i] = Dl[i]**(-1)\n AD = np.dot(A, Dn)\n return AD\n def get_spatial_graph(self, num_node, self_link, inward, outward):\n I = self.edge2mat(self_link, num_node)\n In = self.normalize_digraph(self.edge2mat(inward, num_node))\n Out = self.normalize_digraph(self.edge2mat(outward, num_node))\n A = np.stack((I, In, Out))\n return A\n def get_adjacency_matrix(self, labeling_mode=None):"
+ },
+ {
+ "comment": "This code is part of the CTRGCN class in the PaddleVideo library, which represents a specific type of model for skeleton-based action recognition. The function within this code block is used to return an adjacency matrix (A) based on a given labeling mode. If no labeling mode is specified, it returns the adjacency matrix from the instance variables. If the labeling mode is set to 'spatial', it calls another function to generate a spatial adjacency graph. Otherwise, if an invalid labeling mode is provided, it raises a ValueError exception.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":397-425",
+ "content": " if labeling_mode is None:\n return self.A\n if labeling_mode == 'spatial':\n A = self.get_spatial_graph(self.num_node, self.self_link,\n self.inward, self.outward)\n else:\n raise ValueError()\n return A\n@BACKBONES.register()\nclass CTRGCN(nn.Layer):\n \"\"\"\n CTR-GCN model from:\n `\"Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition\" `_\n Args:\n num_point: int, numbers of sketeton point.\n num_person: int, numbers of person.\n base_channel: int, model's hidden dim.\n graph: str, sketeton adjacency matrix name.\n graph_args: dict, sketeton adjacency graph class args.\n in_channels: int, channels of vertex coordinate. 2 for (x,y), 3 for (x,y,z). Default 3.\n adaptive: bool, if adjacency matrix can adaptive.\n \"\"\"\n def __init__(self,\n num_point=25,\n num_person=2,\n base_channel=64,"
+ },
+ {
+ "comment": "This code defines the CTRGCN class, which initializes its graph and layers based on input parameters. It includes a batch normalization layer (data_bn) and three TCN_GCN_unit layers (l1, l2, l3). The graph is determined by the 'graph' parameter, with NTUDGraph used if 'ntu_rgb_d'. If another graph is provided, it raises a ValueError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":426-454",
+ "content": " graph='ntu_rgb_d',\n graph_args=dict(),\n in_channels=3,\n adaptive=True):\n super(CTRGCN, self).__init__()\n if graph == 'ntu_rgb_d':\n self.graph = NTUDGraph(**graph_args)\n else:\n raise ValueError()\n A = self.graph.A # 3,25,25\n self.num_point = num_point\n self.data_bn = nn.BatchNorm1D(num_person * in_channels * num_point)\n self.base_channel = base_channel\n self.l1 = TCN_GCN_unit(in_channels,\n self.base_channel,\n A,\n residual=False,\n adaptive=adaptive)\n self.l2 = TCN_GCN_unit(self.base_channel,\n self.base_channel,\n A,\n adaptive=adaptive)\n self.l3 = TCN_GCN_unit(self.base_channel,\n self.base_channel,\n A,"
+ },
+ {
+ "comment": "The code initializes six TCN_GCN_unit layers, each with different configurations, for a CTRGCN model. The first layer (l4) has the base channel as input and output. Following layers (l5 to l8) increase the number of channels or apply strides. This represents a deep TCN-GCN architecture with progressively increasing depth and downsampling.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":455-476",
+ "content": " adaptive=adaptive)\n self.l4 = TCN_GCN_unit(self.base_channel,\n self.base_channel,\n A,\n adaptive=adaptive)\n self.l5 = TCN_GCN_unit(self.base_channel,\n self.base_channel * 2,\n A,\n stride=2,\n adaptive=adaptive)\n self.l6 = TCN_GCN_unit(self.base_channel * 2,\n self.base_channel * 2,\n A,\n adaptive=adaptive)\n self.l7 = TCN_GCN_unit(self.base_channel * 2,\n self.base_channel * 2,\n A,\n adaptive=adaptive)\n self.l8 = TCN_GCN_unit(self.base_channel * 2,\n self.base_channel * 4,\n A,\n stride=2,"
+ },
+ {
+ "comment": "This code defines a neural network model with multiple layers. It uses Paddle's TCN_GCN_unit in the last two layers. The init_weights function initializes batch normalization for the data_bn layer, and the forward function processes input through multiple layers before returning the final output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":477-510",
+ "content": " adaptive=adaptive)\n self.l9 = TCN_GCN_unit(self.base_channel * 4,\n self.base_channel * 4,\n A,\n adaptive=adaptive)\n self.l10 = TCN_GCN_unit(self.base_channel * 4,\n self.base_channel * 4,\n A,\n adaptive=adaptive)\n def init_weights(self):\n bn_init(self.data_bn, 1)\n def forward(self, x):\n N, C, T, V, M = x.shape\n x = paddle.transpose(x, perm=[0, 4, 3, 1, 2])\n x = paddle.reshape(x, (N, M * V * C, T))\n x = self.data_bn(x)\n x = paddle.reshape(x, (N, M, V, C, T))\n x = paddle.transpose(x, perm=(0, 1, 3, 4, 2))\n x = paddle.reshape(x, (N * M, C, T, V))\n x = self.l1(x)\n x = self.l2(x)\n x = self.l3(x)\n x = self.l4(x)\n x = self.l5(x)\n x = self.l6(x)\n x = self.l7(x)\n x = self.l8(x)\n x = self.l9(x)"
+ },
+ {
+ "comment": "This code represents the final step of a neural network function. It applies layer 10 (l10) to input x, and returns both the updated x and the original N, M values. This function seems to be part of a larger model, as it references previous layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ctrgcn.py\":511-513",
+ "content": " x = self.l10(x)\n return x, N, M"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/18afc32c-df62-4b9b-a373-19c278db53f3.json b/docs/doc/18afc32c-df62-4b9b-a373-19c278db53f3.json
new file mode 100644
index 000000000..b4322997e
--- /dev/null
+++ b/docs/doc/18afc32c-df62-4b9b-a373-19c278db53f3.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This script sets up a model environment, downloads weights if needed, initializes the Infer model, and runs inference on input videos while saving results and features.",
+ "details": [
+ {
+ "comment": "This code is a Python script with licensing information and import statements. It imports necessary libraries like numpy, paddle, and others for data processing, model training, and evaluation. The code also sets up the logging format, and checks for CUDA availability and PaddlePaddle version.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/tsn_extractor.py\":0-36",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport os\nimport sys\nimport time\nimport logging\nimport argparse\nimport ast\nimport numpy as np\nimport paddle\nimport paddle.static as static\ntry:\n import cPickle as pickle\nexcept:\n import pickle\nfrom utils.config_utils import *\nimport models\nfrom reader import get_reader\nfrom metrics import get_metrics\nfrom utils.utility import check_cuda\nfrom utils.utility import check_version\nlogging.root.handlers = []\nFORMAT = '[%(levelname)s: %(filename)s: %(lineno)4d]: %(message)s'"
+ },
+ {
+ "comment": "This code defines a function `parse_args()` to parse command-line arguments for training a model. The arguments include model name, config file path, whether to use GPU, weight path, and batch size. It uses argparse module for easy argument handling. By default, it sets the model name to 'AttentionCluster', config file path to 'configs/attention_cluster.txt', uses GPU if not specified otherwise, automatically downloads weights from Paddle if no specific path is provided, and sets batch size to 1.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/tsn_extractor.py\":37-65",
+ "content": "logging.basicConfig(level=logging.DEBUG, format=FORMAT, stream=sys.stdout)\nlogger = logging.getLogger(__name__)\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--model_name',\n type=str,\n default='AttentionCluster',\n help='name of model to train.')\n parser.add_argument('--config',\n type=str,\n default='configs/attention_cluster.txt',\n help='path to config file of model')\n parser.add_argument('--use_gpu',\n type=ast.literal_eval,\n default=True,\n help='default use gpu.')\n parser.add_argument(\n '--weights',\n type=str,\n default=None,\n help=\n 'weight path, None to automatically download weights provided by Paddle.'\n )\n parser.add_argument('--batch_size',\n type=int,\n default=1,\n help='sample number in a batch for inference.')"
+ },
+ {
+ "comment": "The code defines command line arguments for the TsnExtractor. It sets default values and provides help messages for each argument. The function then parses these arguments to create an 'args' object, which can be used throughout the program. Additionally, the 'infer' function is defined but not implemented.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/tsn_extractor.py\":66-92",
+ "content": " parser.add_argument('--filelist',\n type=str,\n default='./data/TsnExtractor.list',\n help='path to inferenece data file lists file.')\n parser.add_argument('--log_interval',\n type=int,\n default=1,\n help='mini-batch interval to log.')\n parser.add_argument('--infer_topk',\n type=int,\n default=20,\n help='topk predictions to restore.')\n parser.add_argument('--save_dir',\n type=str,\n default=os.path.join('data', 'tsn_features'),\n help='directory to store tsn feature results')\n parser.add_argument('--video_path',\n type=str,\n default=None,\n help='directory to store results')\n args = parser.parse_args()\n return args\ndef infer(args):\n # parse config\n config = parse_config(args.config)"
+ },
+ {
+ "comment": "The code initializes the Infer model with provided configurations and merges them to create the infer_config. It then builds the input, model, and gets feeds and outputs for inference. The place and executor are set based on whether or not GPU is used. The filelist and video path are checked for existence before initializing the infer reader with the model name, mode (infer), and configurations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/tsn_extractor.py\":93-117",
+ "content": " infer_config = merge_configs(config, 'infer', vars(args))\n print_configs(infer_config, \"Infer\")\n infer_model = models.get_model(args.model_name,\n infer_config,\n mode='infer',\n is_videotag=True)\n infer_model.build_input(use_dataloader=False)\n infer_model.build_model()\n infer_feeds = infer_model.feeds()\n infer_outputs = infer_model.outputs()\n place = paddle.CUDAPlace(0) if args.use_gpu else paddle.CPUPlace()\n exe = static.Executor(place)\n exe.run(static.default_startup_program())\n filelist = args.filelist or infer_config.INFER.filelist\n filepath = args.video_path or infer_config.INFER.get('filepath', '')\n if filepath != '':\n assert os.path.exists(filepath), \"{} not exist.\".format(filepath)\n else:\n assert os.path.exists(filelist), \"{} not exist.\".format(filelist)\n # get infer reader\n infer_reader = get_reader(args.model_name.upper(), 'infer', infer_config)"
+ },
+ {
+ "comment": "This code snippet checks if the weights (model parameters) are provided as an argument. If not, it downloads them from Paddle's servers. Then, it loads the weights into the model and creates a DataFeeder for feeding data during inference. It also initializes metrics to measure inference performance. The code then iterates over each input video, running inference with the loaded model, and saving the results for each frame.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/tsn_extractor.py\":119-143",
+ "content": " if args.weights:\n assert os.path.exists(\n args.weights), \"Given weight dir {} not exist.\".format(args.weights)\n # if no weight files specified, download weights from paddle\n weights = args.weights or infer_model.get_weights()\n infer_model.load_test_weights(exe, weights, static.default_main_program())\n infer_feeder = paddle.fluid.DataFeeder(place=place, feed_list=infer_feeds)\n fetch_list = infer_model.fetches()\n infer_metrics = get_metrics(args.model_name.upper(), 'infer', infer_config)\n infer_metrics.reset()\n if not os.path.isdir(args.save_dir):\n os.makedirs(args.save_dir)\n for infer_iter, data in enumerate(infer_reader()):\n data_feed_in = [items[:-1] for items in data]\n video_id = [items[-1] for items in data]\n bs = len(video_id)\n feature_outs = exe.run(fetch_list=fetch_list,\n feed=infer_feeder.feed(data_feed_in))\n for i in range(bs):\n filename = video_id[i].split('/')[-1][:-4]"
+ },
+ {
+ "comment": "Saves extracted features from the PaddleVideo/applications/VideoTag/tsn_extractor.py module using numpy's save function, then logs the end of feature extraction and calls infer function with argument args.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/tsn_extractor.py\":144-157",
+ "content": " np.save(os.path.join(args.save_dir, filename + '.npy'),\n feature_outs[0][i]) #shape: seg_num*feature_dim\n logger.info(\"Feature extraction End~\")\nif __name__ == \"__main__\":\n args = parse_args()\n # check whether the installed paddle is compiled with GPU\n check_cuda(args.use_gpu)\n check_version()\n logger.info(args)\n infer(args)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/195955fc-23f4-4ec7-8751-022d035c39d1.json b/docs/doc/195955fc-23f4-4ec7-8751-022d035c39d1.json
new file mode 100644
index 000000000..1c2562b8a
--- /dev/null
+++ b/docs/doc/195955fc-23f4-4ec7-8751-022d035c39d1.json
@@ -0,0 +1,90 @@
+{
+ "summary": "The code introduces a \"Scale\" class for image scaling and a MultiScaleCrop pipeline in PaddleVideo. It supports random or multi-crop based on test mode, maintaining aspect ratio while resizing/cropping images. A slower pathway is created by selecting specific frames from the fast_pathway array, rearranging dimensions, and then combined with the original for a list of frames before adding to 'results' dictionary.",
+ "details": [
+ {
+ "comment": "This code registers a new class \"Scale\" for image scaling in PaddleVideo's VideoQualityAssessment module. The Scale class takes a short_size parameter and scales the images accordingly. It is registered as part of the PIPELINES in the application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":0-34",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport random\nimport numpy as np\nimport math\nfrom PIL import Image\nfrom ..registry import PIPELINES\nfrom collections.abc import Sequence\n@PIPELINES.register()\nclass Scale(object):\n \"\"\"\n Scale images.\n Args:\n short_size(float | int): Short size of an image will be scaled to the short_size.\n \"\"\"\n def __init__(self, short_size):\n self.short_size = short_size\n def __call__(self, results):"
+ },
+ {
+ "comment": "The code defines a function that resizes PIL.Image objects in a list according to their aspect ratios and the short_size provided. If an image's width is less than or equal to its height, it is appended to resized_imgs without any modification. Otherwise, if the width is greater than the height, the image is scaled to fit within a square with the given short_size, maintaining aspect ratio using bilinear interpolation. If the height is greater than the width, the image is also scaled to fit within a square with the given short_size, again maintaining aspect ratio using bilinear interpolation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":35-60",
+ "content": " \"\"\"\n Performs resize operations.\n Args:\n imgs (Sequence[PIL.Image]): List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n resized_imgs: List where each item is a PIL.Image after scaling.\n \"\"\"\n imgs = results['imgs']\n resized_imgs = []\n for i in range(len(imgs)):\n img = imgs[i]\n w, h = img.size\n if (w <= h and w == self.short_size) or (h <= w\n and h == self.short_size):\n resized_imgs.append(img)\n continue\n if w < h:\n ow = self.short_size\n oh = int(self.short_size * 4.0 / 3.0)\n resized_imgs.append(img.resize((ow, oh), Image.BILINEAR))\n else:\n oh = self.short_size\n ow = int(self.short_size * 4.0 / 3.0)\n resized_imgs.append(img.resize((ow, oh), Image.BILINEAR))"
+ },
+ {
+ "comment": "The code registers a custom pipeline for random cropping of images in PaddleVideo. It takes a target size as an argument and initializes the class with that target size. The __call__ method is used to perform random crop operations on a list of images. It first retrieves the original image sizes, ensures they are larger than the target size, then randomly selects x1 and y1 coordinates for the crop region, and appends the cropped image to a new list which is returned at the end.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":61-94",
+ "content": " results['imgs'] = resized_imgs\n return results\n@PIPELINES.register()\nclass RandomCrop(object):\n \"\"\"\n Random crop images.\n Args:\n target_size(int): Random crop a square with the target_size from an image.\n \"\"\"\n def __init__(self, target_size):\n self.target_size = target_size\n def __call__(self, results):\n \"\"\"\n Performs random crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n crop_imgs: List where each item is a PIL.Image after random crop.\n \"\"\"\n imgs = results['imgs']\n w, h = imgs[0].size\n th, tw = self.target_size, self.target_size\n assert (w >= self.target_size) and (h >= self.target_size), \\\n \"image width({}) and height({}) should be larger than crop size {}\".format(\n w, h, self.target_size)\n crop_images = []\n x1 = random.randint(0, w - tw)\n y1 = random.randint(0, h - th)"
+ },
+ {
+ "comment": "This code performs center cropping of images to a specified target size. It iterates through the list of images, checks if they are already at the target size, and appends them to the crop_images list. If the image is not at the target size, it crops the image to the center square of the original image and adds it to the crop_images list. The final results dictionary contains the list of cropped images. The class CenterCrop initializes with a target_size parameter and defines a __call__ method for applying the center crop operation on input images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":96-129",
+ "content": " for img in imgs:\n if w == tw and h == th:\n crop_images.append(img)\n else:\n crop_images.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n results['imgs'] = crop_images\n return results\n@PIPELINES.register()\nclass CenterCrop(object):\n \"\"\"\n Center crop images.\n Args:\n target_size(int): Center crop a square with the target_size from an image.\n \"\"\"\n def __init__(self, target_size):\n self.target_size = target_size\n def __call__(self, results):\n \"\"\"\n Performs Center crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n ccrop_imgs: List where each item is a PIL.Image after Center crop.\n \"\"\"\n imgs = results['imgs']\n ccrop_imgs = []\n for img in imgs:\n w, h = img.size\n th, tw = self.target_size, self.target_size\n assert (w >= self.target_size) and (h >= self.target_size), \\"
+ },
+ {
+ "comment": "MultiScaleCrop applies image resizing and cropping to an input image. The target size, scales, max_distort, fix_crop, and more_fix_crop parameters are used for image manipulation. Images are cropped into smaller ones with varying sizes based on the defined scales.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":130-159",
+ "content": " \"image width({}) and height({}) should be larger than crop size {}\".format(\n w, h, self.target_size)\n x1 = int(round((w - tw) / 2.))\n y1 = int(round((h - th) / 2.))\n ccrop_imgs.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n results['imgs'] = ccrop_imgs\n return results\n@PIPELINES.register()\nclass MultiScaleCrop(object):\n def __init__(\n self,\n target_size, #NOTE: named target size now, but still pass short size in it!\n scales=None,\n max_distort=1,\n fix_crop=True,\n more_fix_crop=True):\n self.target_size = target_size\n self.scales = scales if scales else [1, .875, .75, .66]\n self.max_distort = max_distort\n self.fix_crop = fix_crop\n self.more_fix_crop = more_fix_crop\n def __call__(self, results):\n \"\"\"\n Performs MultiScaleCrop operations.\n Args:\n imgs: List where wach item is a PIL.Image.\n XXX:"
+ },
+ {
+ "comment": "This code defines a function to sample random crop sizes for image augmentation. It first calculates the possible crop sizes based on input size and scales, then filters pairs that have a difference within max_distort. Finally, it randomly chooses one of the filtered pairs for cropping and optionally adds a random offset if fix_crop is False.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":160-191",
+ "content": " results:\n \"\"\"\n imgs = results['imgs']\n input_size = [self.target_size, self.target_size]\n im_size = imgs[0].size\n # get random crop offset\n def _sample_crop_size(im_size):\n image_w, image_h = im_size[0], im_size[1]\n base_size = min(image_w, image_h)\n crop_sizes = [int(base_size * x) for x in self.scales]\n crop_h = [\n input_size[1] if abs(x - input_size[1]) < 3 else x\n for x in crop_sizes\n ]\n crop_w = [\n input_size[0] if abs(x - input_size[0]) < 3 else x\n for x in crop_sizes\n ]\n pairs = []\n for i, h in enumerate(crop_h):\n for j, w in enumerate(crop_w):\n if abs(i - j) <= self.max_distort:\n pairs.append((w, h))\n crop_pair = random.choice(pairs)\n if not self.fix_crop:\n w_offset = random.randint(0, image_w - crop_pair[0])"
+ },
+ {
+ "comment": "This code generates a list of crop positions for an image. If the image height is greater than the second value in the crop pair, it randomly selects a horizontal offset. Otherwise, it calculates step sizes for width and height, and creates a list of crop positions using these steps. Additional crop positions are added if self.more_fix_crop is True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":192-214",
+ "content": " h_offset = random.randint(0, image_h - crop_pair[1])\n else:\n w_step = (image_w - crop_pair[0]) / 4\n h_step = (image_h - crop_pair[1]) / 4\n ret = list()\n ret.append((0, 0)) # upper left\n if w_step != 0:\n ret.append((4 * w_step, 0)) # upper right\n if h_step != 0:\n ret.append((0, 4 * h_step)) # lower left\n if h_step != 0 and w_step != 0:\n ret.append((4 * w_step, 4 * h_step)) # lower right\n if h_step != 0 or w_step != 0:\n ret.append((2 * w_step, 2 * h_step)) # center\n if self.more_fix_crop:\n ret.append((0, 2 * h_step)) # center left\n ret.append((4 * w_step, 2 * h_step)) # center right\n ret.append((2 * w_step, 4 * h_step)) # lower center\n ret.append((2 * w_step, 0 * h_step)) # upper center\n ret.append((1 * w_step, 1 * h_step)) # upper left quarter"
+ },
+ {
+ "comment": "The code randomly samples crop sizes from a set of predefined ratios, crops the input images accordingly, resizes them to the desired input size, and adds the flipped or cropped images to the results dictionary. It also includes an optional RandomFlip pipeline that randomly flips the image with a given probability.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":215-246",
+ "content": " ret.append((3 * w_step, 1 * h_step)) # upper right quarter\n ret.append((1 * w_step, 3 * h_step)) # lower left quarter\n ret.append((3 * w_step, 3 * h_step)) # lower righ quarter\n w_offset, h_offset = random.choice(ret)\n return crop_pair[0], crop_pair[1], w_offset, h_offset\n crop_w, crop_h, offset_w, offset_h = _sample_crop_size(im_size)\n crop_img_group = [\n img.crop((offset_w, offset_h, offset_w + crop_w, offset_h + crop_h))\n for img in imgs\n ]\n ret_img_group = [\n img.resize((input_size[0], input_size[1]), Image.BILINEAR)\n for img in crop_img_group\n ]\n results['imgs'] = ret_img_group\n return results\n@PIPELINES.register()\nclass RandomFlip(object):\n \"\"\"\n Random Flip images.\n Args:\n p(float): Random flip images with the probability p.\n \"\"\"\n def __init__(self, p=0.5):\n self.p = p\n def __call__(self, results):"
+ },
+ {
+ "comment": "This code defines two classes: \"RandomFlip\" and \"Image2Array\". RandomFlip performs random flips on a list of PIL images, while Image2Array converts a PIL image to a numpy array with optional transpose. Both are registered as pipelines using @PIPELINES.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":247-280",
+ "content": " \"\"\"\n Performs random flip operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n flip_imgs: List where each item is a PIL.Image after random flip.\n \"\"\"\n imgs = results['imgs']\n v = random.random()\n if v < self.p:\n results['imgs'] = [\n img.transpose(Image.FLIP_LEFT_RIGHT) for img in imgs\n ]\n else:\n results['imgs'] = imgs\n return results\n@PIPELINES.register()\nclass Image2Array(object):\n \"\"\"\n transfer PIL.Image to Numpy array and transpose dimensions from 'dhwc' to 'dchw'.\n Args:\n transpose: whether to transpose or not, default True, False for slowfast.\n \"\"\"\n def __init__(self, transpose=True):\n self.transpose = transpose\n def __call__(self, results):\n \"\"\"\n Performs Image to NumpyArray operations.\n Args:\n imgs: List where each item is a PIL.Image."
+ },
+ {
+ "comment": "This function converts a list of PIL images to a numpy array, optionally transposes it if needed, and stores the result in the 'imgs' key of the results dictionary. Additionally, the Normalization class initializes with mean and std values for normalization and reshapes them to fit the tensor shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":281-309",
+ "content": " For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n np_imgs: Numpy array.\n \"\"\"\n imgs = results['imgs']\n np_imgs = (np.stack(imgs)).astype('float32')\n if self.transpose:\n np_imgs = np_imgs.transpose(0, 3, 1, 2) #nchw\n results['imgs'] = np_imgs\n return results\n@PIPELINES.register()\nclass Normalization(object):\n \"\"\"\n Normalization.\n Args:\n mean(Sequence[float]): mean values of different channels.\n std(Sequence[float]): std values of different channels.\n tensor_shape(list): size of mean, default [3,1,1]. For slowfast, [1,1,1,3]\n \"\"\"\n def __init__(self, mean, std, tensor_shape=[3, 1, 1]):\n if not isinstance(mean, Sequence):\n raise TypeError(\n 'Mean must be list, tuple or np.ndarray, but got {type(mean)}')\n if not isinstance(std, Sequence):\n raise TypeError(\n 'Std must be list, tuple or np.ndarray, but got {type(std)}')\n self.mean = np.array(mean).reshape(tensor_shape).astype(np.float32)"
+ },
+ {
+ "comment": "The code is a part of the PaddleVideo library's VideoQualityAssessment module. It performs normalization operations on image arrays and registers a JitterScale class for image scaling with random short size selection between min_size and max_size, also including cycling factors for default minimum size functionality.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":310-343",
+ "content": " self.std = np.array(std).reshape(tensor_shape).astype(np.float32)\n def __call__(self, results):\n \"\"\"\n Performs normalization operations.\n Args:\n imgs: Numpy array.\n return:\n np_imgs: Numpy array after normalization.\n \"\"\"\n imgs = results['imgs']\n norm_imgs = imgs / 255.\n norm_imgs -= self.mean\n norm_imgs /= self.std\n results['imgs'] = norm_imgs\n return results\n@PIPELINES.register()\nclass JitterScale(object):\n \"\"\"\n Scale image, while the target short size is randomly select between min_size and max_size.\n Args:\n min_size: Lower bound for random sampler.\n max_size: Higher bound for random sampler.\n \"\"\"\n def __init__(self,\n min_size,\n max_size,\n short_cycle_factors=[0.5, 0.7071],\n default_min_size=256):\n self.default_min_size = default_min_size\n self.orig_min_size = self.min_size = min_size\n self.max_size = max_size"
+ },
+ {
+ "comment": "This code performs jitter resize operations and applies random scaling. It takes a sequence of PIL.Image, scales each item based on min_size, max_size, and short_cycle_factors. If the number of images is less than 1, it throws an error. The size is determined by randomly selecting values between min_size and max_size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":344-369",
+ "content": " self.short_cycle_factors = short_cycle_factors\n def __call__(self, results):\n \"\"\"\n Performs jitter resize operations.\n Args:\n imgs (Sequence[PIL.Image]): List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n resized_imgs: List where each item is a PIL.Image after scaling.\n \"\"\"\n short_cycle_idx = results.get('short_cycle_idx')\n if short_cycle_idx in [0, 1]:\n self.min_size = int(\n round(self.short_cycle_factors[short_cycle_idx] *\n self.default_min_size))\n else:\n self.min_size = self.orig_min_size\n imgs = results['imgs']\n size = int(round(np.random.uniform(self.min_size, self.max_size)))\n assert (len(imgs) >= 1) , \\\n \"len(imgs):{} should be larger than 1\".format(len(imgs))\n width, height = imgs[0].size\n if (width <= height and width == size) or (height <= width\n and height == size):"
+ },
+ {
+ "comment": "This code resizes the input images to a specified size while maintaining aspect ratio, and then applies random crop for multi-clip testing in the MultiCrop class. The target_size parameter determines the output image's dimensions after resizing and cropping.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":370-402",
+ "content": " return results\n new_width = size\n new_height = size\n if width < height:\n new_height = int(math.floor((float(height) / width) * size))\n else:\n new_width = int(math.floor((float(width) / height) * size))\n frames_resize = []\n for j in range(len(imgs)):\n img = imgs[j]\n scale_img = img.resize((new_width, new_height), Image.BILINEAR)\n frames_resize.append(scale_img)\n results['imgs'] = frames_resize\n return results\n@PIPELINES.register()\nclass MultiCrop(object):\n \"\"\"\n Random crop image.\n This operation can perform multi-crop during multi-clip test, as in slowfast model.\n Args:\n target_size(int): Random crop a square with the target_size from an image.\n \"\"\"\n def __init__(self,\n target_size,\n default_crop_size=224,\n short_cycle_factors=[0.5, 0.7071],\n test_mode=False):\n self.orig_target_size = self.target_size = target_size"
+ },
+ {
+ "comment": "The function performs random crop operations on images. It takes a list of PIL Images as input and returns the cropped images. The code checks if the current short cycle index is 0 or 1, in which case it adjusts the target size based on the short_cycle_factors variable. If the image size matches the target size, it skips the crop operation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":403-429",
+ "content": " self.short_cycle_factors = short_cycle_factors\n self.default_crop_size = default_crop_size\n self.test_mode = test_mode\n def __call__(self, results):\n \"\"\"\n Performs random crop operations.\n Args:\n imgs: List where each item is a PIL.Image.\n For example, [PIL.Image0, PIL.Image1, PIL.Image2, ...]\n return:\n crop_imgs: List where each item is a PIL.Image after random crop.\n \"\"\"\n imgs = results['imgs']\n spatial_sample_index = results['spatial_sample_index']\n spatial_num_clips = results['spatial_num_clips']\n short_cycle_idx = results.get('short_cycle_idx')\n if short_cycle_idx in [0, 1]:\n self.target_size = int(\n round(self.short_cycle_factors[short_cycle_idx] *\n self.default_crop_size))\n else:\n self.target_size = self.orig_target_size # use saved value before call\n w, h = imgs[0].size\n if w == self.target_size and h == self.target_size:"
+ },
+ {
+ "comment": "This function performs image cropping with or without random cropping. If not in test mode, it randomly selects x and y offsets within the image boundaries to crop an area of size self.target_size. In test mode, it performs multi-crop by dividing the image into equal parts based on spatial_num_clips, ensuring each part has a minimum size of self.target_size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":430-451",
+ "content": " return results\n assert (w >= self.target_size) and (h >= self.target_size), \\\n \"image width({}) and height({}) should be larger than crop size({},{})\".format(w, h, self.target_size, self.target_size)\n frames_crop = []\n if not self.test_mode:\n x_offset = random.randint(0, w - self.target_size)\n y_offset = random.randint(0, h - self.target_size)\n else: #multi-crop\n x_gap = int(\n math.ceil((w - self.target_size) / (spatial_num_clips - 1)))\n y_gap = int(\n math.ceil((h - self.target_size) / (spatial_num_clips - 1)))\n if h > w:\n x_offset = int(math.ceil((w - self.target_size) / 2))\n if spatial_sample_index == 0:\n y_offset = 0\n elif spatial_sample_index == spatial_num_clips - 1:\n y_offset = h - self.target_size\n else:\n y_offset = y_gap * spatial_sample_index\n else:"
+ },
+ {
+ "comment": "The code takes a list of images and crops them based on specified offset values to create new images with the desired target size. It then appends these cropped images to a list, stores them in 'frames_crop', and returns a dictionary containing 'imgs'. The function PackOutput is used for getting the slow pathway from the fast pathway based on the alpha factor in the SlowFast model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":452-483",
+ "content": " y_offset = int(math.ceil((h - self.target_size) / 2))\n if spatial_sample_index == 0:\n x_offset = 0\n elif spatial_sample_index == spatial_num_clips - 1:\n x_offset = w - self.target_size\n else:\n x_offset = x_gap * spatial_sample_index\n for img in imgs:\n nimg = img.crop((x_offset, y_offset, x_offset + self.target_size,\n y_offset + self.target_size))\n frames_crop.append(nimg)\n results['imgs'] = frames_crop\n return results\n@PIPELINES.register()\nclass PackOutput(object):\n \"\"\"\n In slowfast model, we want to get slow pathway from fast pathway based on\n alpha factor.\n Args:\n alpha(int): temporal length of fast/slow\n \"\"\"\n def __init__(self, alpha):\n self.alpha = alpha\n def __call__(self, results):\n fast_pathway = results['imgs']\n # sample num points between start and end\n slow_idx_start = 0"
+ },
+ {
+ "comment": "This code is creating a slower pathway by selecting specific frames from the fast_pathway array and rearranging the dimensions. The slower pathway is then combined with the original fast_pathway to create a list of frames, which is added to the 'results' dictionary before returning it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/augmentations.py\":484-497",
+ "content": " slow_idx_end = fast_pathway.shape[0] - 1\n slow_idx_num = fast_pathway.shape[0] // self.alpha\n slow_idxs_select = np.linspace(slow_idx_start, slow_idx_end,\n slow_idx_num).astype(\"int64\")\n slow_pathway = fast_pathway[slow_idxs_select]\n # T H W C -> C T H W.\n slow_pathway = slow_pathway.transpose(3, 0, 1, 2)\n fast_pathway = fast_pathway.transpose(3, 0, 1, 2)\n # slow + fast\n frames_list = [slow_pathway, fast_pathway]\n results['imgs'] = frames_list\n return results"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1a1eeeb7-7406-42a4-9b1d-489aa086755c.json b/docs/doc/1a1eeeb7-7406-42a4-9b1d-489aa086755c.json
new file mode 100644
index 000000000..e25ae44e6
--- /dev/null
+++ b/docs/doc/1a1eeeb7-7406-42a4-9b1d-489aa086755c.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines an LSTM Attention Model class with parameters and a forward method for computation, applying LSTM layers in both directions and performing dynamic LSTM on input tensor. It uses dropout, FC layer, sequence_softmax, scaling, and sum pooling to obtain the final output.",
+ "details": [
+ {
+ "comment": "This code is for a class called LSTMAttentionModel, which represents an LSTM Attention Model. It has three parameters: bias_attr, embedding_size (default 512), lstm_size (default 1024), and drop_rate (default 0.5). The class has an __init__ method to initialize these parameters and a forward method for performing the model's computation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/lstm_attention.py\":0-30",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.static as static\nclass LSTMAttentionModel(object):\n \"\"\"LSTM Attention Model\"\"\"\n def __init__(self,\n bias_attr,\n embedding_size=512,\n lstm_size=1024,\n drop_rate=0.5):\n self.lstm_size = lstm_size\n self.embedding_size = embedding_size\n self.drop_rate = drop_rate\n def forward(self, input, is_training):"
+ },
+ {
+ "comment": "This code initializes an LSTM layer for video tagging. It applies two fully connected layers (fc) to the input, one for forward and one for backward direction. The forward LSTM layer is created using dynamic_lstm function with size 4 times the lstm_size attribute and no reverse operation. The backward LSTM layer is also created similarly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/lstm_attention.py\":31-57",
+ "content": " input_fc = static.nn.fc(\n x=input,\n size=self.embedding_size,\n activation='tanh',\n bias_attr=paddle.ParamAttr(\n regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)),\n name='rgb_fc')\n lstm_forward_fc = static.nn.fc(\n x=input_fc,\n size=self.lstm_size * 4,\n activation=None,\n bias_attr=False, # video_tag\n name='rgb_fc_forward')\n lstm_forward, _ = paddle.fluid.layers.dynamic_lstm(input=lstm_forward_fc,\n size=self.lstm_size * 4,\n is_reverse=False,\n name='rgb_lstm_forward')\n lsmt_backward_fc = static.nn.fc(\n x=input_fc,\n size=self.lstm_size * 4,\n activation=None,\n bias_attr=False, #video_tag\n name='rgb_fc_backward')"
+ },
+ {
+ "comment": "This code performs dynamic LSTM on input tensor with forward and backward directions, concatenates the results, applies dropout, then feeds the result into an FC layer for weight assignment using sequence_softmax. The final output is obtained by scaling the previous result with the weights and applying a sum pooling.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/lstm_attention.py\":59-82",
+ "content": " lstm_backward, _ = paddle.fluid.layers.dynamic_lstm(input=lsmt_backward_fc,\n size=self.lstm_size * 4,\n is_reverse=True,\n name='rgb_lstm_backward')\n lstm_concat = paddle.concat(x=[lstm_forward, lstm_backward],\n axis=1)\n lstm_dropout = paddle.nn.functional.dropout2d(x=lstm_concat,\n p=self.drop_rate,\n training=is_training)\n lstm_weight = static.nn.fc(\n x=lstm_dropout,\n size=1,\n activation='sequence_softmax',\n bias_attr=False, #video_tag\n name='rgb_weight')\n scaled = paddle.multiply(x=lstm_dropout,\n y=lstm_weight)\n lstm_pool = paddle.static.nn.sequence_pool(input=scaled, pool_type='sum')\n return lstm_pool"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1a454a9c-ff43-4008-9b16-77e5a4df958d.json b/docs/doc/1a454a9c-ff43-4008-9b16-77e5a4df958d.json
new file mode 100644
index 000000000..e7a8b3a65
--- /dev/null
+++ b/docs/doc/1a454a9c-ff43-4008-9b16-77e5a4df958d.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The code defines a PaddleVideo classification head base class and function for loss/accuracy calculation, supporting binary, multi-class, and specific MRI scenarios with label smoothing. It also calculates top5 accuracy, hard/soft labels, and performs all-reduce operation in distributed training.",
+ "details": [
+ {
+ "comment": "Base class for head part, all subclass should overwrite init_weights method for initializing weights.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":0-33",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..builder import build_loss\nfrom paddlevideo.utils import get_logger, get_dist_info\nlogger = get_logger(\"paddlevideo\")\nclass BaseHead(nn.Layer):\n \"\"\"Base class for head part.\n All head should subclass it.\n All subclass should overwrite:\n - Methods: ```init_weights```, initializing weights."
+ },
+ {
+ "comment": "This code is defining a base class for a classification head in PaddleVideo. It has an `__init__` method that sets the number of classes, input channels, and loss configuration. It also builds the loss function using the provided configuration. The `forward` method must be implemented by any subclasses, as it defines how the head will run during model inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":34-64",
+ "content": " - Methods: ```forward```, forward function.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channels in input feature.\n loss_cfg (dict): Config for building loss. Default: dict(type='CrossEntropyLoss').\n ls_eps (float): label smoothing epsilon. Default: 0. .\n \"\"\"\n def __init__(\n self,\n num_classes=None,\n in_channels=None,\n loss_cfg=dict(\n name=\"CrossEntropyLoss\"\n ), #TODO(shipping): only pass a name or standard build cfg format.\n #multi_class=False, NOTE(shipping): not supported now.\n ls_eps=0.):\n super().__init__()\n self.num_classes = num_classes\n self.in_channels = in_channels\n self.loss_func = build_loss(loss_cfg)\n #self.multi_class = multi_class NOTE(shipping): not supported now\n self.ls_eps = ls_eps\n @abstractmethod\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n \"\"\"\n raise NotImplemented"
+ },
+ {
+ "comment": "This function calculates the loss based on model output (scores) and target output (labels). It returns a dictionary containing 'loss', 'top1_acc', and 'top5_acc'. If labels are single, they are expanded. Label smoothing is applied if ls_eps is non-zero and not in valid mode. The loss function is used if label smoothing is not applicable. Top-1 and top-5 accuracy are also calculated if top-5 is set to True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":66-90",
+ "content": " def loss(self, scores, labels, valid_mode=False, if_top5=True, **kwargs):\n \"\"\"Calculate the loss accroding to the model output ```scores```,\n and the target ```labels```.\n Args:\n scores (paddle.Tensor): The output of the model.\n labels (paddle.Tensor): The target output of the model.\n Returns:\n losses (dict): A dict containing field 'loss'(mandatory) and 'top1_acc', 'top5_acc'(optional).\n \"\"\"\n if len(labels) == 1: #commonly case\n labels = labels[0]\n losses = dict()\n if self.ls_eps != 0. and not valid_mode: # label_smooth\n loss = self.label_smooth_loss(scores, labels, **kwargs)\n else:\n loss = self.loss_func(scores, labels, **kwargs)\n if if_top5:\n top1, top5 = self.get_acc(scores, labels, valid_mode)\n losses['top1'] = top1\n losses['top5'] = top5\n losses['loss'] = loss\n else:"
+ },
+ {
+ "comment": "This code handles different cases for classification tasks. For binary and multi-class tasks, it calculates top1 accuracy and loss, while for the specific case of MRI with three labels (mix_up), it applies label smoothing or regular loss function and averages results for each sample to get the final loss and top1 accuracy.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":91-112",
+ "content": " top1 = self.get_acc(scores, labels, valid_mode, if_top5)\n losses['top1'] = top1\n losses['loss'] = loss\n return losses\n # MRI\u76ee\u524d\u4e8c\u5206\u7c7b\u65e0top5\n elif len(labels) == 3: # mix_up\n labels_a, labels_b, lam = labels\n lam = lam[0] # get lam value\n losses = dict()\n if self.ls_eps != 0:\n loss_a = self.label_smooth_loss(scores, labels_a, **kwargs)\n loss_b = self.label_smooth_loss(scores, labels_b, **kwargs)\n else:\n loss_a = self.loss_func(scores, labels_a, **kwargs)\n loss_b = self.loss_func(scores, labels_b, **kwargs)\n loss = lam * loss_a + (1 - lam) * loss_b\n if if_top5:\n top1a, top5a = self.get_acc(scores, labels_a, valid_mode)\n top1b, top5b = self.get_acc(scores, labels_b, valid_mode)\n top1 = lam * top1a + (1 - lam) * top1b\n top5 = lam * top5a + (1 - lam) * top5b"
+ },
+ {
+ "comment": "Function defines a loss function for classification tasks, returning a dictionary of losses including top1 and overall loss. If valid_mode is True, calculates accuracy on validation set, otherwise on training set. Top1 accuracies for two sets are combined with specified lambda value to calculate final top1.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":113-142",
+ "content": " losses['top1'] = top1\n losses['top5'] = top5\n losses['loss'] = loss\n else:\n top1a = self.get_acc(scores, labels_a, valid_mode, if_top5)\n top1b = self.get_acc(scores, labels_b, valid_mode, if_top5)\n top1 = lam * top1a + (1 - lam) * top1b\n losses['top1'] = top1\n losses['loss'] = loss\n return losses\n else:\n raise NotImplemented\n def label_smooth_loss(self, scores, labels, **kwargs):\n \"\"\"\n Args:\n scores (paddle.Tensor): [N, num_classes]\n labels (paddle.Tensor): [N, ]\n Returns:\n paddle.Tensor: [1,]\n \"\"\"\n if paddle.is_compiled_with_custom_device('npu'):\n \"\"\"\n Designed for the lack of temporary operators of NPU,\n main idea is to split smooth loss into uniform distribution loss\n and hard label calculation\n \"\"\"\n hard_loss = (1.0 - self.ls_eps) * F.cross_entropy(scores, labels)"
+ },
+ {
+ "comment": "Code is for computing loss and accuracy in a classification model. If the hard label is given, it calculates uniform_loss based on scores and adds it to hard_loss for total loss. Otherwise, it computes soft labels using one-hot encoding and label smoothing, then calculates loss using the provided loss function with soft_label set to True. The get_acc function computes top1 and top5 accuracy, averages them across all cards if valid mode is on, and returns the accuracy.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":143-163",
+ "content": " uniform_loss = (self.ls_eps / self.num_classes) * (\n -F.log_softmax(scores, -1).sum(-1).mean(0))\n loss = hard_loss + uniform_loss\n else:\n labels = F.one_hot(labels, self.num_classes)\n labels = F.label_smooth(labels, epsilon=self.ls_eps)\n labels = paddle.squeeze(labels, axis=1)\n loss = self.loss_func(scores, labels, soft_label=True, **kwargs)\n return loss\n def get_acc(self, scores, labels, valid_mode, if_top5=True):\n if if_top5:\n top1 = paddle.metric.accuracy(input=scores, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=scores, label=labels, k=5)\n _, world_size = get_dist_info()\n #NOTE(shipping): deal with multi cards validate\n if world_size > 1 and valid_mode: #reduce sum when valid\n paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM)\n top1 = top1 / world_size\n paddle.distributed.all_reduce("
+ },
+ {
+ "comment": "This code calculates the top1 and optionally top5 accuracy for a classification task. If distributed training is enabled, it performs all-reduce operation on the calculated metrics to ensure consistency across multiple cards/devices. The reduction operation used is sum, and the results are divided by the total number of devices (world_size) to obtain an average value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/base.py\":164-177",
+ "content": " top5, op=paddle.distributed.ReduceOp.SUM)\n top5 = top5 / world_size\n return top1, top5\n else:\n top1 = paddle.metric.accuracy(input=scores, label=labels, k=1)\n _, world_size = get_dist_info()\n #NOTE(shipping): deal with multi cards validate\n if world_size > 1 and valid_mode: #reduce sum when valid\n paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM)\n top1 = top1 / world_size\n return top1"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1a584934-3d9e-422a-a9a6-81d39d0bccc0.json b/docs/doc/1a584934-3d9e-422a-a9a6-81d39d0bccc0.json
new file mode 100644
index 000000000..6dc458268
--- /dev/null
+++ b/docs/doc/1a584934-3d9e-422a-a9a6-81d39d0bccc0.json
@@ -0,0 +1,245 @@
+{
+ "summary": "The code presents a PaddlePaddle BertEmbeddings class for BERT model embeddings in video action recognition, utilizing self-attention and ACTBERT's backbone for multimodal inputs including text, video, and action data.",
+ "details": [
+ {
+ "comment": "This code is a Python file containing a class named \"BertEmbeddings\" within the PaddlePaddle framework. The class inherits from nn.Layer and appears to contain embeddings for the BERT model. This code also includes comments with copyright information, license details, and an import section with necessary libraries. It introduces a dictionary, ACT2FN, that maps activation functions for use in the BertEmbeddings class.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":0-31",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport sys\nimport numpy as np\nimport math\nimport copy\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout)\nfrom paddle.nn.initializer import Constant, Normal\nfrom ...utils.save_load import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nACT2FN = {\"gelu\": F.gelu, \"relu\": F.relu, \"swish\": F.swish}\nclass BertEmbeddings(nn.Layer):"
+ },
+ {
+ "comment": "BertEmbeddings initializes word, position, and token_type embeddings for a given vocabulary size, maximum position embedding size, type vocab size, hidden size, and hidden dropout probability. Forward function uses input ids and token type ids to generate position ids and then combines the different embeddings with layer normalization and dropout.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":32-51",
+ "content": " \"\"\"Construct the embeddings from word, position and token_type embeddings.\n \"\"\"\n def __init__(self, vocab_size, max_position_embeddings, type_vocab_size,\n hidden_size, hidden_dropout_prob):\n super(BertEmbeddings, self).__init__()\n self.word_embeddings = nn.Embedding(vocab_size,\n hidden_size,\n padding_idx=0)\n self.position_embeddings = nn.Embedding(max_position_embeddings,\n hidden_size)\n self.token_type_embeddings = nn.Embedding(type_vocab_size, hidden_size)\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(hidden_dropout_prob)\n def forward(self, input_ids, token_type_ids=None):\n seq_length = input_ids.shape[1]\n position_ids = paddle.arange(end=seq_length, dtype=\"int64\")\n position_ids = position_ids.unsqueeze(0).expand_as(input_ids)\n if token_type_ids is None:"
+ },
+ {
+ "comment": "This code defines the `ActBert` class, which is a backbone model for video action recognition. It initializes word embeddings, position embeddings, and token type embeddings. The class also includes a forward function that combines these embeddings, applies layer normalization and dropout, and returns the result. Additionally, there's the `BertImageEmbeddings` class which takes image features and their locations as input and uses linear layers to generate embeddings for both, followed by layer normalization and dropout.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":52-74",
+ "content": " token_type_ids = paddle.zeros_like(input_ids)\n words_embeddings = self.word_embeddings(input_ids) #8,36 -> 8,36,768\n position_embeddings = self.position_embeddings(\n position_ids) #8,36 -> 8,36,768\n token_type_embeddings = self.token_type_embeddings(\n token_type_ids) #8,36 -> 8,36,768\n embeddings = words_embeddings + position_embeddings + token_type_embeddings\n embeddings = self.LayerNorm(embeddings)\n embeddings = self.dropout(embeddings)\n return embeddings\nclass BertImageEmbeddings(nn.Layer):\n def __init__(self, v_feature_size, v_hidden_size, v_hidden_dropout_prob):\n super(BertImageEmbeddings, self).__init__()\n self.image_embeddings = nn.Linear(v_feature_size, v_hidden_size)\n self.image_location_embeddings = nn.Linear(5, v_hidden_size)\n self.LayerNorm = nn.LayerNorm(v_hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(v_hidden_dropout_prob)\n def forward(self, input_ids, input_loc):"
+ },
+ {
+ "comment": "This code defines two classes: `BertActionEmbeddings` and `BertSelfAttention`. The former takes action features as input, linearly projects them into hidden states, normalizes these using LayerNorm, applies dropout, and returns the embeddings. The latter performs self-attention over the embeddings produced by `BertActionEmbeddings`, applies a feed-forward network, applies LayerNorm, and finally applies dropout before returning the output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":75-99",
+ "content": " img_embeddings = self.image_embeddings(\n input_ids) #8,37,2048 -> 8,37,1024\n loc_embeddings = self.image_location_embeddings(\n input_loc) #8,37,5 -> 8,37,1024\n embeddings = self.LayerNorm(img_embeddings + loc_embeddings)\n embeddings = self.dropout(embeddings)\n return embeddings # shape: bs*seq_len*hs\nclass BertActionEmbeddings(nn.Layer):\n def __init__(self, a_feature_size, a_hidden_size, a_hidden_dropout_prob):\n super(BertActionEmbeddings, self).__init__()\n self.action_embeddings = nn.Linear(a_feature_size, a_hidden_size)\n self.LayerNorm = nn.LayerNorm(a_hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(a_hidden_dropout_prob)\n def forward(self, input_ids):\n action_embeddings = self.action_embeddings(\n input_ids) #8,5,2048 -> 8,5,768\n embeddings = self.LayerNorm(action_embeddings)\n embeddings = self.dropout(embeddings)\n return embeddings\nclass BertSelfAttention(nn.Layer):"
+ },
+ {
+ "comment": "This code defines a BertSelfAttention class with parameters: hidden_size, num_attention_heads, and attention_probs_dropout_prob. It checks if the hidden size is divisible by the number of attention heads. If not, it raises a ValueError. Then, it calculates attention_head_size and all_head_size. Finally, it initializes query, key, value linear layers and dropout layer for attention probabilities.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":100-122",
+ "content": " def __init__(self, hidden_size, num_attention_heads,\n attention_probs_dropout_prob):\n super(BertSelfAttention, self).__init__()\n if hidden_size % num_attention_heads != 0:\n raise ValueError(\n \"The hidden size (%d) is not a multiple of the number of attention \"\n \"heads (%d)\" % (hidden_size, num_attention_heads))\n self.num_attention_heads = num_attention_heads\n self.attention_head_size = int(hidden_size / num_attention_heads)\n self.all_head_size = self.num_attention_heads * self.attention_head_size\n self.query = nn.Linear(hidden_size, self.all_head_size)\n self.key = nn.Linear(hidden_size, self.all_head_size)\n self.value = nn.Linear(hidden_size, self.all_head_size)\n self.dropout = nn.Dropout(attention_probs_dropout_prob)\n def transpose_for_scores(self, x):\n new_x_shape = x.shape[:-1] + [\n self.num_attention_heads,\n self.attention_head_size,\n ]\n x = x.reshape(new_x_shape)"
+ },
+ {
+ "comment": "This code performs multi-head attention in an attention mechanism. It transposes the query, key, and value layers before calculating raw attention scores via dot product. The results are then normalized into probabilities using softmax. The attention mask is applied to the attention scores for masked self-attention.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":123-143",
+ "content": " return x.transpose((0, 2, 1, 3))\n def forward(self, hidden_states, attention_mask):\n mixed_query_layer = self.query(hidden_states)\n mixed_key_layer = self.key(hidden_states)\n mixed_value_layer = self.value(hidden_states)\n query_layer = self.transpose_for_scores(mixed_query_layer)\n key_layer = self.transpose_for_scores(mixed_key_layer)\n value_layer = self.transpose_for_scores(mixed_value_layer)\n # Take the dot product between \"query\" and \"key\" to get the raw attention scores.\n attention_scores = paddle.matmul(query_layer,\n key_layer.transpose((0, 1, 3, 2)))\n attention_scores = attention_scores / math.sqrt(\n self.attention_head_size)\n # Apply the attention mask is (precomputed for all layers in BertModel forward() function)\n attention_scores = attention_scores + attention_mask\n # Normalize the attention scores to probabilities.\n attention_probs = nn.Softmax(axis=-1)(attention_scores)"
+ },
+ {
+ "comment": "This code defines a BertSelfOutput layer that takes input hidden states, applies linear transformation and dropout for regularization, then passes the output through layer normalization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":145-168",
+ "content": " # This is actually dropping out entire tokens to attend to, which might\n # seem a bit unusual, but is taken from the original Transformer paper.\n attention_probs = self.dropout(attention_probs)\n context_layer = paddle.matmul(attention_probs, value_layer)\n context_layer = context_layer.transpose((0, 2, 1, 3))\n new_context_layer_shape = context_layer.shape[:-2] + [\n self.all_head_size\n ]\n context_layer = context_layer.reshape(new_context_layer_shape)\n return context_layer, attention_probs\nclass BertSelfOutput(nn.Layer):\n def __init__(self, hidden_size, hidden_dropout_prob):\n super(BertSelfOutput, self).__init__()\n self.dense = nn.Linear(hidden_size, hidden_size)\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(hidden_dropout_prob)\n def forward(self, hidden_states, input_tensor):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.dropout(hidden_states)"
+ },
+ {
+ "comment": "This code defines three classes: `ActBert`, `BertAttention`, and `BertIntermediate`. \n\n`ActBert` appears to be a model that includes `BertAttention` and `BertIntermediate` as its layers. The `BertAttention` class defines forward function for attention mechanism, which takes in an input tensor and attention mask, and returns output and attention probabilities. The `BertIntermediate` class appears to be a dense layer with linear activation function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":169-191",
+ "content": " hidden_states = self.LayerNorm(hidden_states + input_tensor)\n return hidden_states\nclass BertAttention(nn.Layer):\n def __init__(self, hidden_size, hidden_dropout_prob, num_attention_heads,\n attention_probs_dropout_prob):\n super(BertAttention, self).__init__()\n self.self = BertSelfAttention(hidden_size, num_attention_heads,\n attention_probs_dropout_prob)\n self.output = BertSelfOutput(hidden_size, hidden_dropout_prob)\n def forward(self, input_tensor, attention_mask):\n self_output, attention_probs = self.self(input_tensor, attention_mask)\n attention_output = self.output(self_output, input_tensor)\n return attention_output, attention_probs\nclass BertIntermediate(nn.Layer):\n def __init__(self, hidden_size, intermediate_size, hidden_act):\n super(BertIntermediate, self).__init__()\n self.dense = nn.Linear(hidden_size, intermediate_size)\n if isinstance(hidden_act, str) or (sys.version_info[0] == 2"
+ },
+ {
+ "comment": "Code defines a class for an attention-based transformer model. It includes an intermediate activation function and forward pass layers for processing input, applying dropout regularization, and normalizing outputs. The BertEntAttention layer is the core module for the transformer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":192-218",
+ "content": " and isinstance(hidden_act, str)):\n self.intermediate_act_fn = ACT2FN[hidden_act]\n else:\n self.intermediate_act_fn = hidden_act\n def forward(self, hidden_states):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.intermediate_act_fn(hidden_states)\n return hidden_states\nclass BertOutput(nn.Layer):\n def __init__(self, intermediate_size, hidden_size, hidden_dropout_prob):\n super(BertOutput, self).__init__()\n self.dense = nn.Linear(intermediate_size, hidden_size)\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout = nn.Dropout(hidden_dropout_prob)\n def forward(self, hidden_states, input_tensor):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.dropout(hidden_states)\n hidden_states = self.LayerNorm(hidden_states + input_tensor)\n return hidden_states\nclass BertEntAttention(nn.Layer):\n \"\"\"Core mudule of tangled transformer."
+ },
+ {
+ "comment": "This code defines a BertEntAttention class with parameters for hidden size, vision input hidden size, attention probabilities dropout probabilities, and bi-directional hidden size. It also checks if the hidden size is a multiple of the number of attention heads. The class initializes attributes such as the number of attention heads, attention head size, all head size, and linear layers for self-attention in vision input.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":219-245",
+ "content": " \"\"\"\n def __init__(\n self,\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n bi_num_attention_heads,\n ):\n super(BertEntAttention, self).__init__()\n if bi_hidden_size % bi_num_attention_heads != 0:\n raise ValueError(\n \"The hidden size (%d) is not a multiple of the number of attention \"\n \"heads (%d)\" % (bi_hidden_size, bi_num_attention_heads))\n self.num_attention_heads = bi_num_attention_heads\n self.attention_head_size = int(bi_hidden_size / bi_num_attention_heads)\n self.all_head_size = self.num_attention_heads * self.attention_head_size\n # self attention layers for vision input\n self.query1 = nn.Linear(v_hidden_size, self.all_head_size)\n self.key1 = nn.Linear(v_hidden_size, self.all_head_size)"
+ },
+ {
+ "comment": "This code defines layers for self-attention in the ACTBERT model, including linear and dropout layers for text, action, action_text, and action_vision inputs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":246-266",
+ "content": " self.value1 = nn.Linear(v_hidden_size, self.all_head_size)\n self.dropout1 = nn.Dropout(v_attention_probs_dropout_prob)\n # self attention layers for text input\n self.query2 = nn.Linear(hidden_size, self.all_head_size)\n self.key2 = nn.Linear(hidden_size, self.all_head_size)\n self.value2 = nn.Linear(hidden_size, self.all_head_size)\n self.dropout2 = nn.Dropout(attention_probs_dropout_prob)\n # self attention layers for action input\n self.query3 = nn.Linear(a_hidden_size, self.all_head_size)\n self.key3 = nn.Linear(a_hidden_size, self.all_head_size)\n self.value3 = nn.Linear(a_hidden_size, self.all_head_size)\n self.dropout3 = nn.Dropout(a_attention_probs_dropout_prob)\n # self attention layers for action_text\n self.key_at = nn.Linear(bi_hidden_size, self.all_head_size)\n self.value_at = nn.Linear(bi_hidden_size, self.all_head_size)\n self.dropout_at = nn.Dropout(av_attention_probs_dropout_prob)\n # self attention layers for action_vision"
+ },
+ {
+ "comment": "This code defines a model for attention mechanism in a transformer architecture, used for both vision and text inputs. The key steps involve creating linear layers for keys and values, applying dropout, transposing the input tensors for scoring, and forwarding the input through these operations for both vision and text inputs separately.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":267-298",
+ "content": " self.key_av = nn.Linear(bi_hidden_size, self.all_head_size)\n self.value_av = nn.Linear(bi_hidden_size, self.all_head_size)\n self.dropout_av = nn.Dropout(at_attention_probs_dropout_prob)\n def transpose_for_scores(self, x):\n new_x_shape = x.shape[:-1] + [\n self.num_attention_heads,\n self.attention_head_size,\n ]\n x = x.reshape(new_x_shape)\n return x.transpose((0, 2, 1, 3))\n def forward(\n self,\n input_tensor1,\n attention_mask1,\n input_tensor2,\n attention_mask2,\n input_tensor3,\n attention_mask3,\n ):\n # for vision input.\n mixed_query_layer1 = self.query1(input_tensor1)\n mixed_key_layer1 = self.key1(input_tensor1)\n mixed_value_layer1 = self.value1(input_tensor1)\n query_layer1 = self.transpose_for_scores(mixed_query_layer1)\n key_layer1 = self.transpose_for_scores(mixed_key_layer1)\n value_layer1 = self.transpose_for_scores(mixed_value_layer1)\n # for text input:"
+ },
+ {
+ "comment": "This code is performing multi-head attention operation. It first separates the input tensor into two parts, with each part going through its own set of linear layers to create query, key, and value tensors. Then it transposes the query and key tensors before computing attention scores by taking the dot product of the transposed query and key tensors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":299-320",
+ "content": " mixed_query_layer2 = self.query2(input_tensor2)\n mixed_key_layer2 = self.key2(input_tensor2)\n mixed_value_layer2 = self.value2(input_tensor2)\n query_layer2 = self.transpose_for_scores(mixed_query_layer2)\n key_layer2 = self.transpose_for_scores(mixed_key_layer2)\n value_layer2 = self.transpose_for_scores(mixed_value_layer2)\n # for action input:\n mixed_query_layer3 = self.query3(input_tensor3)\n mixed_key_layer3 = self.key3(input_tensor3)\n mixed_value_layer3 = self.value3(input_tensor3)\n query_layer3 = self.transpose_for_scores(mixed_query_layer3)\n key_layer3 = self.transpose_for_scores(mixed_key_layer3)\n value_layer3 = self.transpose_for_scores(mixed_value_layer3)\n def do_attention(query_layer, key_layer, value_layer, attention_mask,\n dropout):\n \"\"\" compute attention \"\"\"\n attention_scores = paddle.matmul(query_layer,\n key_layer.transpose((0, 1, 3, 2)))"
+ },
+ {
+ "comment": "This code calculates attention scores between queries, keys, and values, normalizes them to probabilities using softmax, applies dropout, performs matrix multiplication with the values, transposes the result, reshapes it, and returns the context layer. It follows the Transformer paper's approach of dropping out entire tokens to attend to.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":321-341",
+ "content": " attention_scores = attention_scores / math.sqrt(\n self.attention_head_size)\n attention_scores = attention_scores + attention_mask\n # Normalize the attention scores to probabilities.\n attention_probs = nn.Softmax(axis=-1)(attention_scores)\n # This is actually dropping out entire tokens to attend to, which might\n # seem a bit unusual, but is taken from the original Transformer paper.\n attention_probs = dropout(attention_probs)\n context_layer = paddle.matmul(attention_probs, value_layer)\n context_layer = context_layer.transpose((0, 2, 1, 3))\n new_context_layer_shape = context_layer.shape[:-2] + [\n self.all_head_size\n ]\n context_layer = context_layer.reshape(new_context_layer_shape)\n return context_layer\n context_av = do_attention(query_layer3, key_layer1, value_layer1,\n attention_mask1, self.dropout_av)"
+ },
+ {
+ "comment": "This code is performing attention mechanism for multi-scale context fusion. It uses interpolation to resize the context features, then adds them to the original key layers (key_layer2 and key_layer1). The purpose of this is to incorporate contextual information from different scales into the model's understanding.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":342-360",
+ "content": " context_at = do_attention(query_layer3, key_layer2, value_layer2,\n attention_mask2, self.dropout_at)\n context_key_av = self.key_av(context_av).transpose((0, 2, 1))\n # interpolate only support 4-D tensor now.\n context_key_av = F.interpolate(context_key_av.unsqueeze(-1),\n size=(key_layer2.shape[2],\n 1)).squeeze(-1)\n context_key_av = self.transpose_for_scores(\n context_key_av.transpose((0, 2, 1)))\n key_layer2 = key_layer2 + context_key_av\n context_key_at = self.key_at(context_at).transpose((0, 2, 1))\n context_key_at = F.interpolate(context_key_at.unsqueeze(-1),\n size=(key_layer1.shape[2],\n 1)).squeeze(-1)\n context_key_at = self.transpose_for_scores(\n context_key_at.transpose((0, 2, 1)))\n key_layer1 = key_layer1 + context_key_at"
+ },
+ {
+ "comment": "This code snippet is performing cross-attention in a transformer model. It first interpolates and adds context vectors to value layers, then applies attention mechanisms to compute context layers. This process helps to capture dependencies between different parts of the input data effectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":362-380",
+ "content": " context_val_av = self.value_at(context_av).transpose((0, 2, 1))\n context_val_av = F.interpolate(context_val_av.unsqueeze(-1),\n size=(value_layer2.shape[2],\n 1)).squeeze(-1)\n context_val_av = self.transpose_for_scores(\n context_val_av.transpose((0, 2, 1)))\n value_layer2 = value_layer2 + context_val_av\n context_val_at = self.value_at(context_at).transpose((0, 2, 1))\n context_val_at = F.interpolate(context_val_at.unsqueeze(-1),\n size=(value_layer1.shape[2],\n 1)).squeeze(-1)\n context_val_at = self.transpose_for_scores(\n context_val_at.transpose((0, 2, 1)))\n value_layer1 = value_layer1 + context_val_at\n context_layer1 = do_attention(query_layer1, key_layer1, value_layer1,\n attention_mask1, self.dropout1)\n context_layer2 = do_attention(query_layer2, key_layer2, value_layer2,"
+ },
+ {
+ "comment": "The code defines a class \"BertEntOutput\" with several layers including dense and dropout. It uses bi-hidden size, hidden size, v_hidden_size, and corresponding dropout probabilities for initializing the layers. This class seems to be part of a model architecture where it performs layer normalization, applies dropout regularization, and linear transformations to process input data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":381-408",
+ "content": " attention_mask2, self.dropout2)\n context_layer3 = do_attention(query_layer3, key_layer3, value_layer3,\n attention_mask3, self.dropout3)\n return context_layer1, context_layer2, context_layer3 # vision, text, action\nclass BertEntOutput(nn.Layer):\n def __init__(\n self,\n bi_hidden_size,\n hidden_size,\n v_hidden_size,\n v_hidden_dropout_prob,\n hidden_dropout_prob,\n ):\n super(BertEntOutput, self).__init__()\n self.dense1 = nn.Linear(bi_hidden_size, v_hidden_size)\n self.LayerNorm1 = nn.LayerNorm(v_hidden_size, epsilon=1e-12)\n self.dropout1 = nn.Dropout(v_hidden_dropout_prob)\n self.dense2 = nn.Linear(bi_hidden_size, hidden_size)\n self.LayerNorm2 = nn.LayerNorm(hidden_size, epsilon=1e-12)\n self.dropout2 = nn.Dropout(hidden_dropout_prob)\n self.dense3 = nn.Linear(bi_hidden_size, hidden_size)\n self.LayerNorm3 = nn.LayerNorm(hidden_size, epsilon=1e-12)"
+ },
+ {
+ "comment": "This code defines a BertLayer class with Dropout layers and dense layers, performing attention mechanism. The forward method applies these layers to hidden_states1,2,3, and returns the updated hidden states after adding them with their corresponding input tensors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":409-439",
+ "content": " self.dropout3 = nn.Dropout(hidden_dropout_prob)\n def forward(\n self,\n hidden_states1,\n input_tensor1,\n hidden_states2,\n input_tensor2,\n hidden_states3,\n input_tensor3,\n ):\n context_state1 = self.dense1(hidden_states1)\n context_state1 = self.dropout1(context_state1)\n context_state2 = self.dense2(hidden_states2)\n context_state2 = self.dropout2(context_state2)\n context_state3 = self.dense3(hidden_states3)\n context_state3 = self.dropout3(context_state3)\n hidden_states1 = self.LayerNorm1(context_state1 + input_tensor1)\n hidden_states2 = self.LayerNorm2(context_state2 + input_tensor2)\n hidden_states3 = self.LayerNorm3(context_state3 + input_tensor3)\n return hidden_states1, hidden_states2, hidden_states3\nclass BertLayer(nn.Layer):\n def __init__(self, hidden_size, intermediate_size, hidden_act,\n hidden_dropout_prob, num_attention_heads,\n attention_probs_dropout_prob):"
+ },
+ {
+ "comment": "This code defines a BertLayer class and a BertConnectionLayer class. The BertLayer class includes an attention layer, an intermediate layer, and an output layer. It has a forward function that performs the calculations for these layers. The BertConnectionLayer class is a subclass of nn.Layer with various parameters for hidden sizes and attention dropout probabilities.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":440-460",
+ "content": " super(BertLayer, self).__init__()\n self.attention = BertAttention(hidden_size, hidden_dropout_prob,\n num_attention_heads,\n attention_probs_dropout_prob)\n self.intermediate = BertIntermediate(hidden_size, intermediate_size,\n hidden_act)\n self.output = BertOutput(intermediate_size, hidden_size,\n hidden_dropout_prob)\n def forward(self, hidden_states, attention_mask):\n attention_output, attention_probs = self.attention(\n hidden_states, attention_mask)\n intermediate_output = self.intermediate(attention_output)\n layer_output = self.output(intermediate_output, attention_output)\n return layer_output, attention_probs\nclass BertConnectionLayer(nn.Layer):\n def __init__(self, hidden_size, v_hidden_size, a_hidden_size,\n bi_hidden_size, bi_num_attention_heads,\n attention_probs_dropout_prob, v_attention_probs_dropout_prob,"
+ },
+ {
+ "comment": "This code initializes a BertConnectionLayer object with various parameters including hidden size, attention probabilities dropout probability, and intermediate size. It also initializes two other objects: BertEntAttention and BertEntOutput. The BertEntAttention object is responsible for performing entity-based attention while the BertEntOutput object is responsible for producing the output of the connection layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":461-486",
+ "content": " a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob, intermediate_size,\n v_intermediate_size, a_intermediate_size, hidden_act,\n v_hidden_act, a_hidden_act, hidden_dropout_prob,\n v_hidden_dropout_prob, a_hidden_dropout_prob):\n super(BertConnectionLayer, self).__init__()\n self.ent_attention = BertEntAttention(\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n bi_num_attention_heads,\n )\n self.ent_output = BertEntOutput(\n bi_hidden_size,\n hidden_size,\n v_hidden_size,\n v_hidden_dropout_prob,\n hidden_dropout_prob,"
+ },
+ {
+ "comment": "This code defines a model with three input streams (v, t, and a) and initializes intermediate layers and output layers for each stream. The forward function takes in two pairs of input tensors and attention masks for each stream.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":487-511",
+ "content": " )\n self.v_intermediate = BertIntermediate(v_hidden_size,\n v_intermediate_size,\n v_hidden_act)\n self.v_output = BertOutput(v_intermediate_size, v_hidden_size,\n v_hidden_dropout_prob)\n self.t_intermediate = BertIntermediate(hidden_size, intermediate_size,\n hidden_act)\n self.t_output = BertOutput(intermediate_size, hidden_size,\n hidden_dropout_prob)\n self.a_intermediate = BertIntermediate(a_hidden_size,\n a_intermediate_size,\n a_hidden_act)\n self.a_output = BertOutput(a_intermediate_size, a_hidden_size,\n a_hidden_dropout_prob)\n def forward(\n self,\n input_tensor1,\n attention_mask1,\n input_tensor2,\n attention_mask2,"
+ },
+ {
+ "comment": "This function computes the layer outputs for three different pathways using the ActBert Encoder. It utilizes attention masks and input tensors for each pathway, passing them through various intermediate layers before returning the final layer outputs. The BertEncoder class represents a combination of three multi-BertLayers and BertConnectionLayer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":512-538",
+ "content": " input_tensor3,\n attention_mask3,\n ):\n ent_output1, ent_output2, ent_output3 = self.ent_attention(\n input_tensor1, attention_mask1, input_tensor2, attention_mask2,\n input_tensor3, attention_mask3)\n attention_output1, attention_output2, attention_output3 = self.ent_output(\n ent_output1, input_tensor1, ent_output2, input_tensor2, ent_output3,\n input_tensor3)\n intermediate_output1 = self.v_intermediate(attention_output1)\n layer_output1 = self.v_output(intermediate_output1, attention_output1)\n intermediate_output2 = self.t_intermediate(attention_output2)\n layer_output2 = self.t_output(intermediate_output2, attention_output2)\n intermediate_output3 = self.a_intermediate(attention_output3)\n layer_output3 = self.a_output(intermediate_output3, attention_output3)\n return layer_output1, layer_output2, layer_output3\nclass BertEncoder(nn.Layer):\n \"\"\"\n ActBert Encoder, consists 3 pathway of multi-BertLayers and BertConnectionLayer."
+ },
+ {
+ "comment": "This code defines the `BertEncoder` class, which initializes various parameters for the BERT model's encoder. These parameters include attention IDs, fixed layer positions, hidden sizes, and dropout probabilities for different components of the model. The class extends `super(BertEncoder, self).__init__()`, indicating it inherits from another class.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":539-575",
+ "content": " \"\"\"\n def __init__(\n self,\n v_ent_attention_id,\n t_ent_attention_id,\n a_ent_attention_id,\n fixed_t_layer,\n fixed_v_layer,\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n intermediate_size,\n v_intermediate_size,\n a_intermediate_size,\n hidden_act,\n v_hidden_act,\n a_hidden_act,\n hidden_dropout_prob,\n v_hidden_dropout_prob,\n a_hidden_dropout_prob,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n num_attention_heads,\n v_num_attention_heads,\n a_num_attention_heads,\n bi_num_attention_heads,\n num_hidden_layers,\n v_num_hidden_layers,\n a_num_hidden_layers,\n ):\n super(BertEncoder, self).__init__()\n self.v_ent_attention_id = v_ent_attention_id\n self.t_ent_attention_id = t_ent_attention_id"
+ },
+ {
+ "comment": "This code initializes three BertLayer objects and one BertConnectionLayer object with different hidden sizes, intermediate sizes, activation functions, dropout probabilities, and attention head numbers. These layers will be used for encoding input sequences in the Actor-Critic Transformer (ACT) model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":576-593",
+ "content": " self.a_ent_attention_id = a_ent_attention_id\n self.fixed_t_layer = fixed_t_layer\n self.fixed_v_layer = fixed_v_layer\n layer = BertLayer(hidden_size, intermediate_size, hidden_act,\n hidden_dropout_prob, num_attention_heads,\n attention_probs_dropout_prob)\n v_layer = BertLayer(v_hidden_size, v_intermediate_size, v_hidden_act,\n v_hidden_dropout_prob, v_num_attention_heads,\n v_attention_probs_dropout_prob)\n a_layer = BertLayer(a_hidden_size, a_intermediate_size, a_hidden_act,\n a_hidden_dropout_prob, a_num_attention_heads,\n a_attention_probs_dropout_prob)\n connect_layer = BertConnectionLayer(\n hidden_size, v_hidden_size, a_hidden_size, bi_hidden_size,\n bi_num_attention_heads, attention_probs_dropout_prob,\n v_attention_probs_dropout_prob, a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob, at_attention_probs_dropout_prob,"
+ },
+ {
+ "comment": "This code defines a model with separate layers for text (txt_layer), vision (v_layer), and action (a_layer) embeddings. It also includes a connect_layer to combine the visual and action information for attention masks. The forward method takes input embeddings, attention masks, and an optional parameter to output all encoded layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":594-621",
+ "content": " intermediate_size, v_intermediate_size, a_intermediate_size,\n hidden_act, v_hidden_act, a_hidden_act, hidden_dropout_prob,\n v_hidden_dropout_prob, a_hidden_dropout_prob)\n self.layer = nn.LayerList(\n [copy.deepcopy(layer) for _ in range(num_hidden_layers)]) #12\n self.v_layer = nn.LayerList(\n [copy.deepcopy(v_layer) for _ in range(v_num_hidden_layers)]) #2\n self.a_layer = nn.LayerList(\n [copy.deepcopy(a_layer) for _ in range(a_num_hidden_layers)]) #3\n self.c_layer = nn.LayerList([\n copy.deepcopy(connect_layer) for _ in range(len(v_ent_attention_id))\n ] #2 [0,1]\n )\n def forward(\n self,\n txt_embedding,\n image_embedding,\n action_embedding,\n txt_attention_mask,\n image_attention_mask,\n action_attention_mask,\n output_all_encoded_layers=True,\n ):\n v_start, a_start, t_start = 0, 0, 0\n count = 0\n all_encoder_layers_t = []"
+ },
+ {
+ "comment": "This code initializes empty lists for all encoder layers in vision and audio. It then iterates through the given layer IDs, splitting them into vision (v), audio (a), and time (t) layers. The code asserts that the fixed time layer is less than or equal to the last time layer, and the fixed vision layer is less than or equal to the last vision layer. Next, it iterates through all vision layers from the start index up to but not including the fixed vision layer. Inside this loop, it applies the corresponding vision layer to the image embedding and attention mask using Paddle's no_grad context manager. Finally, it loops over all vision layers from the start index to the end index (fixed vision layer excluded), applying the corresponding vision layer to the image embedding and attention mask.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":622-644",
+ "content": " all_encoder_layers_v = []\n all_encoder_layers_a = []\n for v_layer_id, a_layer_id, t_layer_id in zip(self.v_ent_attention_id,\n self.a_ent_attention_id,\n self.t_ent_attention_id):\n v_end = v_layer_id\n a_end = a_layer_id\n t_end = t_layer_id\n assert self.fixed_t_layer <= t_end\n assert self.fixed_v_layer <= v_end\n ### region embedding\n for idx in range(v_start,\n self.fixed_v_layer): #\u4e24\u6b21\u8bad\u7ec3\uff0c\u8fd9\u4e2a\u5faa\u73af\u90fd\u6ca1\u6709\u8fdb\u53bb #\u524d\u9762\u7684\u5c42\u56fa\u5b9a\u4f4f\n with paddle.no_grad():\n image_embedding, image_attention_probs = self.v_layer[idx](\n image_embedding, image_attention_mask)\n v_start = self.fixed_v_layer\n for idx in range(v_start, v_end):\n image_embedding, image_attention_probs = self.v_layer[idx](\n image_embedding, image_attention_mask)"
+ },
+ {
+ "comment": "This code is performing multi-modal embedding by separately handling action, text, and image embeddings. It iterates through layers for each modality to compute the embeddings and attention probs. Finally, it combines the embeddings in a specific order before potentially updating start/end indices.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":646-668",
+ "content": " ### action embedding\n for idx in range(a_start, a_end):\n action_embedding, action_attention_probs = self.a_layer[idx](\n action_embedding, action_attention_mask)\n ### text embedding\n for idx in range(t_start, self.fixed_t_layer):\n with paddle.no_grad():\n txt_embedding, txt_attention_probs = self.layer[idx](\n txt_embedding, txt_attention_mask)\n t_start = self.fixed_t_layer\n for idx in range(t_start, t_end):\n txt_embedding, txt_attention_probs = self.layer[idx](\n txt_embedding, txt_attention_mask)\n image_embedding, txt_embedding, action_embedding = self.c_layer[\n count](image_embedding, image_attention_mask, txt_embedding,\n txt_attention_mask, action_embedding,\n action_attention_mask)\n v_start = v_end\n t_start = t_end\n a_start = a_end"
+ },
+ {
+ "comment": "This code is responsible for encoding text, image, and action inputs in a neural network model. It iterates over the layers of each input type to produce their respective encoded representations. If output_all_encoded_layers is set to True, it appends all intermediate encoded layers to separate lists; otherwise, only the final encoded layer is stored. This allows for flexibility in selecting which encoded layers to use in further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":669-692",
+ "content": " count += 1\n if output_all_encoded_layers:\n all_encoder_layers_t.append(txt_embedding)\n all_encoder_layers_v.append(image_embedding)\n all_encoder_layers_a.append(action_embedding)\n for idx in range(v_start, len(self.v_layer)): # 1\n image_embedding, image_attention_probs = self.v_layer[idx](\n image_embedding, image_attention_mask)\n for idx in range(a_start, len(self.a_layer)):\n action_embedding, action_attention_probs = self.a_layer[idx](\n action_embedding, action_attention_mask)\n for idx in range(t_start, len(self.layer)):\n txt_embedding, txt_attention_probs = self.layer[idx](\n txt_embedding, txt_attention_mask)\n # add the end part to finish.\n if not output_all_encoded_layers:\n all_encoder_layers_t.append(txt_embedding) #8, 36, 768\n all_encoder_layers_v.append(image_embedding) #8, 37, 1024\n all_encoder_layers_a.append(action_embedding) #8, 5, 768"
+ },
+ {
+ "comment": "The code defines a BertPooler class that pools the model by taking the hidden state corresponding to the first token. It also includes a BertModel class with various parameters for initializing the BERT model, including vocab size, max position embeddings, type vocab size, and feature sizes for different entities (v, a). The code also defines attention IDs and fixed layers for tokens and aspects.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":694-728",
+ "content": " return all_encoder_layers_t, all_encoder_layers_v, all_encoder_layers_a\nclass BertPooler(nn.Layer):\n \"\"\" \"Pool\" the model by simply taking the hidden state corresponding\n to the first token.\n \"\"\"\n def __init__(self, hidden_size, bi_hidden_size):\n super(BertPooler, self).__init__()\n self.dense = nn.Linear(hidden_size, bi_hidden_size)\n self.activation = nn.ReLU()\n def forward(self, hidden_states):\n first_token_tensor = hidden_states[:, 0] #8, 768\n pooled_output = self.dense(first_token_tensor)\n pooled_output = self.activation(pooled_output)\n return pooled_output\nclass BertModel(nn.Layer):\n def __init__(\n self,\n vocab_size,\n max_position_embeddings,\n type_vocab_size,\n v_feature_size,\n a_feature_size,\n num_hidden_layers,\n v_num_hidden_layers,\n a_num_hidden_layers,\n v_ent_attention_id,\n t_ent_attention_id,\n a_ent_attention_id,\n fixed_t_layer,\n fixed_v_layer,"
+ },
+ {
+ "comment": "This code is initializing a BertModel class with various parameters including hidden size, attention-related parameters, and embedding types. It uses the superclass constructor to initialize the base model and then further customizes it by adding word and image embeddings.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":729-758",
+ "content": " hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n intermediate_size,\n v_intermediate_size,\n a_intermediate_size,\n hidden_act,\n v_hidden_act,\n a_hidden_act,\n hidden_dropout_prob,\n v_hidden_dropout_prob,\n a_hidden_dropout_prob,\n attention_probs_dropout_prob,\n v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n num_attention_heads,\n v_num_attention_heads,\n a_num_attention_heads,\n bi_num_attention_heads,\n ):\n super(BertModel, self).__init__()\n # initilize word embedding\n self.embeddings = BertEmbeddings(vocab_size, max_position_embeddings,\n type_vocab_size, hidden_size,\n hidden_dropout_prob)\n # initlize the region embedding\n self.v_embeddings = BertImageEmbeddings(v_feature_size, v_hidden_size,"
+ },
+ {
+ "comment": "This code is initializing a model for the ACTBERT backbone, which includes an encoder and action embedding. The model has parameters for various hidden sizes, dropout probabilities, attention head numbers, and hidden layer counts for both textual (v), visual (t), and action (a) components. The model also uses different activation functions for each component.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":759-774",
+ "content": " v_hidden_dropout_prob)\n # initlize the action embedding\n self.a_embeddings = BertActionEmbeddings(a_feature_size, a_hidden_size,\n a_hidden_dropout_prob)\n self.encoder = BertEncoder(\n v_ent_attention_id, t_ent_attention_id, a_ent_attention_id,\n fixed_t_layer, fixed_v_layer, hidden_size, v_hidden_size,\n a_hidden_size, bi_hidden_size, intermediate_size,\n v_intermediate_size, a_intermediate_size, hidden_act, v_hidden_act,\n a_hidden_act, hidden_dropout_prob, v_hidden_dropout_prob,\n a_hidden_dropout_prob, attention_probs_dropout_prob,\n v_attention_probs_dropout_prob, a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob, at_attention_probs_dropout_prob,\n num_attention_heads, v_num_attention_heads, a_num_attention_heads,\n bi_num_attention_heads, num_hidden_layers, v_num_hidden_layers,"
+ },
+ {
+ "comment": "This code defines a class for a model that takes in text, action, and image features as inputs. It initializes three poolers for text, action, and visual features. The forward function processes the input data and returns encoded layers based on the inputs received. The output_all_encoded_layers parameter allows getting all encoded layers if set to True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":775-799",
+ "content": " a_num_hidden_layers)\n self.t_pooler = BertPooler(hidden_size, bi_hidden_size)\n self.v_pooler = BertPooler(v_hidden_size, bi_hidden_size)\n self.a_pooler = BertPooler(a_hidden_size, bi_hidden_size)\n def forward(\n self,\n text_ids,\n action_feat,\n image_feat,\n image_loc,\n token_type_ids=None,\n text_mask=None,\n image_mask=None,\n action_mask=None,\n output_all_encoded_layers=False,\n ):\n \"\"\"\n text_ids: input text ids. Shape: [batch_size, seqence_length]\n action_feat: input action feature. Shape: [batch_size, action_length, action_feature_dim]\n image_feat: input image feature. Shape: [batch_size, region_length, image_feature_dim]]\n image_loc: input region location. Shape: [batch_size, region_length, region_location_dim]\n token_type_ids: segment ids of each video clip. Shape: [batch_size, seqence_length]\n text_mask: text mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, seqence_length]"
+ },
+ {
+ "comment": "This code checks if the input masks for text, token_type, image, and action are None. If any of them are None, it generates a mask with the same shape as the corresponding feature tensor and fills it with ones (real tokens) or zeros (padding tokens). The attention mask is created from the 2D tensor mask to be used in the multi-head attention mechanism, which broadcasts to [batch_size, num_heads, from_seq_length, to_seq_length].",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":800-818",
+ "content": " image_mask: image mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, region_length]\n action_mask: action mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, action_length]\n output_all_encoded_layers: is output encoded layers feature or not. Type: Bool.\n \"\"\"\n if text_mask is None:\n text_mask = paddle.ones_like(text_ids)\n if token_type_ids is None:\n token_type_ids = paddle.zeros_like(text_ids)\n if image_mask is None:\n image_mask = paddle.ones(image_feat.shape[0],\n image_feat.shape[1]).astype(text_ids.dtype)\n if action_mask is None:\n action_mask = paddle.ones(action_feat.shape[0],\n action_feat.shape[1]).astype(\n text_ids.dtype)\n # We create a 3D attention mask from a 2D tensor mask.\n # Sizes are [batch_size, 1, 1, to_seq_length]\n # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]."
+ },
+ {
+ "comment": "This code segment is part of a backbone model for the ACTBERT. It creates extended masks for text, image, and action inputs by unsqueezing the existing masks along dimensions 1 and 2. The function set_mask is then used to multiply each mask with -10000.0 at positions where we want to attend, effectively removing those positions from the attention process. This is done for all three input types: text, image, and action. Finally, the code applies the embeddings to the text inputs using self.embeddings.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":819-837",
+ "content": " extended_text_mask = text_mask.unsqueeze(1).unsqueeze(2)\n extended_image_mask = image_mask.unsqueeze(1).unsqueeze(2)\n extended_action_mask = action_mask.unsqueeze(1).unsqueeze(2)\n # Since attention_mask is 1.0 for positions we want to attend and 0.0 for\n # masked positions, this operation will create a tensor which is 0.0 for\n # positions we want to attend and -10000.0 for masked positions.\n # Since we are adding it to the raw scores before the softmax, this is\n # effectively the same as removing these entirely.\n def set_mask(extended_attention_mask):\n extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0\n return extended_attention_mask\n extended_text_mask = set_mask(extended_text_mask)\n extended_image_mask = set_mask(extended_image_mask)\n extended_action_mask = set_mask(extended_action_mask)\n t_embedding_output = self.embeddings(text_ids, token_type_ids)\n v_embedding_output = self.v_embeddings(image_feat, image_loc)"
+ },
+ {
+ "comment": "This code is part of a backbone for a multimodal model, specifically ACTBERT. It first computes the embedding outputs for text (t), vision (v), and action (a) features. Then it passes these embeddings to an encoder to obtain encoded layers for each modality. The last hidden state from each encoder is used as a sequence output, and a pooled output is also computed using separate poolers for each modality. If output_all_encoded_layers is False, the code reduces the encoded layers to their last hidden states.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":838-864",
+ "content": " a_embedding_output = self.a_embeddings(action_feat)\n # var = [t_embedding_output, v_embedding_output, a_embedding_output]\n # import numpy as np\n # for i, item in enumerate(var):\n # np.save('tmp/' + str(i)+'.npy', item.numpy())\n encoded_layers_t, encoded_layers_v, encoded_layers_a = self.encoder(\n t_embedding_output,\n v_embedding_output,\n a_embedding_output,\n extended_text_mask,\n extended_image_mask,\n extended_action_mask,\n output_all_encoded_layers=output_all_encoded_layers,\n )\n sequence_output_t = encoded_layers_t[-1] #get item from list\n sequence_output_v = encoded_layers_v[-1]\n sequence_output_a = encoded_layers_a[-1]\n pooled_output_t = self.t_pooler(sequence_output_t)\n pooled_output_v = self.v_pooler(sequence_output_v)\n pooled_output_a = self.a_pooler(sequence_output_a)\n if not output_all_encoded_layers:\n encoded_layers_t = encoded_layers_t[-1]"
+ },
+ {
+ "comment": "This code defines two classes: `BertPredictionHeadTransform` and `BertLMPredictionHead`. The former is a transform layer used in BERT's prediction heads. It applies a dense layer followed by an activation function and layer normalization. The latter is the prediction head itself, which takes input hidden states, applies the transform defined in `BertPredictionHeadTransform`, and returns output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":865-891",
+ "content": " encoded_layers_v = encoded_layers_v[-1]\n encoded_layers_a = encoded_layers_a[-1]\n return encoded_layers_t, encoded_layers_v, encoded_layers_a, \\\n pooled_output_t, pooled_output_v, pooled_output_a\n# For Head\nclass BertPredictionHeadTransform(nn.Layer):\n def __init__(self, hidden_size, hidden_act):\n super(BertPredictionHeadTransform, self).__init__()\n self.dense = nn.Linear(hidden_size, hidden_size)\n if isinstance(hidden_act, str) or (sys.version_info[0] == 2\n and isinstance(hidden_act, str)):\n self.transform_act_fn = ACT2FN[hidden_act]\n else:\n self.transform_act_fn = hidden_act\n self.LayerNorm = nn.LayerNorm(hidden_size, epsilon=1e-12)\n def forward(self, hidden_states):\n hidden_states = self.dense(hidden_states)\n hidden_states = self.transform_act_fn(hidden_states)\n hidden_states = self.LayerNorm(hidden_states)\n return hidden_states\nclass BertLMPredictionHead(nn.Layer):"
+ },
+ {
+ "comment": "This code initializes the BertLMPredictionHead class, which is a part of the BERT model. It takes in hidden_size, hidden_act, and bert_model_embedding_weights as parameters. The class uses these to initialize its transform and decoder components. The decoder component has a weight equal to the input embedding weights with an output-only bias for each token. This implementation avoids creating additional large parameters by directly assigning the input embedding weights to the decoder's weight attribute.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":892-908",
+ "content": " def __init__(self, hidden_size, hidden_act, bert_model_embedding_weights):\n super(BertLMPredictionHead, self).__init__()\n self.transform = BertPredictionHeadTransform(hidden_size, hidden_act)\n # The output weights are the same as the input embeddings, but there is\n # an output-only bias for each token.\n assert bert_model_embedding_weights.shape[1] == hidden_size\n vocab_size = bert_model_embedding_weights.shape[0]\n # another implementation which would create another big params:\n # self.decoder = nn.Linear(hidden_size, vocab_size) # NOTE bias default: constant 0.0\n # self.decoder.weight = self.create_parameter(shape=[hidden_size, vocab_size],\n # default_initializer=nn.initializer.Assign(\n # bert_model_embedding_weights.t())) # transpose\n self.decoder_weight = bert_model_embedding_weights\n self.decoder_bias = self.create_parameter("
+ },
+ {
+ "comment": "This code defines three classes: ActBert, BertImageActionPredictionHead, and BertPreTrainingHeads. ActBert is a layer that performs attention and feedforward operations in BERT's transformer layers. The BertImageActionPredictionHead class is responsible for the image action prediction task in BERT. Finally, the BertPreTrainingHeads class includes multiple layers for pre-training tasks such as masked language modeling, next sentence prediction, and SQUAD question answering.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":909-936",
+ "content": " shape=[vocab_size],\n dtype=bert_model_embedding_weights.dtype,\n is_bias=True) # NOTE bias default: constant 0.0\n def forward(self, hidden_states):\n hidden_states = self.transform(hidden_states)\n hidden_states = paddle.tensor.matmul(\n hidden_states, self.decoder_weight,\n transpose_y=True) + self.decoder_bias\n return hidden_states\nclass BertImageActionPredictionHead(nn.Layer):\n def __init__(self, hidden_size, hidden_act, target_size):\n super(BertImageActionPredictionHead, self).__init__()\n self.transform = BertPredictionHeadTransform(hidden_size, hidden_act)\n self.decoder = nn.Linear(hidden_size, target_size)\n def forward(self, hidden_states):\n hidden_states = self.transform(hidden_states)\n hidden_states = self.decoder(hidden_states)\n return hidden_states\nclass BertPreTrainingHeads(nn.Layer):\n def __init__(self, hidden_size, v_hidden_size, a_hidden_size,\n bi_hidden_size, hidden_act, v_hidden_act, a_hidden_act,"
+ },
+ {
+ "comment": "The code defines a class BertPreTrainingHeads that extends an existing class. It initializes the necessary modules for prediction and fusion. The forward function performs pooling and fusion operations based on the specified fusion method ('sum').",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":937-955",
+ "content": " v_target_size, a_target_size, fusion_method,\n bert_model_embedding_weights):\n super(BertPreTrainingHeads, self).__init__()\n self.predictions = BertLMPredictionHead(hidden_size, hidden_act,\n bert_model_embedding_weights)\n self.seq_relationship = nn.Linear(bi_hidden_size, 2)\n self.imagePredictions = BertImageActionPredictionHead(\n v_hidden_size, v_hidden_act, v_target_size) # visual class number\n self.actionPredictions = BertImageActionPredictionHead(\n a_hidden_size, a_hidden_act, a_target_size) # action class number\n self.fusion_method = fusion_method\n self.dropout = nn.Dropout(0.1)\n def forward(self, sequence_output_t, sequence_output_v, sequence_output_a,\n pooled_output_t, pooled_output_v, pooled_output_a):\n if self.fusion_method == 'sum':\n pooled_output = self.dropout(pooled_output_t + pooled_output_v +\n pooled_output_a)"
+ },
+ {
+ "comment": "In this code snippet, the model is returning prediction scores for text (t), video (v), and action (a) inputs along with a sequence relationship score. The model is a multi-modal pre-training BERT model with fusion method 'mul'. If the fusion method is not 'mul', it will raise an assertion error.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":956-985",
+ "content": " elif self.fusion_method == 'mul':\n pooled_output = self.dropout(pooled_output_t * pooled_output_v +\n pooled_output_a)\n else:\n assert False\n prediction_scores_t = self.predictions(\n sequence_output_t) # 8\uff0c 36 \uff0c30522\n seq_relationship_score = self.seq_relationship(pooled_output) # 8, 2\n prediction_scores_v = self.imagePredictions(\n sequence_output_v) # 8, 37, 1601\n prediction_scores_a = self.actionPredictions(\n sequence_output_a) # 8, 5, 401\n return prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score\n@BACKBONES.register()\nclass BertForMultiModalPreTraining(nn.Layer):\n \"\"\"BERT model with multi modal pre-training heads.\n \"\"\"\n def __init__(\n self,\n vocab_size=30522,\n max_position_embeddings=512,\n type_vocab_size=2,\n v_target_size=1601,\n a_target_size=700,\n v_feature_size=2048,\n a_feature_size=2048,"
+ },
+ {
+ "comment": "This code defines a custom transformer backbone model for ACT-BERT, with specific configurations for the text (t), video (v), and audio (a) modalities. It includes parameters such as hidden layer numbers, sizes, activation functions, dropout rates, attention heads, and fusion method. The pretrained parameter is set to None.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":986-1017",
+ "content": " num_hidden_layers=12,\n v_num_hidden_layers=2,\n a_num_hidden_layers=3,\n t_ent_attention_id=[10, 11],\n v_ent_attention_id=[0, 1],\n a_ent_attention_id=[0, 1],\n fixed_t_layer=0,\n fixed_v_layer=0,\n hidden_size=768,\n v_hidden_size=1024,\n a_hidden_size=768,\n bi_hidden_size=1024,\n intermediate_size=3072,\n v_intermediate_size=1024,\n a_intermediate_size=3072,\n hidden_act=\"gelu\",\n v_hidden_act=\"gelu\",\n a_hidden_act=\"gelu\",\n hidden_dropout_prob=0.1,\n v_hidden_dropout_prob=0.1,\n a_hidden_dropout_prob=0.1,\n attention_probs_dropout_prob=0.1,\n v_attention_probs_dropout_prob=0.1,\n a_attention_probs_dropout_prob=0.1,\n av_attention_probs_dropout_prob=0.1,\n at_attention_probs_dropout_prob=0.1,\n num_attention_heads=12,\n v_num_attention_heads=8,\n a_num_attention_heads=12,\n bi_num_attention_heads=8,\n fusion_method=\"mul\",\n pretrained=None,"
+ },
+ {
+ "comment": "This function defines the input parameters for an ActBERT model, including vocabulary size, maximum position embedding, type vocab size, visual and action target sizes, feature sizes for vision and actions, number of hidden layers in text, visual, and action transformers, index IDs for BertConnectionLayer, and a fixed layer index for the text transformer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1018-1033",
+ "content": " ):\n \"\"\"\n vocab_size: vocabulary size. Default: 30522.\n max_position_embeddings: max position id. Default: 512.\n type_vocab_size: max segment id. Default: 2.\n v_target_size: class number of visual word. Default: 1601.\n a_target_size: class number of action word. Default: 700.\n v_feature_size: input visual feature dimension. Default: 2048.\n a_feature_size: input action feature dimension. Default: 2048.\n num_hidden_layers: number of BertLayer in text transformer. Default: 12.\n v_num_hidden_layers: number of BertLayer in visual transformer. Default: 2.\n a_num_hidden_layers: number of BertLayer in action transformer. Default:3.\n t_ent_attention_id: index id of BertConnectionLayer in text transformer. Default: [10, 11].\n v_ent_attention_id: index id of BertConnectionLayer in visual transformer. Default:[0, 1].\n a_ent_attention_id: index id of BertConnectionLayer in action transformer. Default:[0, 1].\n fixed_t_layer: index id of fixed BertLayer in text transformer. Default: 0."
+ },
+ {
+ "comment": "This code defines parameters for different BertLayers in a model. Fixed_v_layer is the index of a fixed BertLayer in the visual transformer, hidden_size is the hidden size for text and other BERT layers, v_hidden_size is the hidden size for visual BERT layer, a_hidden_size is the hidden size for action BERT layer, bi_hidden_size is the hidden size for BertConnectionLayer, intermediate_size is the intermediate size for text and other BERT layers, v_intermediate_size is the intermediate size for visual BERT layer, a_intermediate_size is the intermediate size for action BERT layer, hidden_act is the activation function for text BERT layer, v_hidden_act is the activation function for visual BERT layer, a_hidden_act is the activation function for action BERT layer, and hidden_dropout_prob is the dropout probability for text embedding layer. All default values are provided in case no other values are specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1034-1046",
+ "content": " fixed_v_layer: index id of fixed BertLayer in visual transformer. Default: 0.\n hidden_size: hidden size in text BertLayer. Default: 768.\n v_hidden_size: hidden size in visual BertLayer. Default: 1024.\n a_hidden_size: hidden size in action BertLayer. Default: 768.\n bi_hidden_size: hidden size in BertConnectionLayer. Default: 1024,\n intermediate_size: intermediate size in text BertLayer. Default: 3072.\n v_intermediate_size: intermediate size in visual BertLayer. Default: 1024.\n a_intermediate_size: intermediate size in text BertLayer. Default: 3072.\n hidden_act: hidden activation function in text BertLayer. Default: \"gelu\".\n v_hidden_act: hidden activation function in visual BertLayer. Default: \"gelu\".\n a_hidden_act: hidden activation function in action BertLayer. Default: \"gelu\".\n hidden_dropout_prob: hidden dropout probability in text Embedding Layer. Default: 0.1\n v_hidden_dropout_prob: hidden dropout probability in visual Embedding Layer. Default: 0.1"
+ },
+ {
+ "comment": "This code snippet defines default values for various parameters in a transformer model. These parameters include hidden dropout probabilities, attention dropout probabilities, number of attention heads, and fusion methods. The default values are provided for the text, visual, and action BertLayers as well as the BertConnectionLayer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1047-1057",
+ "content": " a_hidden_dropout_prob: hidden dropout probability in action Embedding Layer. Default: 0.1\n attention_probs_dropout_prob: attention dropout probability in text BertLayer. Default: 0.1\n v_attention_probs_dropout_prob: attention dropout probability in visual BertLayer. Default: 0.1\n a_attention_probs_dropout_prob: attention dropout probability in action BertLayer. Default: 0.1\n av_attention_probs_dropout_prob: attention dropout probability in action-visual BertConnectionLayer. Default: 0.1\n at_attention_probs_dropout_prob: attention dropout probability in action-text BertConnectionLayer. Default: 0.1\n num_attention_heads: number of heads in text BertLayer. Default: 12.\n v_num_attention_heads: number of heads in visual BertLayer. Default: 8.\n a_num_attention_heads: number of heads in action BertLayer. Default: 12.\n bi_num_attention_heads: number of heads in BertConnectionLayer. Default: 8.\n fusion_method: methods of fusing pooled output from 3 transformer. Default: \"mul\"."
+ },
+ {
+ "comment": "This code initializes an instance of the BertForMultiModalPreTraining class, which is a pre-trained model for multi-modal tasks. It takes in various parameters such as vocab_size, max_position_embeddings, type_vocab_size, v_feature_size, a_feature_size, num_hidden_layers, v_num_hidden_layers, a_num_hidden_layers, v_ent_attention_id, t_ent_attention_id, a_ent_attention_id, fixed_t_layer, fixed_v_layer, hidden_size, v_hidden_size, a_hidden_size, bi_hidden_size, intermediate_size, v_intermediate_size, a_intermediate_size, hidden_act, v_hidden_act, a_hidden_act, hidden_dropout_prob, v_hidden_dropout_prob, and a_hidden_dropout_prob. These parameters define the architecture and behavior of the model. The super() function is used to call a method from the parent class, in this case, BertModel. The pretrained variable indicates whether the model should use pre-trained weights or not.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1058-1091",
+ "content": " \"\"\"\n super(BertForMultiModalPreTraining, self).__init__()\n self.pretrained = pretrained\n self.vocab_size = vocab_size\n self.a_target_size = a_target_size\n self.bert = BertModel(\n vocab_size,\n max_position_embeddings,\n type_vocab_size,\n v_feature_size,\n a_feature_size,\n num_hidden_layers,\n v_num_hidden_layers,\n a_num_hidden_layers,\n v_ent_attention_id,\n t_ent_attention_id,\n a_ent_attention_id,\n fixed_t_layer,\n fixed_v_layer,\n hidden_size,\n v_hidden_size,\n a_hidden_size,\n bi_hidden_size,\n intermediate_size,\n v_intermediate_size,\n a_intermediate_size,\n hidden_act,\n v_hidden_act,\n a_hidden_act,\n hidden_dropout_prob,\n v_hidden_dropout_prob,\n a_hidden_dropout_prob,\n attention_probs_dropout_prob,"
+ },
+ {
+ "comment": "This code initializes the parameters of a pre-trained ACTBERT model. It checks if the model has been pre-trained and, if not, initializes the weights for the layers (using normal distribution with standard deviation 0.02).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1092-1115",
+ "content": " v_attention_probs_dropout_prob,\n a_attention_probs_dropout_prob,\n av_attention_probs_dropout_prob,\n at_attention_probs_dropout_prob,\n num_attention_heads,\n v_num_attention_heads,\n a_num_attention_heads,\n bi_num_attention_heads,\n )\n self.cls = BertPreTrainingHeads(\n hidden_size, v_hidden_size, a_hidden_size, bi_hidden_size,\n hidden_act, v_hidden_act, a_hidden_act, v_target_size,\n a_target_size, fusion_method,\n self.bert.embeddings.word_embeddings.weight)\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, (nn.Linear, nn.Embedding)):\n weight_init_(layer, 'Normal', std=0.02)"
+ },
+ {
+ "comment": "The code defines a function \"forward\" that takes text_ids, action_feat, image_feat, image_loc, token_type_ids, text_mask, image_mask, and action_mask as input. The text_ids represent input text ids of shape [batch_size, sequence_length]. Action_feat is the input action feature of shape [batch_size, action_length, action_feature_dim], while image_feat is the input image feature of shape [batch_size, region_length+1, image_feature_dim] (adding 1 for global image feature). Image_loc represents the input region location of shape [batch_size, region_length+1, region_location_dim] (adding 1 for global image feature location). Token_type_ids represent segment ids of each video clip and are of shape [batch_size, sequence_length]. Text_mask is a binary mask representing real tokens as 1 and padding tokens as 0 with shape [batch_size, sequence_length]. Image_mask and action_mask also serve similar functions but for image and action respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1116-1136",
+ "content": " elif isinstance(layer, nn.LayerNorm):\n weight_init_(layer, 'Constant', value=1)\n def forward(\n self,\n text_ids, #8,36\n action_feat, #8,5,2048\n image_feat, #8,37,2048\n image_loc, #8,37,5\n token_type_ids=None, #8,36\n text_mask=None, #8,36\n image_mask=None, #8,37\n action_mask=None, #8,5\n ):\n \"\"\"\n text_ids: input text ids. Shape: [batch_size, seqence_length]\n action_feat: input action feature. Shape: [batch_size, action_length, action_feature_dim]\n image_feat: input image feature. Shape: [batch_size, region_length+1, image_feature_dim]], add 1 for image global feature.\n image_loc: input region location. Shape: [batch_size, region_length+1, region_location_dim], add 1 for image global feature location.\n token_type_ids: segment ids of each video clip. Shape: [batch_size, seqence_length]\n text_mask: text mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, seqence_length]"
+ },
+ {
+ "comment": "This code is a function that takes in text IDs, action feature, image feature, image location, token type IDs, text mask, image mask, and action mask as inputs. It uses the BERT model to process these inputs and returns prediction scores for each input (text, vision, action) and sequence relationship score.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/actbert.py\":1137-1157",
+ "content": " image_mask: image mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, region_length]\n action_mask: action mask, 1 for real tokens and 0 for padding tokens. Shape: [batch_size, action_length]\n \"\"\"\n sequence_output_t, sequence_output_v, sequence_output_a, \\\n pooled_output_t, pooled_output_v, pooled_output_a = self.bert(\n text_ids,\n action_feat,\n image_feat,\n image_loc,\n token_type_ids,\n text_mask,\n image_mask,\n action_mask,\n output_all_encoded_layers=False,\n )\n prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score = self.cls(\n sequence_output_t, sequence_output_v, sequence_output_a,\n pooled_output_t, pooled_output_v, pooled_output_a)\n return prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1aa47eef-6ab0-4aee-91fd-da4e1ad068c0.json b/docs/doc/1aa47eef-6ab0-4aee-91fd-da4e1ad068c0.json
new file mode 100644
index 000000000..958c2a7b0
--- /dev/null
+++ b/docs/doc/1aa47eef-6ab0-4aee-91fd-da4e1ad068c0.json
@@ -0,0 +1,10 @@
+{
+ "summary": "Copyright notice, Apache License v2.0, software distributed as is without warranties or conditions. Imports registry and utils modules, defines build_metric function that builds metric using provided configuration.",
+ "details": [
+ {
+ "comment": "Copyright notice, Apache License v2.0, software distributed as is without warranties or conditions. Imports registry and utils modules, defines build_metric function that builds metric using provided configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/metrics/build.py\":0-19",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import METRIC\nfrom ..utils import build\ndef build_metric(cfg):\n return build(cfg, METRIC)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1b7cfb16-3135-4285-88ab-77f1b90d4dd9.json b/docs/doc/1b7cfb16-3135-4285-88ab-77f1b90d4dd9.json
new file mode 100644
index 000000000..40310f681
--- /dev/null
+++ b/docs/doc/1b7cfb16-3135-4285-88ab-77f1b90d4dd9.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code sets up PaddleVideo on Linux, supports multi-card training and testing with PaddlePaddle. It provides log format for phases, resumes sessions, fine-tunes with pretrained params and best accuracy achieved. The code launches PaddleVideo in distributed mode with 4 GPUs, tests, exports model, introduces `use_gpu` parameter, and benchmark results are available in benchmark document.",
+ "details": [
+ {
+ "comment": "This code provides instructions for setting up the environment, preparing data using the PaddleVideo library, and explains its supported functions. It also mentions that it only supports Linux operation systems with GPU environments and gives an example of how to run the library. The code outlines the default destination folders for output, log files, and inference files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/usage.md\":0-27",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../zh-CN/usage.md) | English\n# Usage\n---\nPlease refer to [installation documents](./install.md) to prepare the enviroment, and follow the steps mentioned in the [data preparation documents](./dataset/) to construct dataset, we will take you through the basic functions supported by PaddleVideo, all of it takes the ucf101 dataset with frame format as example.\nPaddleVideo only support linux operation system and GPU running time environment now.\nDefault detination folder of PaddleVideo files. running the [example config](../../configs/example.yaml) as example.\n```\nPaddleVideo\n \u251c\u2500\u2500 paddlevideo\n \u251c\u2500\u2500 ... #other source codes\n \u251c\u2500\u2500 output #ouput destination\n | \u251c\u2500\u2500 example\n | | \u251c\u2500\u2500 example_best.pdparams #path_to_weights\n | | \u2514\u2500\u2500 ... \n | \u2514\u2500\u2500 ... \n \u251c\u2500\u2500 log #log file destination.\n | \u251c\u2500\u2500 worker.0\n | \u251c\u2500\u2500 worker.1\n | \u2514\u2500\u2500 ... \n \u2514\u2500\u2500 inference #inference files destination.\n \u251c\u2500\u2500 .pdiparams file\n \u251c\u2500\u2500 .pdimodel file\n \u2514\u2500\u2500 .pdiparmas.info file"
+ },
+ {
+ "comment": "This code demonstrates how to train and test a model using PaddlePaddle, a popular deep learning framework. The training process involves running multi-card training scripts or tests by executing the `paddle.distributed.launch` command with appropriate arguments such as GPU selection, script path, and optional configuration file. The configuration file allows for flexible updates like changing batch sizes on the fly. After starting the training, log files are generated for tracking progress and analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/usage.md\":28-70",
+ "content": "```\n\n## 1. Train and Test\nStart running multi-cards training scripts or test scripts by `paddle.distributed.launch`, or run the `run.sh` directly.\n```bash\nsh run.sh\n```\nWe put all the start commands in advanced in the ```run.sh```, please uncomment the selected one to run.\n\n### 1.1 Train\nSwitch `--validate` on to validating while training.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n --validate \\\n -c ./configs/example.yaml\n```\nIndicating `-c` to set configuration, and one can flexible add `-o` in the script to update it.\n```bash\npython -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --validate \\\n -o DATASET.batch_size=16\n```\nIndicating `-o DATASET.batch_size=16` can update batch size to 16, please refer to [configuration](tutorials/config.md#config-yaml-details) for more information.\nAfter starting training, log files will generated, "
+ },
+ {
+ "comment": "The code shows log output format for training and validation phases, including time, epoch, batch ID, metrics, elapse time (execution time), and ips (instances per second). It also displays the best accuracy achieved during training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/usage.md\":70-88",
+ "content": "and its format is shown as below, it will output to both the screen and files. Default destination of log is under the `.log/` folder, and stored in the files named like `worker.0`, `worker.1` ...\n[train phase] current time, current epoch/ total epoch, batch id, metrics, elapse time, ips, etc.:\n [12/28 17:31:26] epoch:[ 1/80 ] train step:0 loss: 0.04656 lr: 0.000100 top1: 1.00000 top5: 1.00000 elapse: 0.326 reader: 0.001s ips: 98.22489 instance/sec.\n[eval phase] current time, current epoch/ total epoch, batch id, metrics, elapse time, ips, etc.:\n [12/28 17:31:32] epoch:[ 80/80 ] val step:0 loss: 0.20538 top1: 0.88281 top5: 0.99219 elapse: 1.589 reader: 0.000s ips: 20.14003 instance/sec.\n[epoch end] current time, metrics, elapse time, ips, etc.\n [12/28 17:31:38] END epoch:80 val loss_avg: 0.52208 top1_avg: 0.84398 top5_avg: 0.97393 elapse_avg: 0.234 reader_avg: 0.000 elapse_sum: 7.021s ips: 136.73686 instance/sec.\n[the best Acc] \n [12/28 17:28:42] Already save the best model (top1 acc)0.8494"
+ },
+ {
+ "comment": "The code provides instructions on how to use PaddleVideo for three different tasks: resuming a training session, finetuning with pretrained parameters, and testing. In the resume task, the user should indicate \"-o resume_epoch\" to continue from a specific epoch, while in finetuning, \"--weights\" is used to load pretrained parameters. The test mode is activated using \"--test\". PaddleVideo will not load unmatched parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/usage.md\":90-131",
+ "content": "\n### 1.2 Resume\nIndicate `-o resume_epoch` to resume, It will training from ```resume_epoch``` epoch, PaddleVideo will auto load optimizers parameters and checkpoints from `./output` folder, as it is the default output destination.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --validate \\\n -o resume_epoch=5\n```\n\n### 1.3 Finetune\nIndicate `--weights` to load pretrained parameters, PaddleVideo will auto treat it as a finetune mission.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --validate \\\n --weights=./outputs/example/path_to_weights\n```\nNote: PaddleVideo will NOT load shape unmatched parameters.\n\n### 1.4 Test\nSwitch `--test` on to start test mode, and indicate `--weights` to load pretrained model.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3"
+ },
+ {
+ "comment": "This code is launching PaddleVideo in distributed mode with four GPUs, running the main.py script with a specified configuration file, and performing testing using weights from a particular path. Then it exports the model for inference by specifying the configuration file, pretrained weights, and output directory. Lastly, it uses the PaddleInference engine to infer a video using the exported model files, input video file, and optional TensorRT acceleration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/usage.md\":133-173",
+ "content": "python3 -m paddle.distributed.launch \\\n --gpus=\"0,1,2,3\" \\\n main.py \\\n -c ./configs/example.yaml \\\n --test \\\n --weights=./output/example/path_to_weights\n```\n\n## 2. Infer\nFirst, export model.\nIndicate `-c` to set configuration, `-p` to load pretrained model, `-o` to set inference files destination.\n```bash\npython tools/export_model.py \\\n -c ./configs/example.yaml \\\n -p ./output/example/path_to_weights \\\n -o ./inference\n```\nIt will generate `model_name.pdmodel` , `model_name.pdiparams` and `model_name.pdiparames.info`.\nSecond, start PaddleInference engine to infer a video.\n```bash\npython tools/predict.py \\\n --input_file \"data/example.avi\" \\\n --model_file \"./inference/example.pdmodel\" \\\n --params_file \"./inference/example.pdiparams\" \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nAttributes:\n+ `input_file`: input file path or input directory, which contains input files(s).\n+ `model_file`: pdmodel file path.\n+ `params_file`: pdiparams file path.\n+ `use_tensorrt`: use tensorrt to acclerate or not, default: False."
+ },
+ {
+ "comment": "This code snippet is referring to the `use_gpu` parameter in PaddleVideo, which enables or disables GPU usage for inferencing. The default setting is set to True and benchmark results are available in the [benchmark](./benchmark.md) document.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/usage.md\":174-176",
+ "content": "+ `use_gpu`: use gpu to infer or not, default: True.\nbenchmark results are shown in th [benchmark](./benchmark.md)."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1ce4b7d2-cf40-4f79-94ce-2aa2e8d987c6.json b/docs/doc/1ce4b7d2-cf40-4f79-94ce-2aa2e8d987c6.json
new file mode 100644
index 000000000..19924d59d
--- /dev/null
+++ b/docs/doc/1ce4b7d2-cf40-4f79-94ce-2aa2e8d987c6.json
@@ -0,0 +1,45 @@
+{
+ "summary": "The PP-TimeSformer model is an enhanced version of TimeSformer for video recognition tasks, trained on Kinetics-400 dataset and supports multi-GPU. It uses PaddleVideo with Vision Transformer backbone for testing and exports PP-TimeSformer for prediction using a specific config file.",
+ "details": [
+ {
+ "comment": "This code describes the PP-TimeSformer video classification model, an improved version of the TimeSformer model. It outlines the training, testing, and inference processes, as well as providing data preparation instructions for Kinetics-400 dataset. The table shows the accuracy of different versions of the model on Kinetics-400 dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":0-28",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/recognition/pp-timesformer.md) | English\n# TimeSformer Video Classification Model\n## Content\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Inference](#Inference)\n- [Reference](#Reference)\n## Introduction\nWe have improved the [TimeSformer model](./timesformer.md) and obtained a more accurate 2D practical video classification model **PP-TimeSformer**. Without increasing the amount of parameters and calculations, the accuracy on the UCF-101, Kinetics-400 and other data sets significantly exceeds the original version. The accuracy on the Kinetics-400 data set is shown in the table below.\n| Version | Top1 |\n| :------ | :----: |\n| Ours ([swa](#refer-anchor-1)+distill+16frame) | 79.44 |\n| Ours ([swa](#refer-anchor-1)+distill) | 78.87 |\n| Ours ([swa](#refer-anchor-1)) | **78.61** |\n| [mmaction2](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/timesformer#kinetics-400) | 77.92 |\n## Data\nK400 data download and preparation please refer to [Kinetics-400 data preparation](../../dataset/k400.md)"
+ },
+ {
+ "comment": "This code snippet explains how to download and prepare data for training a video recognition model. It mentions the required data sets, pre-trained models, and the specific commands to download and configure them.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":30-57",
+ "content": "UCF101 data download and preparation please refer to [UCF-101 data preparation](../../dataset/ucf101.md)\n## Train\n### Kinetics-400 data set training\n#### Download and add pre-trained models\n1. Download the image pre-training model [ViT_base_patch16_224_miil_21k.pdparams](https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ViT_base_patch16_224_pretrained.pdparams) as Backbone initialization parameters, or download through wget command\n ```bash\n wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ViT_base_patch16_224_pretrained.pdparams\n ```\n2. Open `PaddleVideo/configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml`, and fill in the downloaded weight storage path below `pretrained:`\n ```yaml\n MODEL:\n framework: \"RecognizerTransformer\"\n backbone:\n name: \"VisionTransformer_tweaks\"\n pretrained: fill in the path here\n ```\n#### Start training\n- The Kinetics400 data set uses 8 cards for training, and the start command of the training method is as follows:"
+ },
+ {
+ "comment": "This code runs PaddlePaddle's Timesformer model for video recognition using a specific configuration file. It uses multiple GPUs and supports AMP mixed-precision training for faster processing. The script is customizable, allowing you to train or test on different datasets by modifying the configuration file's name.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":59-74",
+ "content": " ```bash\n # videos data format\n python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptimesformer main.py --validate -c configs/recognition/ pptimesformer/pptimesformer_k400_videos.yaml\n ```\n- Turn on amp mixed-precision training to speed up the training process. The training start command is as follows:\n ```bash\n export FLAGS_conv_workspace_size_limit=800 # MB\n export FLAGS_cudnn_exhaustive_search=1\n export FLAGS_cudnn_batchnorm_spatial_persistent=1\n # videos data format\n python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptimesformer main.py --amp --validate -c configs /recognition/pptimesformer/pptimesformer_k400_videos.yaml\n ```\n- In addition, you can customize and modify the parameter configuration to achieve the purpose of training/testing on different data sets. It is recommended that the naming method of the configuration file is `model_dataset name_file format_data format_sampling method.yaml` , Please refer to [config](../../tutorials/config.md) for parameter usage."
+ },
+ {
+ "comment": "The PP-TimeSformer model is tested during training, and the best test accuracy can be found in the log with keyword \"best\". However, the verification index recorded in the log may not represent the final test score, so a separate testing script should be used to obtain the accurate result. Two such scripts are provided for 8-frames and 16-frames testing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":77-91",
+ "content": "## Test\n- The PP-TimeSformer model is verified synchronously during training. You can find the keyword `best` in the training log to obtain the model test accuracy. The log example is as follows:\n ```\n Already save the best model (top1 acc)0.7258\n ```\n- Because the sampling method of the PP-TimeSformer model test mode is a slightly slower but higher accuracy **UniformCrop**, which is different from the **RandomCrop** used in the verification mode during the training process, so the verification index recorded in the training log` topk Acc` does not represent the final test score, so after the training is completed, you can use the test mode to test the best model to obtain the final index. The command is as follows:\n ```bash\n # 8-frames testing script\n python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptimesformer main.py --test -c configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml -w \"output/ppTimeSformer/ppTimeSformer_best.pdparams\"\n # 16-frames testing script"
+ },
+ {
+ "comment": "This code is launching the PaddleVideo model for testing using Vision Transformer backbone with UniformCrop sampling method and 8 segments. It's running on multiple GPUs and using a specific configuration file, yaml, to set parameters like backbone, sampling method, number of segments, target size, and checkpoint file. The resulting test indicators are presented in tabular format for Kinetics-400 validation dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":92-107",
+ "content": " python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_pptimesformer main.py --test \\\n -c configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml \\\n -o MODEL.backbone.num_seg=16 \\\n -o MODEL.runtime_cfg.test.num_seg=16 \\\n -o PIPELINE.test.decode.num_seg=16 \\\n -o PIPELINE.test.sample.num_seg=16 \\\n -w \"data/ppTimeSformer_k400_16f_distill.pdparams\"\n ```\n When the test configuration uses the following parameters, the test indicators on the validation data set of Kinetics-400 are as follows:\n | backbone | Sampling method | num_seg | target_size | Top-1 | checkpoints |\n | :----------------: | :-------------: | :-----: | :---------: | :---- | :----------------------------------------------------------: |\n | Vision Transformer | UniformCrop | 8 | 224 | 78.61 | [ppTimeSformer_k400_8f.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ppTimeSformer_k400_8f.pdparams) |\n | Vision Transformer | UniformCrop | 8 | 224 | "
+ },
+ {
+ "comment": "The code snippet is exporting the PP-TimeSformer model for video recognition. The model uses linspace sampling strategy, uniformly generating sparse sampling points in time and space to create one clip from a single video. The command uses Python script `export_model.py`, with config file `configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml` and model parameters file `data/ppTimeSformer_k400_8f.pdparams`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":107-119",
+ "content": "78.87 | [ppTimeSformer_k400_8f_distill.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ppTimeSformer_k400_8f_distill.pdparams) |\n | Vision Transformer | UniformCrop | 16 | 224 | 79.44 | [ppTimeSformer_k400_16f_distill.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ppTimeSformer_k400_16f_distill.pdparams) |\n- During the test, the PP-TimeSformer video sampling strategy is to use linspace sampling: in time sequence, from the first frame to the last frame of the video sequence to be sampled, `num_seg` sparse sampling points (including endpoints) are uniformly generated; spatially , Select 3 areas to sample at both ends of the long side and the middle position (left, middle, right or top, middle, and bottom). A total of 1 clip is sampled for 1 video.\n## Inference\n### Export inference model\n```bash\npython3.7 tools/export_model.py -c configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml \\\n -p data/ppTimeSformer_k400_8f.pdparams \\"
+ },
+ {
+ "comment": "This code is for inference using PaddlePaddle's ppTimeSformer model. The command generates the required model structure and weight files for prediction and then executes the predict.py script with the given input file, configuration, model files, and parameters. It displays the top-1 class and score for the video file provided, trained on Kinetics 400 dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":120-146",
+ "content": " -o inference/ppTimeSformer\n```\nThe above command will generate the model structure file `ppTimeSformer.pdmodel` and the model weight file `ppTimeSformer.pdiparams` required for prediction.\n- For the meaning of each parameter, please refer to [Model Reasoning Method](../../start.md#2-Model Reasoning)\n### Use predictive engine inference\n```bash\npython3.7 tools/predict.py --input_file data/example.avi \\\n --config configs/recognition/pptimesformer/pptimesformer_k400_videos.yaml \\\n --model_file inference/ppTimeSformer/ppTimeSformer.pdmodel \\\n --params_file inference/ppTimeSformer/ppTimeSformer.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nThe output example is as follows:\n```\nCurrent video file: data/example.avi\n top-1 class: 5\n top-1 score: 0.9997474551200867\n```\nIt can be seen that using the ppTimeSformer model trained on Ki"
+ },
+ {
+ "comment": "This code snippet is discussing the prediction of a category name using the PP-Timesformer model, specifically for predicting the content of `data/example.avi`. The predicted category id is 5 and its corresponding category name is \"archery\". This information is derived from the category id and name correspondence table `data/k400/Kinetics-400_label_list.txt`. The code provides references to several related papers which have influenced or been used in this model's development.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/pp-timesformer.md\":146-155",
+ "content": "netics-400 to predict `data/example.avi`, the output top1 category id is `5`, and the confidence is 0.99. By referring to the category id and name correspondence table `data/k400/Kinetics-400_label_list.txt`, it can be known that the predicted category name is `archery`.\n## Reference\n- [Is Space-TimeAttention All You Need for Video Understanding?](https://arxiv.org/pdf/2102.05095.pdf), Gedas Bertasius, Heng Wang, Lorenzo Torresani\n- [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531), Geoffrey Hinton, Oriol Vinyals, Jeff Dean\n\n- [Averaging Weights Leads to Wider Optima and Better Generalization](https://arxiv.org/abs/1803.05407v3), Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov\n- [ImageNet-21K Pretraining for the Masses](https://arxiv.org/pdf/2104.10972v4.pdf), Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1d6e284b-2445-4be9-9b0a-d6056e4139c4.json b/docs/doc/1d6e284b-2445-4be9-9b0a-d6056e4139c4.json
new file mode 100644
index 000000000..c71beb6db
--- /dev/null
+++ b/docs/doc/1d6e284b-2445-4be9-9b0a-d6056e4139c4.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code is part of the PaddleVideo framework's VideoTag application, which imports libraries, sets up a logger, and handles YAML configuration. It logs each key-value pair in the config file, separated by dashed lines.",
+ "details": [
+ {
+ "comment": "This code snippet is part of the PaddleVideo framework's VideoTag application. It imports yaml and AttrDict from utility, sets up a logger for logging messages, defines four configuration section strings, and provides two functions: parse_config() to load config files into an AttrDict object and create_attr_dict() to create an AttrDict object with the specified attributes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/utils/config_utils.py\":0-36",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport yaml\nfrom .utility import AttrDict\nimport logging\nlogger = logging.getLogger(__name__)\nCONFIG_SECS = [\n 'train',\n 'valid',\n 'test',\n 'infer',\n]\ndef parse_config(cfg_file):\n \"\"\"Load a config file into AttrDict\"\"\"\n import yaml\n with open(cfg_file, 'r') as fopen:\n yaml_config = AttrDict(yaml.load(fopen, Loader=yaml.Loader))\n create_attr_dict(yaml_config)\n return yaml_config\ndef create_attr_dict(yaml_config):"
+ },
+ {
+ "comment": "This code includes three functions. The first function, `config_utils.py`, is for processing the yaml configuration by converting certain types into AttrDicts and evaluating string values. The second function, `merge_configs()`, merges argument dictionaries with pre-existing config section dictionaries. It skips None values and attempts to set new attributes. Finally, the third function, `print_configs()`, prints configuration arguments in a formatted manner.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/utils/config_utils.py\":37-72",
+ "content": " from ast import literal_eval\n for key, value in yaml_config.items():\n if type(value) is dict:\n yaml_config[key] = value = AttrDict(value)\n if isinstance(value, str):\n try:\n value = literal_eval(value)\n except BaseException:\n pass\n if isinstance(value, AttrDict):\n create_attr_dict(yaml_config[key])\n else:\n yaml_config[key] = value\n return\ndef merge_configs(cfg, sec, args_dict):\n assert sec in CONFIG_SECS, \"invalid config section {}\".format(sec)\n sec_dict = getattr(cfg, sec.upper())\n for k, v in args_dict.items():\n if v is None:\n continue\n try:\n if hasattr(sec_dict, k):\n setattr(sec_dict, k, v)\n except:\n pass\n return cfg\ndef print_configs(cfg, mode):\n logger.info(\n \"---------------- {:>5} Arguments ----------------\".format(mode))\n for sec, sec_items in cfg.items():\n logger.info(\"{}:\".format(sec))\n for k, v in sec_items.items():"
+ },
+ {
+ "comment": "The code is logging information for each key-value pair in the configuration file, and then separating each set of logs with a dashed line.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/utils/config_utils.py\":73-74",
+ "content": " logger.info(\" {}:{}\".format(k, v))\n logger.info(\"-------------------------------------------------\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1e0a27ad-0bb8-4301-b8c2-7f1936b08d23.json b/docs/doc/1e0a27ad-0bb8-4301-b8c2-7f1936b08d23.json
new file mode 100644
index 000000000..8b252e429
--- /dev/null
+++ b/docs/doc/1e0a27ad-0bb8-4301-b8c2-7f1936b08d23.json
@@ -0,0 +1,45 @@
+{
+ "summary": "This code accelerates PaddleServing installation with Docker, supports Linux and GPU, simplifies action recognition service deployment, and provides a C++ serving environment setup guide.",
+ "details": [
+ {
+ "comment": "This code introduces the installation process for PaddleServing. It uses Docker to pull a GPU-based docker environment and creates a Serving-based Docker named \"test\". The port 9292 is mapped to access the serving environment, and this setup supports Linux platforms, with Windows currently unsupported.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":0-16",
+ "content": "English | [\u7b80\u4f53\u4e2d\u6587](./readme.md)\n# Model service deployment\n## Introduction\n[Paddle Serving](https://github.com/PaddlePaddle/Serving) aims to help deep learning developers easily deploy online prediction services, support one-click deployment of industrial-grade service capabilities, high concurrency between client and server Efficient communication and support for developing clients in multiple programming languages.\nThis section takes the HTTP prediction service deployment as an example to introduce how to use PaddleServing to deploy the model service in PaddleVideo. Currently, only Linux platform deployment is supported, and Windows platform is not currently supported.\n## Serving installation\nThe Serving official website recommends using docker to install and deploy the Serving environment. First, you need to pull the docker environment and create a Serving-based docker.\n```bash\n# start GPU docker\ndocker pull paddlepaddle/serving:0.7.0-cuda10.2-cudnn7-devel\nnvidia-docker run -p 9292:9292 --name test -dit paddlepaddle/serving:0.7.0-cuda10.2-cudnn7-devel bash"
+ },
+ {
+ "comment": "Code installs necessary packages for PaddlePaddle serving client, app, server (CPU/GPU) and PaddlePaddle (CPU/GPU) in a Docker container using pip. The GPU versions are specified with different CUDA and TensorRT versions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":17-40",
+ "content": "nvidia-docker exec -it test bash\n# start CPU docker\ndocker pull paddlepaddle/serving:0.7.0-devel\ndocker run -p 9292:9292 --name test -dit paddlepaddle/serving:0.7.0-devel bash\ndocker exec -it test bash\n```\nAfter entering docker, you need to install Serving-related python packages.\n```bash\npython3.7 -m pip install paddle-serving-client==0.7.0\npython3.7 -m pip install paddle-serving-app==0.7.0\n#If it is a CPU deployment environment:\npython3.7 -m pip install paddle-serving-server==0.7.0 #CPU\npython3.7 -m pip install paddlepaddle==2.2.0 # CPU\n#If it is a GPU deployment environment\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post102 # GPU with CUDA10.2 + TensorRT6\npython3.7 -m pip install paddlepaddle-gpu==2.2.0 # GPU with CUDA10.2\n#Other GPU environments need to confirm the environment and then choose which one to execute\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post101 # GPU with CUDA10.1 + TensorRT6\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post112 # GPU with CUDA11.2 + TensorRT8"
+ },
+ {
+ "comment": "This code snippet provides instructions for speeding up the installation process and deploying an action recognition service using PaddleServing. It explains how to convert a saved inference model into a Serving model, using PP-TSM as an example.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":41-64",
+ "content": "```\n* If the installation speed is too slow, you can change the source through `-i https://pypi.tuna.tsinghua.edu.cn/simple` to speed up the installation process.\n* For more environment and corresponding installation packages, see: https://github.com/PaddlePaddle/Serving/blob/v0.9.0/doc/Install_Linux_Env_CN.md\n## Action recognition service deployment\n### Model conversion\nWhen using PaddleServing for service deployment, you need to convert the saved inference model into a Serving model. The following uses the PP-TSM model as an example to introduce how to deploy the action recognition service.\n- Download PP-TSM inference model and convert to Serving model:\n ```bash\n # Enter PaddleVideo directory\n cd PaddleVideo\n # Download the inference model and extract it to ./inference\n mkdir ./inference\n pushd ./inference\n wget https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM.zip\n unzip ppTSM.zip\n popd\n # Convert to Serving model\n pushd deploy/cpp_serving\n python3.7 -m paddle_serving_client.convert \\"
+ },
+ {
+ "comment": "The code is specifying the directory, model filename, and parameters filename for a PaddleVideo inference program conversion. It also sets the serving server and client executables to be used after the conversion. The `dirname` parameter holds the storage path of the converted model files. If no specific filenames are provided (model_filename or params_filename), the code defaults to \"None\" which will use default filenames (\"__model__\" and None respectively). The serving server and client executables are specified in the code to be used after the conversion process, allowing the model to be served for inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":65-78",
+ "content": " --dirname ../../inference/ppTSM \\\n --model_filename ppTSM.pdmodel \\\n --params_filename ppTSM.pdiparams \\\n --serving_server ./ppTSM_serving_server \\\n --serving_client ./ppTSM_serving_client\n popd\n ```\n | parameter | type | default value | description |\n | ----------------- | ---- | ------------------ | ------- -------------------------------------------------- --- |\n | `dirname` | str | - | The storage path of the model file to be converted. The program structure file and parameter file are saved in this directory. |\n | `model_filename` | str | None | The name of the file storing the model Inference Program structure that needs to be converted. If set to None, use `__model__` as the default filename |\n | `params_filename` | str | None | File name where all parameters of the model to be converted are stored. It needs to be specified if and only if all model parameters are stored in a single binary file. If the model parameters are stored in separate files, set it to None |\n | `serving_"
+ },
+ {
+ "comment": "The code specifies two paths, \"serving_server\" and \"serving_client\", representing the storage locations for model files and configuration files. After model conversion, it generates two folders with associated file formats in the specified folder. Upon obtaining the model files, modify two specific text files to change `alias_name` under `fetch_var`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":78-93",
+ "content": "server` | str | `\"serving_server\"` | The storage path of the converted model files and configuration files. Default is serving_server |\n | `serving_client` | str | `\"serving_client\"` | The converted client configuration file storage path. Default is serving_client |\n- After the inference model conversion is completed, two folders, `ppTSM_serving_client` and `ppTSM_serving_server` will be generated under the `deploy/cpp_serving` folder, with the following formats:\n ```bash\n PaddleVideo/deploy/cpp_serving\n \u251c\u2500\u2500 ppTSM_serving_client\n \u2502 \u251c\u2500\u2500 serving_client_conf.prototxt\n \u2502 \u2514\u2500\u2500 serving_client_conf.stream.prototxt\n \u2514\u2500\u2500 ppTSM_serving_server\n \u251c\u2500\u2500 ppTSM.pdiparams\n \u251c\u2500\u2500 ppTSM.pdmodel\n \u251c\u2500\u2500 serving_server_conf.prototxt\n \u2514\u2500\u2500 serving_server_conf.stream.prototxt\n ```\n After getting the model file, you need to modify `serving_client_conf.prototxt` under `ppTSM_serving_client` and `serving_server_conf.prototxt` under `ppTSM_serving_server` respectively, and change `alias_name` under `fetch_var` in both files to `outputs`"
+ },
+ {
+ "comment": "This code demonstrates a rename function for compatibility in model deployment. The modified `serving_server_conf.prototxt` shows how to alias the input and output names in the configuration file. This allows different models to be inferred and deployed without modifying the code, only by altering the `alias_name`. The `cpp_serving` directory contains scripts for starting the pipeline service, C++ serving service, and sending prediction requests.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":95-121",
+ "content": " **Remarks**: In order to be compatible with the deployment of different models, Serving provides the function of input and output renaming. In this way, when different models are inferred and deployed, they only need to modify the `alias_name` of the configuration file, and the inference deployment can be completed without modifying the code.\n The modified `serving_server_conf.prototxt` looks like this:\n ```yaml\n feed_var {\n name: \"data_batch_0\"\n alias_name: \"data_batch_0\"\n is_lod_tensor: false\n feed_type: 1\n shape: 8\n shape: 3\n shape: 224\n shape: 224\n }\n fetch_var {\n name: \"linear_2.tmp_1\"\n alias_name: \"outputs\"\n is_lod_tensor: false\n fetch_type: 1\n shape: 400\n }\n ```\n### Service deployment and requests\nThe `cpp_serving` directory contains the code for starting the pipeline service, the C++ serving service and sending the prediction request, including:\n ```bash\n run_cpp_serving.sh # Start the script on the C++ serving server side\n pipeline_http_client.py # The script on the client side to send data and get the prediction results"
+ },
+ {
+ "comment": "This code provides instructions for setting up and running a C++ serving environment for PaddleVideo. It explains how to navigate to the deployment directory, start the service, send requests using serving_client.py, and obtain the model prediction results. If an error occurs during execution, it will display the corresponding log information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":122-151",
+ "content": " paddle_env_install.sh # Install C++ serving environment script\n preprocess_ops.py # file to store preprocessing functions\n ```\n#### C++ Serving\n- Go to the working directory:\n ```bash\n cd deploy/cpp_serving\n ```\n- Start the service:\n ```bash\n # Start in the background, the logs printed during the process will be redirected and saved to nohup.txt\n bash run_cpp_serving.sh\n ```\n- Send the request and get the result:\n```bash\npython3.7 serving_client.py \\\n-n PPTSM \\\n-c ./ppTSM_serving_client/serving_client_conf.prototxt \\\n--input_file=../../data/example.avi\n```\nAfter a successful run, the results of the model prediction will be printed in the cmd window, and the results are as follows:\n ```bash\n I0510 04:33:00.110025 37097 naming_service_thread.cpp:202] brpc::policy::ListNamingService(\"127.0.0.1:9993\"): added 1\n I0510 04:33:01.904764 37097 general_model.cpp:490] [client]logid=0,client_cost=1640.96ms,server_cost=1623.21ms.\n {'class_id': '[5]', 'prob': '[0.9907387495040894]'}\n ```\n**If an error is re"
+ },
+ {
+ "comment": "This code provides instructions for installing the necessary environment and resolving an issue where no result is returned or a decoding error occurs due to proxy settings. Users are advised not to set proxies when starting the service and sending requests, and should use the provided commands to close proxies beforehand. The script `paddle_env_install.sh` can be executed to install relevant environment requirements.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/readme_en.md\":151-164",
+ "content": "ported during the process and it shows that libnvinfer.so.6 cannot be found, you can execute the script `paddle_env_install.sh` to install the relevant environment**\n ```bash\n bash paddle_env_install.sh\n ```\n## FAQ\n**Q1**: No result is returned after the request is sent or an output decoding error is prompted\n**A1**: Do not set the proxy when starting the service and sending the request. You can close the proxy before starting the service and sending the request. The command to close the proxy is:\n```\nunset https_proxy\nunset http_proxy\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1e54cef0-a235-46a7-ab45-c9ac47e4b976.json b/docs/doc/1e54cef0-a235-46a7-ab45-c9ac47e4b976.json
new file mode 100644
index 000000000..c768b7482
--- /dev/null
+++ b/docs/doc/1e54cef0-a235-46a7-ab45-c9ac47e4b976.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code introduces a VideoMix operator for data augmentation in image classification tasks, using mixup and cutmix operations with controllable parameters.",
+ "details": [
+ {
+ "comment": "Mixup class implements a mixup operator for PaddleVideo. It takes an alpha value as input and ensures it is greater than 0. The __call__ method takes a batch of images and labels, combines them with random weights determined by the alpha value, and returns the mixed up image batch and label batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/mix.py\":0-33",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass Mixup(object):\n \"\"\"\n Mixup operator.\n Args:\n alpha(float): alpha value.\n \"\"\"\n def __init__(self, alpha=0.2):\n assert alpha > 0., \\\n 'parameter alpha[%f] should > 0.0' % (alpha)\n self.alpha = alpha\n def __call__(self, batch):\n imgs, labels = list(zip(*batch))\n imgs = np.array(imgs)"
+ },
+ {
+ "comment": "The code defines a Cutmix class for a mixup operator. It takes an alpha value as input, and randomly generates new images by cutting out a part of the original image and pasting it on top of another image, with alpha value determining the ratio of the two. The function rand_bbox is used to determine the dimensions and location of the cutout box.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/mix.py\":34-69",
+ "content": " labels = np.array(labels)\n bs = len(batch)\n idx = np.random.permutation(bs)\n lam = np.random.beta(self.alpha, self.alpha)\n lams = np.array([lam] * bs, dtype=np.float32)\n imgs = lam * imgs + (1 - lam) * imgs[idx]\n return list(zip(imgs, labels, labels[idx], lams))\n@PIPELINES.register()\nclass Cutmix(object):\n \"\"\" Cutmix operator\n Args:\n alpha(float): alpha value.\n \"\"\"\n def __init__(self, alpha=0.2):\n assert alpha > 0., \\\n 'parameter alpha[%f] should > 0.0' % (alpha)\n self.alpha = alpha\n def rand_bbox(self, size, lam):\n \"\"\" rand_bbox \"\"\"\n w = size[2]\n h = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(w * cut_rat)\n cut_h = np.int(h * cut_rat)\n # uniform\n cx = np.random.randint(w)\n cy = np.random.randint(h)\n bbx1 = np.clip(cx - cut_w // 2, 0, w)\n bby1 = np.clip(cy - cut_h // 2, 0, h)\n bbx2 = np.clip(cx + cut_w // 2, 0, w)\n bby2 = np.clip(cy + cut_h // 2, 0, h)"
+ },
+ {
+ "comment": "This code defines a VideoMix operator that performs data augmentation by either mixing or cutting images from different samples in the batch. The mixup_alpha and cutmix_alpha parameters control the degree of blending between samples, while the cutmix_prob parameter determines the probability of applying the cutmix operation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/mix.py\":71-102",
+ "content": " return bbx1, bby1, bbx2, bby2\n def __call__(self, batch):\n imgs, labels = list(zip(*batch))\n imgs = np.array(imgs)\n labels = np.array(labels)\n bs = len(batch)\n idx = np.random.permutation(bs)\n lam = np.random.beta(self.alpha, self.alpha)\n bbx1, bby1, bbx2, bby2 = self.rand_bbox(imgs.shape, lam)\n imgs[:, :, bbx1:bbx2, bby1:bby2] = imgs[idx, :, bbx1:bbx2, bby1:bby2]\n lam = 1 - (float(bbx2 - bbx1) * (bby2 - bby1) /\n (imgs.shape[-2] * imgs.shape[-1]))\n lams = np.array([lam] * bs, dtype=np.float32)\n return list(zip(imgs, labels, labels[idx], lams))\n@PIPELINES.register()\nclass VideoMix(object):\n \"\"\"\n VideoMix operator.\n Args:\n cutmix_prob(float): prob choose cutmix\n mixup_alpha(float): alpha for mixup aug\n cutmix_alpha(float): alpha for cutmix aug\n \"\"\"\n def __init__(self, cutmix_prob=0.5, mixup_alpha=0.2, cutmix_alpha=1.0):\n assert cutmix_prob > 0., \\\n 'parameter cutmix_prob[%f] should > 0.0' % (cutmix_prob)"
+ },
+ {
+ "comment": "This code asserts that mixup_alpha and cutmix_alpha are greater than 0.0, sets the cutmix_prob, creates Mixup and Cutmix objects with the provided alphas, and defines a __call__ method to randomly choose between applying either Mixup or Cutmix to the batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/mix.py\":103-115",
+ "content": " assert mixup_alpha > 0., \\\n 'parameter mixup_alpha[%f] should > 0.0' % (mixup_alpha)\n assert cutmix_alpha > 0., \\\n 'parameter cutmix_alpha[%f] should > 0.0' % (cutmix_alpha)\n self.cutmix_prob = cutmix_prob\n self.mixup = Mixup(mixup_alpha)\n self.cutmix = Cutmix(cutmix_alpha)\n def __call__(self, batch):\n if np.random.random() < self.cutmix_prob:\n return self.cutmix(batch)\n else:\n return self.mixup(batch)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1e9290c5-1d3a-441b-9232-fa75817c48a4.json b/docs/doc/1e9290c5-1d3a-441b-9232-fa75817c48a4.json
new file mode 100644
index 000000000..1fa0c47d0
--- /dev/null
+++ b/docs/doc/1e9290c5-1d3a-441b-9232-fa75817c48a4.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The `weight_init_` function initializes layer weights in PaddlePaddle with custom functions, supporting various initialization types such as Xavier and Uniform.",
+ "details": [
+ {
+ "comment": "This function, `weight_init_`, initializes the weights of a PaddlePaddle layer with user-defined functions. The function takes in a layer, an initialization function, optional weight and bias names, and additional keyword arguments. It performs an in-place parameter initialization and supports various types of initialization functions such as Xavier, Uniform, and others.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/weight_init.py\":0-35",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport numpy as np\nimport paddle.nn.initializer as init\ndef weight_init_(layer,\n func,\n weight_name=None,\n bias_name=None,\n bias_value=0.0,\n **kwargs):\n \"\"\"\n In-place params init function.\n Usage:\n .. code-block:: python\n import paddle\n import numpy as np\n data = np.ones([3, 4], dtype='float32')\n linear = paddle.nn.Linear(4, 4)"
+ },
+ {
+ "comment": "This code initializes the weights and biases of a neural network layer using the PaddlePaddle framework. It checks if the layer has weight and bias attributes, then applies weight initialization functions and potentially overrides their names.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/weight_init.py\":36-54",
+ "content": " input = paddle.to_tensor(data)\n print(linear.weight)\n linear(input)\n weight_init_(linear, 'Normal', 'fc_w0', 'fc_b0', std=0.01, mean=0.1)\n print(linear.weight)\n \"\"\"\n if hasattr(layer, 'weight') and layer.weight is not None:\n getattr(init, func)(**kwargs)(layer.weight)\n if weight_name is not None:\n # override weight name\n layer.weight.name = weight_name\n if hasattr(layer, 'bias') and layer.bias is not None:\n init.Constant(bias_value)(layer.bias)\n if bias_name is not None:\n # override bias name\n layer.bias.name = bias_name"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1ef7a7d7-f557-4529-95b5-1a41b11cb022.json b/docs/doc/1ef7a7d7-f557-4529-95b5-1a41b11cb022.json
new file mode 100644
index 000000000..2a69eeb47
--- /dev/null
+++ b/docs/doc/1ef7a7d7-f557-4529-95b5-1a41b11cb022.json
@@ -0,0 +1,150 @@
+{
+ "summary": "The code defines ResNetSlowFast and SlowFast models for video recognition and computer vision tasks, respectively, with separate pathways for slow and fast processing using 3D convolutional layers and multi-pathway models.",
+ "details": [
+ {
+ "comment": "Copyright notice and license information for the code. Imports necessary modules, defines function to get convolutional layer initialization parameters, and a function to set batch normalization layer parameters. No actual model or functionality defined yet.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":0-32",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import KaimingNormal\nfrom ..registry import BACKBONES\nfrom paddlevideo.utils.multigrid import get_norm\nimport sys\nimport numpy as np\nimport paddle.distributed as dist\n# seed random seed\npaddle.framework.seed(0)\n# get init parameters for conv layer\ndef get_conv_init(fan_out):\n return KaimingNormal(fan_in=fan_out)\ndef get_bn_param_attr(bn_weight=1.0, coeff=0.0):"
+ },
+ {
+ "comment": "This code defines a BottleneckTransform class in PaddleVideo for video models. It performs Tx1x1, 1x3x3, 1x1x1 transformations with variable temporal kernel sizes. The constructor takes in arguments like dim_in, dim_out, temp_kernel_size, stride, and more to configure the transformation layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":33-65",
+ "content": " param_attr = paddle.ParamAttr(\n initializer=paddle.nn.initializer.Constant(bn_weight),\n regularizer=paddle.regularizer.L2Decay(coeff))\n return param_attr\n\"\"\"Video models.\"\"\"\nclass BottleneckTransform(paddle.nn.Layer):\n \"\"\"\n Bottleneck transformation: Tx1x1, 1x3x3, 1x1x1, where T is the size of\n temporal kernel.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck."
+ },
+ {
+ "comment": "This code defines a class called BottleneckTransform with parameters such as dim_in, dim_out, stride, dim_inner, num_groups, and dilation. It also has attributes like _inplace_relu, _eps, and norm_module for various operations and settings. The _construct method is used to initialize the class with these parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":66-86",
+ "content": " dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): if True, calculate the relu on the original\n input without allocating new memory.\n eps (float): epsilon for batch norm.\n dilation (int): size of dilation.\n \"\"\"\n super(BottleneckTransform, self).__init__()\n self.temp_kernel_size = temp_kernel_size\n self._inplace_relu = inplace_relu\n self._eps = eps\n self._stride_1x1 = stride_1x1\n self.norm_module = norm_module\n self._construct(dim_in, dim_out, stride, dim_inner, num_groups,\n dilation)\n def _construct(self, dim_in, dim_out, stride, dim_inner, num_groups,"
+ },
+ {
+ "comment": "Defines a Conv3D layer for the ResNet_SlowFast backbone, with specified dimensions and stride. Initializes Conv3D weights using get_conv_init function and includes batch normalization (BN) and ReLU activation layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":87-112",
+ "content": " dilation):\n str1x1, str3x3 = (stride, 1) if self._stride_1x1 else (1, stride)\n fan = (dim_inner) * (self.temp_kernel_size * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.a = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_inner,\n kernel_size=[self.temp_kernel_size, 1, 1],\n stride=[1, str1x1, str1x1],\n padding=[int(self.temp_kernel_size // 2), 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.a_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x3x3, BN, ReLU.\n fan = (dim_inner) * (1 * 3 * 3)\n initializer_tmp = get_conv_init(fan)\n self.b = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_inner,"
+ },
+ {
+ "comment": "This code defines a 3D convolutional layer with specific kernel sizes, strides, padding, and grouping. It also includes batch normalization layers for the intermediate and output features. The initializer functions are used to set the weights of each layer, with different initializers for different layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":113-138",
+ "content": " kernel_size=[1, 3, 3],\n stride=[1, str3x3, str3x3],\n padding=[0, dilation, dilation],\n groups=num_groups,\n dilation=[1, dilation, dilation],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.b_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x1x1, BN.\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.c = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_out,\n kernel_size=[1, 1, 1],\n stride=[1, 1, 1],\n padding=[0, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.c_bn = self.norm_module(\n num_features=dim_out,"
+ },
+ {
+ "comment": "ResNetSlowFast forward function performs convolutions and Batch Normalization for each branch (2a, 2b, 2c), then applies ReLU activation. ResBlock is a layer implementing residual blocks with specified dimensions, stride, inner dimension, groups, dilation, and normalization method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":139-179",
+ "content": " epsilon=self._eps,\n weight_attr=get_bn_param_attr(bn_weight=0.0),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n # Branch2a.\n x = self.a(x)\n x = self.a_bn(x)\n x = F.relu(x)\n # Branch2b.\n x = self.b(x)\n x = self.b_bn(x)\n x = F.relu(x)\n # Branch2c\n x = self.c(x)\n x = self.c_bn(x)\n return x\nclass ResBlock(paddle.nn.Layer):\n \"\"\"\n Residual block.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups=1,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n ResBlock class constructs redisual blocks. More details can be found in:\n Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\n \"Deep residual learning for image recognition.\""
+ },
+ {
+ "comment": "This code defines the arguments for constructing a ResNet bottleneck. It includes parameters for input and output channel dimensions, temporal kernel size, stride, transform function, inner dimension, number of groups for convolution, whether to apply stride to 1x1 or 3x3 conv, inplace_relu flag, and epsilon for batch normalization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":180-197",
+ "content": " https://arxiv.org/abs/1512.03385\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck.\n trans_func (string): transform function to be used to construct the\n bottleneck.\n dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): calculate the relu on the original input\n without allocating new memory.\n eps (float): epsilon for batch norm."
+ },
+ {
+ "comment": "The code defines a ResBlock class, which is a residual block used in deep neural networks. It initializes the block with input and output dimensions, kernel size, stride, inner dimension, number of groups, and skip connection settings. The constructor method _construct creates a 3D convolution layer for the skip connection if there is a change in dimensions or stride.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":198-236",
+ "content": " dilation (int): size of dilation.\n \"\"\"\n super(ResBlock, self).__init__()\n self._inplace_relu = inplace_relu\n self._eps = eps\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n # Use skip connection with projection if dim or res change.\n if (dim_in != dim_out) or (stride != 1):\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.branch1 = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=1,\n stride=[1, stride, stride],"
+ },
+ {
+ "comment": "This code defines a ResNet SlowFast backbone model. It includes convolution layers, batch normalization layers, and BottleneckTransform modules. The forward function checks if the \"branch1\" attribute exists to handle different stages of the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":237-259",
+ "content": " padding=0,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False,\n dilation=1)\n self.branch1_bn = self.norm_module(\n num_features=dim_out,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n self.branch2 = BottleneckTransform(dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation,\n norm_module=self.norm_module)\n def forward(self, x):\n if hasattr(self, \"branch1\"):"
+ },
+ {
+ "comment": "The code defines a ResNet stage for multi-pathway (SlowFast) cases in video recognition. It takes one or more tensors as input and applies branching to separate paths with different kernel sizes. The output is added together, passed through ReLU activation, and returned. This stage supports 1x1 stride option and uses BatchNorm3D for normalization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":260-293",
+ "content": " x1 = self.branch1(x)\n x1 = self.branch1_bn(x1)\n x2 = self.branch2(x)\n x = paddle.add(x=x1, y=x2)\n else:\n x2 = self.branch2(x)\n x = paddle.add(x=x, y=x2)\n x = F.relu(x)\n return x\nclass ResStage(paddle.nn.Layer):\n \"\"\"\n Stage of 3D ResNet. It expects to have one or more tensors as input for\n multi-pathway (SlowFast) cases. More details can be found here:\n Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n stride,\n temp_kernel_sizes,\n num_blocks,\n dim_inner,\n num_groups,\n num_block_temp_kernel,\n dilation,\n stride_1x1=False,\n inplace_relu=True,\n norm_module=paddle.nn.BatchNorm3D):"
+ },
+ {
+ "comment": "The ResStage class constructor takes several lists as arguments to build p streams of pathways, controlling input and output dimensions, temporal kernel sizes, strides, and block numbers for each pathway.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":294-311",
+ "content": " \"\"\"\n The `__init__` method of any subclass should also contain these arguments.\n ResStage builds p streams, where p can be greater or equal to one.\n Args:\n dim_in (list): list of p the channel dimensions of the input.\n Different channel dimensions control the input dimension of\n different pathways.\n dim_out (list): list of p the channel dimensions of the output.\n Different channel dimensions control the input dimension of\n different pathways.\n temp_kernel_sizes (list): list of the p temporal kernel sizes of the\n convolution in the bottleneck. Different temp_kernel_sizes\n control different pathway.\n stride (list): list of the p strides of the bottleneck. Different\n stride control different pathway.\n num_blocks (list): list of p numbers of blocks for each of the\n pathway.\n dim_inner (list): list of the p inner channel dimensions of the"
+ },
+ {
+ "comment": "This code defines a ResStage class for a residual block. It takes input dimensions and channel dimensions as parameters, and initializes the number of blocks and temporal kernel sizes based on these inputs. The code also ensures that the provided number of block temporary kernel sizes does not exceed the specified number of blocks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":312-329",
+ "content": " input. Different channel dimensions control the input dimension\n of different pathways.\n num_groups (list): list of number of p groups for the convolution.\n num_groups=1 is for standard ResNet like networks, and\n num_groups>1 is for ResNeXt like networks.\n num_block_temp_kernel (list): extent the temp_kernel_sizes to\n num_block_temp_kernel blocks, then fill temporal kernel size\n of 1 for the rest of the layers.\n dilation (list): size of dilation for each pathway.\n \"\"\"\n super(ResStage, self).__init__()\n assert all((num_block_temp_kernel[i] <= num_blocks[i]\n for i in range(len(temp_kernel_sizes))))\n self.num_blocks = num_blocks\n self.temp_kernel_sizes = [\n (temp_kernel_sizes[i] * num_blocks[i])[:num_block_temp_kernel[i]] +\n [1] * (num_blocks[i] - num_block_temp_kernel[i])\n for i in range(len(temp_kernel_sizes))"
+ },
+ {
+ "comment": "The code initializes a ResNet SlowFast model by creating instances of blocks based on given parameters. It ensures that the input and output dimensions are correctly set, creates the desired number of pathways, and applies the specified norm module. The constructor then iterates over each pathway and block, creating ResBlock instances with appropriate sizes and configurations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":330-371",
+ "content": " ]\n assert (len({\n len(dim_in),\n len(dim_out),\n len(temp_kernel_sizes),\n len(stride),\n len(num_blocks),\n len(dim_inner),\n len(num_groups),\n len(num_block_temp_kernel),\n }) == 1)\n self.num_pathways = len(self.num_blocks)\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n for pathway in range(self.num_pathways):\n for i in range(self.num_blocks[pathway]):\n res_block = ResBlock(\n dim_in[pathway] if i == 0 else dim_out[pathway],\n dim_out[pathway],\n self.temp_kernel_sizes[pathway][i],"
+ },
+ {
+ "comment": "This code defines a ResNet backbone with slow-fast pathways, which includes residual blocks and basic stem modules. The `forward` method processes inputs from each pathway and returns the outputs as a list. It uses getattr to access the correct residual block module for each iteration in each pathway.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":372-403",
+ "content": " stride[pathway] if i == 0 else 1,\n dim_inner[pathway],\n num_groups[pathway],\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation[pathway],\n norm_module=self.norm_module)\n self.add_sublayer(\"pathway{}_res{}\".format(pathway, i),\n res_block)\n def forward(self, inputs):\n output = []\n for pathway in range(self.num_pathways):\n x = inputs[pathway]\n for i in range(self.num_blocks[pathway]):\n m = getattr(self, \"pathway{}_res{}\".format(pathway, i))\n x = m(x)\n output.append(x)\n return output\nclass ResNetBasicStem(paddle.nn.Layer):\n \"\"\"\n ResNe(X)t 3D stem module.\n Performs spatiotemporal Convolution, BN, and Relu following by a\n spatiotemporal pooling.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,"
+ },
+ {
+ "comment": "This code defines a class for ResNet basic stem module with options to specify kernel, stride, padding, and batch normalization. It initializes the Conv3D layer and BatchNorm3D module based on the specified parameters. The constructor also calls the _construct_stem method to further initialize the Conv3D layer and BatchNorm3D module.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":404-431",
+ "content": " kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n super(ResNetBasicStem, self).__init__()\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)\n def _construct_stem(self, dim_in, dim_out):\n fan = (dim_out) * (self.kernel[0] * self.kernel[1] * self.kernel[2])\n initializer_tmp = get_conv_init(fan)\n self._conv = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=self.kernel,\n stride=self.stride,\n padding=self.padding,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = self.norm_module(num_features=dim_out,\n epsilon=self.eps,\n weight_attr=get_bn_param_attr(),"
+ },
+ {
+ "comment": "This code defines a 3D stem module for video input. It consists of convolutional, batch normalization, ReLU, and max pooling operations applied to both slow and fast pathways. The dim_in, dim_out, kernel, stride, padding parameters are used to configure the specifics of these operations. Epsilon (eps) is a small value for numerical stability, and norm_module is the batch normalization module being used.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":432-465",
+ "content": " bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x = self._conv(x)\n x = self._bn(x)\n x = F.relu(x)\n x = F.max_pool3d(x=x,\n kernel_size=[1, 3, 3],\n stride=[1, 2, 2],\n padding=[0, 1, 1],\n data_format=\"NCDHW\")\n return x\nclass VideoModelStem(paddle.nn.Layer):\n \"\"\"\n Video 3D stem module. Provides stem operations of Conv, BN, ReLU, MaxPool\n on input data tensor for slow and fast pathways.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (list): the list of channel dimensions of the inputs.\n dim_out (list): the output dimension of the convolution in the stem\n layer.\n kernel (list): the kernels' size of the convolutions in the stem"
+ },
+ {
+ "comment": "The code defines a VideoModelStem class with parameters for input and output dimensions, temporal kernel size, stride, padding, epsilon for batch norm, and the normalization module. It checks for consistent dimensions and initializes instance variables before calling a constructor method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":466-491",
+ "content": " layers. Temporal kernel size, height kernel size, width kernel\n size in order.\n stride (list): the stride sizes of the convolutions in the stem\n layer. Temporal kernel stride, height kernel size, width kernel\n size in order.\n padding (list): the paddings' sizes of the convolutions in the stem\n layer. Temporal padding size, height padding size, width padding\n size in order.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(VideoModelStem, self).__init__()\n assert (len({\n len(dim_in),\n len(dim_out),\n len(kernel),\n len(stride),\n len(padding),\n }) == 1), \"Input pathway dimensions are not consistent.\"\n self.num_pathways = len(dim_in)\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)"
+ },
+ {
+ "comment": "This code defines a class that constructs a stem for each pathway in ResNet, then applies it to the input tensors. The FuseFastToSlow class fuses information from the Fast pathway to the Slow pathway and returns the fused tensors in order.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":493-517",
+ "content": " def _construct_stem(self, dim_in, dim_out):\n for pathway in range(len(dim_in)):\n stem = ResNetBasicStem(dim_in[pathway], dim_out[pathway],\n self.kernel[pathway], self.stride[pathway],\n self.padding[pathway], self.eps,\n self.norm_module)\n self.add_sublayer(\"pathway{}_stem\".format(pathway), stem)\n def forward(self, x):\n assert (len(x) == self.num_pathways\n ), \"Input tensor does not contain {} pathway\".format(\n self.num_pathways)\n for pathway in range(len(x)):\n m = getattr(self, \"pathway{}_stem\".format(pathway))\n x[pathway] = m(x[pathway])\n return x\nclass FuseFastToSlow(paddle.nn.Layer):\n \"\"\"\n Fuses the information from the Fast pathway to the Slow pathway. Given the\n tensors from Slow pathway and Fast pathway, fuse information from Fast to\n Slow, then return the fused tensors from Slow and Fast pathway in order."
+ },
+ {
+ "comment": "This function initializes the FuseFastToSlow class, which takes in dimensions, fusion parameters, and other options. It sets up a convolutional layer to fuse information from the Fast pathway to the Slow pathway. It uses a specified channel ratio and kernel size for the convolution operation. The epsilon parameter is used for batch normalization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":518-543",
+ "content": " \"\"\"\n def __init__(self,\n dim_in,\n fusion_conv_channel_ratio,\n fusion_kernel,\n alpha,\n fuse_bn_relu=1,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimension of the input.\n fusion_conv_channel_ratio (int): channel ratio for the convolution\n used to fuse from Fast pathway to Slow pathway.\n fusion_kernel (int): kernel size of the convolution used to fuse\n from Fast pathway to Slow pathway.\n alpha (int): the frame rate ratio between the Fast and Slow pathway.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(FuseFastToSlow, self).__init__()\n self.fuse_bn_relu = fuse_bn_relu\n fan = (dim_in * fusion_conv_channel_ratio) * (fusion_kernel * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self._conv_f2s = paddle.nn.Conv3D(\n in_channels=dim_in,"
+ },
+ {
+ "comment": "This code defines a ResNetSlowFast model in PaddlePaddle, which is a variation of the SlowFast network. It includes a fusion convolution layer followed by batch normalization and ReLU activation if the fuse_bn_relu flag is set to True. The forward function performs concatenation of the input features and returns the result. This model is registered under BACKBONES for future use.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":544-571",
+ "content": " out_channels=dim_in * fusion_conv_channel_ratio,\n kernel_size=[fusion_kernel, 1, 1],\n stride=[alpha, 1, 1],\n padding=[fusion_kernel // 2, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = norm_module(num_features=dim_in * fusion_conv_channel_ratio,\n epsilon=eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x_s = x[0]\n x_f = x[1]\n fuse = self._conv_f2s(x_f)\n # TODO: For AVA, set fuse_bn_relu=1, check mAP's improve.\n if self.fuse_bn_relu:\n fuse = self._bn(fuse)\n fuse = F.relu(fuse)\n x_s_fuse = paddle.concat(x=[x_s, fuse], axis=1, name=None)\n return [x_s_fuse, x_f]\n@BACKBONES.register()\nclass ResNetSlowFast(paddle.nn.Layer):\n \"\"\"\n SlowFast model builder for SlowFast network."
+ },
+ {
+ "comment": "This code defines a class ResNetSlowFast, which is a variant of the ResNet architecture with slow and fast paths for video recognition. It takes various parameters such as alpha, beta, bn_norm_type, etc., to build the network. The class also includes methods for initializing the model with the given parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":573-606",
+ "content": " Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(\n self,\n alpha,\n beta,\n bn_norm_type=\"batchnorm\",\n bn_num_splits=1,\n num_pathways=2,\n depth=50,\n num_groups=1,\n input_channel_num=[3, 3],\n width_per_group=64,\n fusion_conv_channel_ratio=2,\n fusion_kernel_sz=7, #5?\n pool_size_ratio=[[1, 1, 1], [1, 1, 1]],\n fuse_bn_relu = 1,\n spatial_strides = [[1, 1], [2, 2], [2, 2], [2, 2]],\n use_pool_af_s2 = 1,\n ):\n \"\"\"\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n super(ResNetSlowFast, self).__init__()\n self.alpha = alpha #8\n self.beta = beta #8\n self.norm_module = get_norm(bn_norm_type, bn_num_splits)\n self.num_pathways = num_pathways\n self.depth = depth"
+ },
+ {
+ "comment": "This code defines a SlowFast model for computer vision tasks. It takes in several parameters including the number of groups, input channel number, and others. The construct_network function builds the SlowFast model with separate pathways for Slow and Fast pathways using different temporal kernels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":607-631",
+ "content": " self.num_groups = num_groups\n self.input_channel_num = input_channel_num\n self.width_per_group = width_per_group\n self.fusion_conv_channel_ratio = fusion_conv_channel_ratio\n self.fusion_kernel_sz = fusion_kernel_sz # NOTE: modify to 7 in 8*8, 5 in old implement\n self.pool_size_ratio = pool_size_ratio\n self.fuse_bn_relu = fuse_bn_relu\n self.spatial_strides = spatial_strides\n self.use_pool_af_s2 = use_pool_af_s2\n self._construct_network()\n def _construct_network(self):\n \"\"\"\n Builds a SlowFast model.\n The first pathway is the Slow pathway\n and the second pathway is the Fast pathway.\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n temp_kernel = [\n [[1], [5]], # conv1 temporal kernel for slow and fast pathway.\n [[1], [3]], # res2 temporal kernel for slow and fast pathway.\n [[1], [3]], # res3 temporal kernel for slow and fast pathway."
+ },
+ {
+ "comment": "The code initializes a SlowFast ResNet backbone model. It defines temporal kernels for res4 and res5 pathways, creates a VideoModelStem layer with specific dimensions and parameters, and a FuseFastToSlow layer for fusion. The code also sets the model stage depth according to the chosen depth (50, 101, or 152).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":632-656",
+ "content": " [[3], [3]], # res4 temporal kernel for slow and fast pathway.\n [[3], [3]],\n ] # res5 temporal kernel for slow and fast pathway.\n self.s1 = VideoModelStem(\n dim_in=self.input_channel_num,\n dim_out=[self.width_per_group, self.width_per_group // self.beta],\n kernel=[temp_kernel[0][0] + [7, 7], temp_kernel[0][1] + [7, 7]],\n stride=[[1, 2, 2]] * 2,\n padding=[\n [temp_kernel[0][0][0] // 2, 3, 3],\n [temp_kernel[0][1][0] // 2, 3, 3],\n ],\n norm_module=self.norm_module)\n self.s1_fuse = FuseFastToSlow(\n dim_in=self.width_per_group // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu)\n # ResNet backbone\n MODEL_STAGE_DEPTH = {50: (3, 4, 6, 3)}\n (d2, d3, d4, d5) = MODEL_STAGE_DEPTH[self.depth]"
+ },
+ {
+ "comment": "This code defines a ResStage, which is a stage in the ResNet SlowFast model. It sets the dimensions and parameters for this stage including input and output widths, kernel sizes, and strides. The code also specifies temporary kernel sizes and spatial strides for this particular stage of the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":658-677",
+ "content": " num_block_temp_kernel = [[3, 3], [4, 4], [6, 6], [3, 3]]\n spatial_dilations = [[1, 1], [1, 1], [1, 1], [1, 1]]\n spatial_strides = self.spatial_strides\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [2, 2]]\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [1, 1]] #TODO:check which value is FAIR's impliment\n out_dim_ratio = self.beta // self.fusion_conv_channel_ratio #4\n dim_inner = self.width_per_group * self.num_groups #64\n self.s2 = ResStage(dim_in=[\n self.width_per_group + self.width_per_group // out_dim_ratio,\n self.width_per_group // self.beta,\n ],\n dim_out=[\n self.width_per_group * 4,\n self.width_per_group * 4 // self.beta,\n ],\n dim_inner=[dim_inner, dim_inner // self.beta],\n temp_kernel_sizes=temp_kernel[1],\n stride=spatial_strides[0],"
+ },
+ {
+ "comment": "The code defines a resnet_slowfast model with two main components: s1 and s3. The s1 component consists of three branches, with the first two having 2x repeat_num_body blocks each, while the third has (2*repeat_num_body + 1) blocks. It also includes norm_module and spatial_dilations for the first branch. The s3 component contains a ResStage layer. The model uses parameters such as width_per_group, out_dim_ratio, dim_inner, repeat_num_body, alpha, beta, fusion_conv_channel_ratio, fusion_kernel_sz, norm_module, and fuse_bn_relu.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":678-703",
+ "content": " num_blocks=[d2] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[0],\n dilation=spatial_dilations[0],\n norm_module=self.norm_module)\n self.s2_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 4 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s3 = ResStage(\n dim_in=[\n self.width_per_group * 4 +\n self.width_per_group * 4 // out_dim_ratio,\n self.width_per_group * 4 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 8,\n self.width_per_group * 8 // self.beta,\n ],\n dim_inner=[dim_inner * 2, dim_inner * 2 // self.beta],"
+ },
+ {
+ "comment": "The code initializes and defines different layers for the ResNet SlowFast model. It includes operations such as creating convolutional layers, fusing fast to slow features, and defining a stage layer with specified input and output dimensions. The alpha, fusion_kernel_sz, out_dim_ratio, beta, dim_inner values are used to control the specifics of these operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":704-732",
+ "content": " temp_kernel_sizes=temp_kernel[2],\n stride=spatial_strides[1],\n num_blocks=[d3] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[1],\n dilation=spatial_dilations[1],\n norm_module=self.norm_module,\n )\n self.s3_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 8 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s4 = ResStage(\n dim_in=[\n self.width_per_group * 8 +\n self.width_per_group * 8 // out_dim_ratio,\n self.width_per_group * 8 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 16,\n self.width_per_group * 16 // self.beta,\n ],\n dim_inner=[dim_inner * 4, dim_inner * 4 // self.beta],"
+ },
+ {
+ "comment": "The code defines the ResStage and FuseFastToSlow modules for a ResNet SlowFast model. It initializes these modules with specific dimensions and parameters, such as number of channels, fusion kernel size, alpha value, and dilation rates. These modules are used to extract features from the input and fuse them together for further processing in the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":733-761",
+ "content": " temp_kernel_sizes=temp_kernel[3],\n stride=spatial_strides[2],\n num_blocks=[d4] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[2],\n dilation=spatial_dilations[2],\n norm_module=self.norm_module,\n )\n self.s4_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 16 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s5 = ResStage(\n dim_in=[\n self.width_per_group * 16 +\n self.width_per_group * 16 // out_dim_ratio,\n self.width_per_group * 16 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 32,\n self.width_per_group * 32 // self.beta,\n ],\n dim_inner=[dim_inner * 8, dim_inner * 8 // self.beta],"
+ },
+ {
+ "comment": "This code defines a ResNet slowfast model with specified parameters, initializes the weights, and forwards the input data. It also includes an optional max-pooling operation for one of its stages (s2) depending on a flag value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":762-787",
+ "content": " temp_kernel_sizes=temp_kernel[4],\n stride=spatial_strides[3],\n num_blocks=[d5] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[3],\n dilation=spatial_dilations[3],\n norm_module=self.norm_module,\n )\n def init_weights(self):\n pass\n def forward(self, x):\n x = self.s1(x) #VideoModelStem\n x = self.s1_fuse(x) #FuseFastToSlow\n x = self.s2(x) #ResStage\n x = self.s2_fuse(x)\n # TODO: For AVA, set use_pool_af_s2=1, check mAP's improve.\n if self.use_pool_af_s2:\n for pathway in range(self.num_pathways):\n x[pathway] = F.max_pool3d(x=x[pathway],\n kernel_size=self.pool_size_ratio[pathway],\n stride=self.pool_size_ratio[pathway],\n padding=[0, 0, 0],\n data_format=\"NCDHW\")"
+ },
+ {
+ "comment": "This code snippet represents the final segment of a neural network model. It processes input data (x) through four sequential layers (s3, s4, and s5), then fuses their outputs before returning the result as output. Each layer is likely responsible for feature extraction or transformation at different levels of the network's architecture.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast.py\":789-794",
+ "content": " x = self.s3(x)\n x = self.s3_fuse(x)\n x = self.s4(x)\n x = self.s4_fuse(x)\n x = self.s5(x)\n return x"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/1feadf68-13cb-4302-9caf-d3d0a954aea9.json b/docs/doc/1feadf68-13cb-4302-9caf-d3d0a954aea9.json
new file mode 100644
index 000000000..3158f5ced
--- /dev/null
+++ b/docs/doc/1feadf68-13cb-4302-9caf-d3d0a954aea9.json
@@ -0,0 +1,25 @@
+{
+ "summary": "ASPPModule is a CNN layer for ASPP modules in Ma-Net, implementing atrous spatial pyramid pooling with Conv2D, BatchNorm, and ReLU activation. The class initializes instance parameters and sets dilations for ASPP modules using _ASPPModule class.",
+ "details": [
+ {
+ "comment": "ASPPModule is a convolutional neural network layer that performs atrous spatial pyramid pooling. It consists of a Conv2D layer, BatchNorm layer, and ReLU activation function for feature extraction and normalization in a hierarchical manner. The weight initialization follows the Kaiming normal distribution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/aspp.py\":0-33",
+ "content": "import math\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom utils.api import kaiming_normal_\nclass _ASPPModule(nn.Layer):\n def __init__(self, inplanes, planes, kernel_size, padding, dilation,\n BatchNorm):\n super(_ASPPModule, self).__init__()\n self.atrous_conv = nn.Conv2D(inplanes,\n planes,\n kernel_size=kernel_size,\n stride=1,\n padding=padding,\n dilation=dilation,\n bias_attr=False)\n self.bn = BatchNorm(planes)\n self.relu = nn.ReLU(True)\n self._init_weight()\n def forward(self, x):\n x = self.atrous_conv(x)\n x = self.bn(x)\n return self.relu(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight)\n elif isinstance(m, nn.BatchNorm2D):"
+ },
+ {
+ "comment": "The code defines a class \"ASPP\" that inherits from \"nn.Layer\". It initializes the instance with parameters such as backbone, output_stride, and BatchNorm. Depending on these inputs, it sets the dilations for the ASPP modules. These ASPP modules are instances of _ASPPModule class with specified input size, output size, kernel size, and dilation rate.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/aspp.py\":34-65",
+ "content": " from utils.api import fill_\n fill_(m.weight, 1)\n from utils.api import zero_\n zero_(m.bias)\nclass ASPP(nn.Layer):\n def __init__(self, backbone, output_stride, BatchNorm):\n super(ASPP, self).__init__()\n if backbone == 'drn':\n inplanes = 512\n elif backbone == 'mobilenet':\n inplanes = 320\n else:\n inplanes = 2048\n if output_stride == 16:\n dilations = [1, 6, 12, 18]\n elif output_stride == 8:\n dilations = [1, 12, 24, 36]\n else:\n raise NotImplementedError\n self.aspp1 = _ASPPModule(inplanes,\n 256,\n 1,\n padding=0,\n dilation=dilations[0],\n BatchNorm=BatchNorm)\n self.aspp2 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[1],"
+ },
+ {
+ "comment": "This code defines three ASPP modules and a global average pooling layer for a neural network, with batch normalization and ReLU activations applied. The convolutional layers have specific dilations and padding values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/aspp.py\":66-88",
+ "content": " dilation=dilations[1],\n BatchNorm=BatchNorm)\n self.aspp3 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[2],\n dilation=dilations[2],\n BatchNorm=BatchNorm)\n self.aspp4 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[3],\n dilation=dilations[3],\n BatchNorm=BatchNorm)\n self.global_avg_pool = nn.Sequential(\n nn.AdaptiveAvgPool2D((1, 1)),\n nn.Conv2D(inplanes, 256, 1, stride=1, bias_attr=False),\n BatchNorm(256), nn.ReLU())\n self.conv1 = nn.Conv2D(1280, 256, 1, bias_attr=False)\n self.bn1 = BatchNorm(256)\n self.relu = nn.ReLU(True)\n self.dropout = nn.Dropout(0.1)"
+ },
+ {
+ "comment": "The code defines a convolutional neural network (CNN) for the ASPP (Aggregated Spatial Pyramid Pooling) module in PaddleVideo's Ma-Net. It includes an initialization function, forward pass computation, and a builder function to create the ASPP module with specified parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/aspp.py\":89-122",
+ "content": " self._init_weight()\n def forward(self, x):\n x1 = self.aspp1(x)\n x2 = self.aspp2(x)\n x3 = self.aspp3(x)\n x4 = self.aspp4(x)\n x5 = self.global_avg_pool(x)\n x5 = F.interpolate(x5,\n size=x4.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat((x1, x2, x3, x4, x5), axis=1)\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n return x\n return self.dropout(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight)\n elif isinstance(m, nn.BatchNorm2D):\n from utils.api import fill_\n fill_(m.weight, 1)\n from utils.api import zero_\n zero_(m.bias)\ndef build_aspp(backbone, output_stride, BatchNorm):\n return ASPP(backbone, output_stride, BatchNorm)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2017add4-c3af-4d8a-9e69-400b8b3d6db6.json b/docs/doc/2017add4-c3af-4d8a-9e69-400b8b3d6db6.json
new file mode 100644
index 000000000..f6f10218b
--- /dev/null
+++ b/docs/doc/2017add4-c3af-4d8a-9e69-400b8b3d6db6.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The code processes video data, creating labels, organizing files, and parsing command line arguments. It allows for segmentation or localization labeling with features such as label conversion and ground truth processing.",
+ "details": [
+ {
+ "comment": "This code reads label files from a specified path, converts the labels to localization format and writes them into another specified output path. It also generates mapping information between the localization format and original format. The function takes prefix_data_path (path to read data), out_path (output path for results), action_dict (dictionary of action mappings) and fps (frames per second) as input parameters. It processes each label file in the prefix_data_path, updating labels_list with converted labels, and writes them to the output path. Finally, it generates mapping information in \"mapping.txt\" format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":0-33",
+ "content": "import json\nimport numpy as np\nimport argparse\nimport os\nfrom tqdm import tqdm\ndef generate_mapping_list_txt(action_dict, out_path):\n out_txt_file_path = os.path.join(out_path, \"mapping.txt\")\n f = open(out_txt_file_path, \"w\", encoding='utf-8')\n for key, action_name in action_dict.items():\n str_str = str(key) + \" \" + action_name + \"\\n\"\n f.write(str_str)\n # add None\n str_str = str(len(action_dict)) + \" None\" + \"\\n\"\n f.write(str_str)\n f.close()\ndef segmentation_convert_localization_label(prefix_data_path, out_path,\n action_dict, fps):\n label_path = os.path.join(prefix_data_path)\n label_txt_name_list = os.listdir(label_path)\n labels_dict = {}\n labels_dict[\"fps\"] = fps\n labels_list = []\n for label_name in tqdm(label_txt_name_list, desc='label convert:'):\n label_dict = {}\n label_dict[\"url\"] = label_name.split(\".\")[0] + \".mp4\"\n label_txt_path = os.path.join(prefix_data_path, label_name)\n with open(label_txt_path, \"r\", encoding='utf-8') as f:"
+ },
+ {
+ "comment": "This code segment reads a ground truth file line by line and counts the total frames. It then identifies action boundaries, creates an actions list, and for each action, it extracts action name, start and end time (in seconds), and the corresponding label ID from the action dictionary to create a label_action_dict. This information will be useful in transforming segmentation labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":34-54",
+ "content": " gt = f.read().split(\"\\n\")[:-1]\n label_dict[\"total_frames\"] = len(gt)\n boundary_index_list = [0]\n before_action_name = gt[0]\n for index in range(1, len(gt)):\n if before_action_name != gt[index]:\n boundary_index_list.append(index)\n before_action_name = gt[index]\n actions_list = []\n for index in range(len(boundary_index_list) - 1):\n if gt[boundary_index_list[index]] != \"None\":\n action_name = gt[boundary_index_list[index]]\n start_sec = float(boundary_index_list[index]) / float(fps)\n end_sec = float(boundary_index_list[index + 1] - 1) / float(fps)\n action_id = action_dict[action_name]\n label_action_dict = {}\n label_action_dict[\"label_names\"] = action_name\n label_action_dict[\"start_id\"] = start_sec\n label_action_dict[\"end_id\"] = end_sec\n label_action_dict[\"label_ids\"] = [action_id]"
+ },
+ {
+ "comment": "This code appears to be part of a larger program that performs video segmentation and labeling. It generates a dictionary containing action labels based on provided ground truth segmentation data, converts segmentation labels into localization format, and saves the results in JSON format for further use. The function `generate_action_dict()` generates an action dictionary, `load_action_dict()` loads an action dictionary from a file, and `localization_convert_segmentation_label()` converts segmentation labels into localization format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":55-89",
+ "content": " actions_list.append(label_action_dict)\n label_dict[\"actions\"] = actions_list\n labels_list.append(label_dict)\n labels_dict[\"gts\"] = labels_list\n output_path = os.path.join(out_path, \"output.json\")\n f = open(output_path, \"w\", encoding='utf-8')\n f.write(json.dumps(labels_dict, indent=4))\n f.close()\ndef generate_action_dict(label):\n action_dict = {}\n for gt in label[\"gts\"]:\n for action in gt[\"actions\"]:\n label_id = action[\"label_ids\"][0]\n label_name = action[\"label_names\"][0]\n action_dict[label_id] = label_name\n return action_dict\ndef load_action_dict(data_path):\n mapping_txt_path = os.path.join(data_path, \"mapping.txt\")\n with open(mapping_txt_path, \"r\", encoding='utf-8') as f:\n actions = f.read().split(\"\\n\")[:-1]\n class2id_map = dict()\n for a in actions:\n class2id_map[a.split()[1]] = int(a.split()[0])\n return class2id_map\ndef localization_convert_segmentation_label(label, prefix_data_path, out_path):"
+ },
+ {
+ "comment": "The code checks if a directory exists and creates it if not. It then loops through each ground truth segmentation in the label, retrieves the corresponding video data, extracts relevant information like feature, action labels, start and end indices, and populates seg_label array with action labels for the specified time range.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":90-118",
+ "content": " path = os.path.join(out_path, \"groundTruth\")\n isExists = os.path.exists(path)\n if not isExists:\n os.makedirs(path)\n print(path + ' \u521b\u5efa\u6210\u529f')\n else:\n print(path + ' \u76ee\u5f55\u5df2\u5b58\u5728')\n fps = float(label[\"fps\"])\n video_list = []\n for gt in tqdm(label[\"gts\"], desc='label convert:'):\n video_name = gt[\"url\"].split(\".\")[0]\n data_path = os.path.join(prefix_data_path, video_name + \".pkl\")\n video_list.append(video_name + \".txt\")\n feature = np.load(data_path, allow_pickle=True)[\"image_feature\"]\n num_feture = feature.shape[0]\n seg_label = [\"None\"] * (num_feture)\n for action in gt[\"actions\"]:\n start_id = action[\"start_id\"]\n end_id = action[\"end_id\"]\n label_name = action[\"label_names\"]\n start_index = int(np.floor(start_id * fps))\n end_index = int(np.floor(end_id * fps)) + 1\n if end_index < num_feture - 1:\n seg_label[start_index:end_index] = label_name * (end_index -"
+ },
+ {
+ "comment": "This code segment is part of a larger program that appears to be related to video data processing. The function is setting up a segmentation label and writing it to a file, as well as creating another list for training purposes. It determines the starting index based on the number of features, and fills in the label accordingly. The code then writes the label and video list to separate text files in the specified output path. This process is controlled by the \"args\" variable which contains command line arguments like \"mode\", \"label_path\", and \"out_path\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":119-146",
+ "content": " start_index)\n elif start_index < num_feture - 1:\n seg_label[start_index:] = label_name * (num_feture -\n start_index)\n else:\n pass\n if len(seg_label) != num_feture:\n seg_label = seg_label[:num_feture]\n out_txt_file_path = os.path.join(out_path, \"groundTruth\",\n video_name + \".txt\")\n str = '\\n'\n f = open(out_txt_file_path, \"w\", encoding='utf-8')\n f.write(str.join(seg_label) + str)\n f.close()\n out_txt_file_path = os.path.join(out_path, \"train_list.txt\")\n str = '\\n'\n f = open(out_txt_file_path, \"w\", encoding='utf-8')\n f.write(str.join(video_list) + str)\n f.close()\ndef main():\n args = get_arguments()\n if args.mode in [\"segmentation\", \"localization\"]:\n if args.mode == \"segmentation\":\n with open(args.label_path, 'r', encoding='utf-8') as json_file:"
+ },
+ {
+ "comment": "The code reads a label file, determines the mode (segmentation or localization), and performs corresponding operations. It uses function calls like generate_action_dict, load_action_dict, segmentation_convert_localization_label. The get_arguments function parses command line arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":147-172",
+ "content": " label = json.load(json_file)\n action_dict = generate_action_dict(label)\n generate_mapping_list_txt(action_dict, args.out_path)\n localization_convert_segmentation_label(label, args.data_path,\n args.out_path)\n elif args.mode == \"localization\":\n action_dict = load_action_dict(args.label_path)\n segmentation_convert_localization_label(args.data_path,\n args.out_path,\n action_dict,\n fps=25.0)\n else:\n raise NotImplementedError\ndef get_arguments():\n \"\"\"\n parse all the arguments from command line inteface\n return a list of parsed arguments\n \"\"\"\n parser = argparse.ArgumentParser(\n description=\"convert segmentation and localization label\")\n parser.add_argument(\"label_path\", type=str, help=\"path of a label file\")"
+ },
+ {
+ "comment": "This code snippet defines command line arguments for the input data path, output path, and mode. It then parses these arguments and returns them, allowing the program to convert segmentation or localization labels as specified by the user.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/transform_segmentation_label.py\":173-194",
+ "content": " parser.add_argument(\n \"data_path\",\n type=str,\n help=\"path of video feature or segmentation label txt.\",\n )\n parser.add_argument(\n \"out_path\",\n type=str,\n help=\"path of output file.\",\n )\n parser.add_argument(\n \"--mode\",\n type=str,\n default=\"segmentation\",\n help=\"Convert segmentation label or localization label.\",\n )\n return parser.parse_args()\nif __name__ == \"__main__\":\n main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/207c687e-6ba7-45cd-ab2d-3dea0024a48a.json b/docs/doc/207c687e-6ba7-45cd-ab2d-3dea0024a48a.json
new file mode 100644
index 000000000..3e8cffb1d
--- /dev/null
+++ b/docs/doc/207c687e-6ba7-45cd-ab2d-3dea0024a48a.json
@@ -0,0 +1,50 @@
+{
+ "summary": "The code initializes models, checks CUDA availability and version, parses command line arguments, trains a video tagging model using data parallelism and saves if needed. The code also checks the version, logs arguments, creates a directory, and proceeds to train using those arguments.",
+ "details": [
+ {
+ "comment": "This code snippet contains the necessary import statements and license information for the VideoTag application in PaddleVideo. It also sets up the logging format and includes utility functions from other modules such as train_utils, config_utils, reader, metrics, and utility. The code checks if CUDA is available and verifies the PaddlePaddle version before proceeding with the training process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":0-31",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport os\nimport sys\nimport argparse\nimport ast\nimport logging\nimport paddle\nimport paddle.static as static\nfrom utils.train_utils import train_with_dataloader\nimport models\nfrom utils.config_utils import *\nfrom reader import get_reader\nfrom metrics import get_metrics\nfrom utils.utility import check_cuda\nfrom utils.utility import check_version\nlogging.root.handlers = []\nFORMAT = '[%(levelname)s: %(filename)s: %(lineno)4d]: %(message)s'"
+ },
+ {
+ "comment": "This code block sets up logging, defines a function parse_args which uses argparse to create an argument parser for specifying model name, config file path, batch size, learning rate and pretrain weights. It provides default values for these arguments in case they are not specified by the user.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":32-59",
+ "content": "logging.basicConfig(level=logging.INFO, format=FORMAT, stream=sys.stdout)\nlogger = logging.getLogger(__name__)\ndef parse_args():\n parser = argparse.ArgumentParser(\"Paddle Video train script\")\n parser.add_argument('--model_name',\n type=str,\n default='AttentionCluster',\n help='name of model to train.')\n parser.add_argument('--config',\n type=str,\n default='configs/attention_cluster.txt',\n help='path to config file of model')\n parser.add_argument(\n '--batch_size',\n type=int,\n default=None,\n help='training batch size. None to use config file setting.')\n parser.add_argument(\n '--learning_rate',\n type=float,\n default=None,\n help='learning rate use for training. None to use config file setting.')\n parser.add_argument('--pretrain',\n type=str,\n default=None,\n help='path to pretrain weights.')"
+ },
+ {
+ "comment": "The code snippet is parsing command line arguments for a training program. The options include whether to use GPU, disable memory optimization, specify the epoch number, set validation interval, and provide a directory to save training snapshots.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":60-81",
+ "content": " parser.add_argument('--use_gpu',\n type=ast.literal_eval,\n default=True,\n help='default use gpu.')\n parser.add_argument('--no_memory_optimize',\n action='store_true',\n default=False,\n help='whether to use memory optimize in train')\n parser.add_argument('--epoch',\n type=int,\n default=None,\n help='epoch number, 0 for read from config file')\n parser.add_argument('--valid_interval',\n type=int,\n default=1,\n help='validation epoch interval, 0 for no validation.')\n parser.add_argument('--save_dir',\n type=str,\n default=os.path.join('data', 'checkpoints'),\n help='directory name to save train snapshoot')\n parser.add_argument('--log_interval',\n type=int,"
+ },
+ {
+ "comment": "This code is parsing command line arguments, loading and merging configuration files, initializing models in training and validation modes, and setting up a static program guard for building the model. It also allows the option to fix random seeds for reproducibility.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":82-108",
+ "content": " default=1,\n help='mini-batch interval to log.')\n parser.add_argument('--fix_random_seed',\n type=ast.literal_eval,\n default=False,\n help='If set True, enable continuous evaluation job.')\n args = parser.parse_args()\n return args\ndef train(args):\n # parse config\n config = parse_config(args.config)\n train_config = merge_configs(config, 'train', vars(args))\n valid_config = merge_configs(config, 'valid', vars(args))\n print_configs(train_config, 'Train')\n train_model = models.get_model(args.model_name, train_config, mode='train')\n valid_model = models.get_model(args.model_name, valid_config, mode='valid')\n # build model\n startup = static.Program()\n train_prog = static.Program()\n if args.fix_random_seed:\n startup.random_seed = 1000\n train_prog.random_seed = 1000\n with static.program_guard(train_prog, startup):\n with paddle.utils.unique_name.guard():"
+ },
+ {
+ "comment": "This code initializes the training and validation models, builds their inputs, models, and feeds. It also sets up the dataloaders, optimizer, and executor for the training phase. If pre-trained parameters are specified, they will be loaded before training starts.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":109-133",
+ "content": " train_model.build_input(use_dataloader=True)\n train_model.build_model()\n # for the input, has the form [data1, data2,..., label], so train_feeds[-1] is label\n train_feeds = train_model.feeds()\n train_fetch_list = train_model.fetches()\n train_loss = train_fetch_list[0]\n optimizer = train_model.optimizer()\n optimizer.minimize(train_loss)\n train_dataloader = train_model.dataloader()\n valid_prog = static.Program()\n with static.program_guard(valid_prog, startup):\n with paddle.utils.unique_name.guard():\n valid_model.build_input(use_dataloader=True)\n valid_model.build_model()\n valid_feeds = valid_model.feeds()\n valid_fetch_list = valid_model.fetches()\n valid_dataloader = valid_model.dataloader()\n place = paddle.CUDAPlace(0) if args.use_gpu else paddle.CPUPlace()\n exe = static.Executor(place)\n exe.run(startup)\n if args.pretrain:\n train_model.load_pretrain_params(exe, args.pretrain, train_prog)"
+ },
+ {
+ "comment": "This code initializes a BuildStrategy and an ExecutionStrategy. It then creates two CompiledPrograms, one for training and one for validation, with data parallelism enabled. The number of GPUs is checked using CUDA_VISIBLE_DEVICES environment variable, and the number of GPUs must match what was set in the train configuration file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":135-160",
+ "content": " build_strategy = static.BuildStrategy()\n build_strategy.enable_inplace = True\n exec_strategy = static.ExecutionStrategy()\n compiled_train_prog = static.CompiledProgram(\n train_prog).with_data_parallel(loss_name=train_loss.name,\n build_strategy=build_strategy,\n exec_strategy=exec_strategy)\n compiled_valid_prog = static.CompiledProgram(\n valid_prog).with_data_parallel(share_vars_from=compiled_train_prog,\n build_strategy=build_strategy,\n exec_strategy=exec_strategy)\n # get reader\n bs_denominator = 1\n if args.use_gpu:\n # check number of GPUs\n gpus = os.getenv(\"CUDA_VISIBLE_DEVICES\", \"\")\n if gpus == \"\":\n pass\n else:\n gpus = gpus.split(\",\")\n num_gpus = len(gpus)\n assert num_gpus == train_config.TRAIN.num_gpus, \\\n \"num_gpus({}) set by CUDA_VISIBLE_DEVICES \" \\"
+ },
+ {
+ "comment": "Sets batch size based on number of GPUs, initializes train and valid readers, gets metrics for training and validation, sets the sample list generator for dataloader.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":161-180",
+ "content": " \"shoud be the same as that \" \\\n \"set in {}({})\".format(\n num_gpus, args.config, train_config.TRAIN.num_gpus)\n bs_denominator = train_config.TRAIN.num_gpus\n train_config.TRAIN.batch_size = int(train_config.TRAIN.batch_size /\n bs_denominator)\n valid_config.VALID.batch_size = int(valid_config.VALID.batch_size /\n bs_denominator)\n train_reader = get_reader(args.model_name.upper(), 'train', train_config)\n valid_reader = get_reader(args.model_name.upper(), 'valid', valid_config)\n # get metrics\n train_metrics = get_metrics(args.model_name.upper(), 'train', train_config)\n valid_metrics = get_metrics(args.model_name.upper(), 'valid', valid_config)\n epochs = args.epoch or train_model.epoch_num()\n exe_places = static.cuda_places() if args.use_gpu else static.cpu_places()\n train_dataloader.set_sample_list_generator(train_reader, places=exe_places)"
+ },
+ {
+ "comment": "The code trains a video tagging model using PaddlePaddle framework. It sets the sample list generator for valid data and then calls the train_with_dataloader function with various parameters such as number of epochs, log and validation intervals, and data loaders for training and testing. The function trains and tests the model, saving it if necessary. The code also checks whether the installed PaddlePaddle is compiled with GPU support based on the argument provided.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":181-204",
+ "content": " valid_dataloader.set_sample_list_generator(valid_reader, places=exe_places)\n train_with_dataloader(exe,\n train_prog,\n compiled_train_prog,\n train_dataloader,\n train_fetch_list,\n train_metrics,\n epochs=epochs,\n log_interval=args.log_interval,\n valid_interval=args.valid_interval,\n save_dir=args.save_dir,\n save_model_name=args.model_name,\n fix_random_seed=args.fix_random_seed,\n compiled_test_prog=compiled_valid_prog,\n test_dataloader=valid_dataloader,\n test_fetch_list=valid_fetch_list,\n test_metrics=valid_metrics)\nif __name__ == \"__main__\":\n args = parse_args()\n # check whether the installed paddle is compiled with GPU\n check_cuda(args.use_gpu)"
+ },
+ {
+ "comment": "This code snippet checks the version, logs the arguments, creates a directory if it doesn't exist, and then proceeds to train using those arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/train.py\":205-211",
+ "content": " check_version()\n logger.info(args)\n if not os.path.exists(args.save_dir):\n os.makedirs(args.save_dir)\n train(args)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/20a62983-2c0e-419c-bea1-c9a43141235c.json b/docs/doc/20a62983-2c0e-419c-bea1-c9a43141235c.json
new file mode 100644
index 000000000..6ea507208
--- /dev/null
+++ b/docs/doc/20a62983-2c0e-419c-bea1-c9a43141235c.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a class \"TransNetV2Loss\" for calculating TransNetV2 model loss with transition_weight and many-hot_loss_weight parameters, using weighted binary cross-entropy loss for one-hot and many-hot predictions. The snippet returns the total loss from TransNetV2 components.",
+ "details": [
+ {
+ "comment": "This code defines a class called \"TransNetV2Loss\" for calculating the loss in TransNetV2 model. It inherits from BaseWeightedLoss and takes transition_weight and many_hot_loss_weight as parameters for customizing the loss calculation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/transnetv2_loss.py\":0-27",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass TransNetV2Loss(BaseWeightedLoss):\n \"\"\"Loss for TransNetV2 model\n \"\"\"\n def __init__(self, transition_weight=5.0, many_hot_loss_weight=0.1):\n self.transition_weight = transition_weight\n self.many_hot_loss_weight = many_hot_loss_weight\n super().__init__()"
+ },
+ {
+ "comment": "This code defines a loss function for the TransNetV2 model, taking in one-hot and many-hot predictions and ground truth labels. It calculates the binary cross-entropy loss for both types of predictions, applies a weighted factor based on transition weight, and averages the losses before summing them together.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/transnetv2_loss.py\":29-53",
+ "content": " def _forward(self, one_hot_pred, one_hot_gt,\n many_hot_pred=None, many_hot_gt=None, reg_losses=None):\n assert transition_weight != 1\n one_hot_pred = one_hot_pred[:, :, 0]\n one_hot_gt = one_hot_gt.astype('float32')\n one_hot_loss = F.binary_cross_entropy_with_logits(logit=one_hot_pred, label=one_hot_gt, reduction='none')\n one_hot_loss *= 1 + one_hot_gt * (transition_weight - 1)\n one_hot_loss = paddle.mean(one_hot_loss)\n many_hot_loss = 0.\n if many_hot_loss_weight != 0. and many_hot_pred is not None:\n many_hot_loss = many_hot_loss_weight * paddle.mean(\n F.binary_cross_entropy_with_logits(logit=many_hot_pred[:, :, 0],\n label=many_hot_gt.astype('float32'), reduction='none'))\n total_loss = one_hot_loss + many_hot_loss\n if reg_losses is not None:\n for name, value in reg_losses.items():\n if value is not None:\n total_loss += value"
+ },
+ {
+ "comment": "This code snippet is returning the total loss computed from various components of the TransNetV2 model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/transnetv2_loss.py\":55-55",
+ "content": " return total_loss"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/21a03f52-a095-4440-ac9c-3d9d95ece5a1.json b/docs/doc/21a03f52-a095-4440-ac9c-3d9d95ece5a1.json
new file mode 100644
index 000000000..b71375835
--- /dev/null
+++ b/docs/doc/21a03f52-a095-4440-ac9c-3d9d95ece5a1.json
@@ -0,0 +1,20 @@
+{
+ "summary": "Mixup class in PaddleVideo enhances video quality assessment by mixing images and labels from batches using adjustable alpha values, while Cutmix operator randomly selects boxes for mixing operations. Data augmentation is applied with random bounding boxes, and lambda is calculated for loss calculation.",
+ "details": [
+ {
+ "comment": "This code defines a Mixup class for video quality assessment using PaddleVideo. It is an operator that randomly mixes images and labels from batches to enhance the model's learning ability, with an adjustable alpha value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/mix.py\":0-35",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport random\nimport numpy as np\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass Mixup(object):\n \"\"\"\n Mixup operator.\n Args:\n alpha(float): alpha value.\n \"\"\"\n def __init__(self, alpha=0.2):\n assert alpha > 0., \\\n 'parameter alpha[%f] should > 0.0' % (alpha)\n self.alpha = alpha\n def __call__(self, batch):\n imgs, labels = list(zip(*batch))\n imgs = np.array(imgs)"
+ },
+ {
+ "comment": "This code defines the Cutmix operator, which is used to mix images and their corresponding labels in a dataset. It takes an alpha parameter that determines the mixing ratio, and randomly selects a box to cut out from each image. It then applies a random mixing operation within this box to create augmented versions of both the image and label. The final output is a list containing the original image, its original label, its new mixed label, and an array of the lambda values used for the mixing process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/mix.py\":36-71",
+ "content": " labels = np.array(labels)\n bs = len(batch)\n idx = np.random.permutation(bs)\n lam = np.random.beta(self.alpha, self.alpha)\n lams = np.array([lam] * bs, dtype=np.float32)\n imgs = lam * imgs + (1 - lam) * imgs[idx]\n return list(zip(imgs, labels, labels[idx], lams))\n@PIPELINES.register()\nclass Cutmix(object):\n \"\"\" Cutmix operator\n Args:\n alpha(float): alpha value.\n \"\"\"\n def __init__(self, alpha=0.2):\n assert alpha > 0., \\\n 'parameter alpha[%f] should > 0.0' % (alpha)\n self.alpha = alpha\n def rand_bbox(self, size, lam):\n \"\"\" rand_bbox \"\"\"\n w = size[2]\n h = size[3]\n cut_rat = np.sqrt(1. - lam)\n cut_w = np.int(w * cut_rat)\n cut_h = np.int(h * cut_rat)\n # uniform\n cx = np.random.randint(w)\n cy = np.random.randint(h)\n bbx1 = np.clip(cx - cut_w // 2, 0, w)\n bby1 = np.clip(cy - cut_h // 2, 0, h)\n bbx2 = np.clip(cx + cut_w // 2, 0, w)\n bby2 = np.clip(cy + cut_h // 2, 0, h)"
+ },
+ {
+ "comment": "This function generates random bounding boxes and applies data augmentation by replacing portions of images with random patches from the same image. It also calculates lambda, which is used for weighting the original and augmented samples in the loss calculation. The function returns the modified images, labels, original labels, and lambdas.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/mix.py\":73-90",
+ "content": " return bbx1, bby1, bbx2, bby2\n def __call__(self, batch):\n imgs, labels = list(zip(*batch))\n imgs = np.array(imgs)\n labels = np.array(labels)\n bs = len(batch)\n idx = np.random.permutation(bs)\n lam = np.random.beta(self.alpha, self.alpha)\n bbx1, bby1, bbx2, bby2 = self.rand_bbox(imgs.shape, lam)\n imgs[:, :, bbx1:bbx2, bby1:bby2] = imgs[idx, :, bbx1:bbx2, bby1:bby2]\n lam = 1 - (float(bbx2 - bbx1) * (bby2 - bby1) /\n (imgs.shape[-2] * imgs.shape[-1]))\n lams = np.array([lam] * bs, dtype=np.float32)\n return list(zip(imgs, labels, labels[idx], lams))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/21fc55b2-e95e-4aec-8338-e3a65b639b4a.json b/docs/doc/21fc55b2-e95e-4aec-8338-e3a65b639b4a.json
new file mode 100644
index 000000000..705ac7667
--- /dev/null
+++ b/docs/doc/21fc55b2-e95e-4aec-8338-e3a65b639b4a.json
@@ -0,0 +1,90 @@
+{
+ "summary": "The PaddleVideo code offers video processing functions, including a VisionTransformer class. It initializes and applies the model using parameters, transformations, and blocks while setting up components for future use.",
+ "details": [
+ {
+ "comment": "This code snippet is from the PaddleVideo library and contains a copyright notice, license information, and several helper functions. The VisionTransformer class will be defined later in the file, which serves as a backbone model for video processing tasks. The code defines constants for zero and one values, a function to convert a single value into a tuple of length 2 (to_2tuple), and a drop path function that applies dropout to inputs with a specified probability during training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":0-36",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections.abc import Callable\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import Constant\nfrom ...utils import load_ckpt\nfrom ..registry import BACKBONES\nfrom ..weight_init import trunc_normal_\n__all__ = ['VisionTransformer']\nzeros_ = Constant(value=0.)\nones_ = Constant(value=1.)\ndef to_2tuple(x):\n return tuple([x] * 2)\ndef drop_path(x, drop_prob=0., training=False):"
+ },
+ {
+ "comment": "This code defines three classes: \"DropPath\", \"Identity\". The DropPath class implements dropout paths (Stochastic Depth) for each sample in the main path of residual blocks. It takes a single parameter, 'drop_prob', to control the probability of dropping out features. If 'drop_prob' is 0 or not training, it returns the input unchanged. The Identity class simply returns its input without any transformation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":37-64",
+ "content": " \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...\n # issuecomment-532968956 ...\n See discussion: https://github.com/tensorflow/tpu/issues/494\n \"\"\"\n if drop_prob == 0. or not training:\n return x\n keep_prob = paddle.to_tensor(1 - drop_prob, dtype=x.dtype)\n shape = (paddle.shape(x)[0], ) + (1, ) * (x.ndim - 1)\n random_tensor = keep_prob + paddle.rand(shape).astype(x.dtype)\n random_tensor = paddle.floor(random_tensor) # binarize\n output = x.divide(keep_prob) * random_tensor\n return output\nclass DropPath(nn.Layer):\n \"\"\"Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).\n \"\"\"\n def __init__(self, drop_prob=None):\n super(DropPath, self).__init__()\n self.drop_prob = drop_prob\n def forward(self, x):\n return drop_path(x, self.drop_prob, self.training)\nclass Identity(nn.Layer):"
+ },
+ {
+ "comment": "The code defines three classes: Identity, Mlp, and Attention. Identity is a simple class that returns its input unchanged. Mlp stands for Multilayer Perceptron, and it's a feed-forward neural network layer. Attention is a class for implementing attention mechanisms in the model. Both Mlp and Attention classes take inputs and return outputs after applying their respective operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":65-103",
+ "content": " def __init__(self):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\nclass Mlp(nn.Layer):\n def __init__(self,\n in_features,\n hidden_features=None,\n out_features=None,\n act_layer=nn.GELU,\n drop=0.0):\n super().__init__()\n out_features = out_features or in_features\n hidden_features = hidden_features or in_features\n self.fc1 = nn.Linear(in_features, hidden_features)\n self.act = act_layer()\n self.fc2 = nn.Linear(hidden_features, out_features)\n self.drop = nn.Dropout(drop)\n def forward(self, x):\n x = self.fc1(x)\n x = self.act(x)\n x = self.drop(x)\n x = self.fc2(x)\n x = self.drop(x)\n return x\nclass Attention(nn.Layer):\n def __init__(self,\n dim,\n num_heads=8,\n qkv_bias=False,\n qk_scale=None,\n attn_drop=0.0,\n proj_drop=0.0):"
+ },
+ {
+ "comment": "This code initializes a multi-head attention layer, and defines the forward pass. It reshapes input into query (Q), key (K), and value (V) matrices, calculates attention scores, applies dropout, and reconstructs output using residual connections and layer normalization. The `Block` class is also defined for building a Vision Transformer model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":104-137",
+ "content": " super().__init__()\n self.num_heads = num_heads\n head_dim = dim // num_heads\n self.scale = qk_scale or head_dim**-0.5\n self.qkv = nn.Linear(dim, dim * 3, bias_attr=qkv_bias)\n self.proj = nn.Linear(dim, dim)\n self.proj_drop = nn.Dropout(proj_drop)\n self.attn_drop = nn.Dropout(attn_drop)\n def forward(self, x):\n N, C = x.shape[1:]\n qkv = self.qkv(x).reshape(\n (-1, N, 3, self.num_heads, C // self.num_heads)).transpose(\n (2, 0, 3, 1, 4))\n q, k, v = qkv[0], qkv[1], qkv[2]\n attn = (q.matmul(k.transpose((0, 1, 3, 2)))) * self.scale\n attn = nn.functional.softmax(attn, axis=-1)\n attn = self.attn_drop(attn)\n x = (attn.matmul(v)).transpose((0, 2, 1, 3)).reshape((-1, N, C))\n x = self.proj(x)\n x = self.proj_drop(x)\n return x\nclass Block(nn.Layer):\n def __init__(self,\n dim,\n num_heads,\n mlp_ratio=4.0,\n qkv_bias=False,"
+ },
+ {
+ "comment": "This function is initializing a backbone model with specified parameters. It takes in arguments like attention_type, norm_layer, and others to define the model's layers, including its attention layer. If norm_layer is a string, it uses the given string as the normalization layer; if it's a Callable, it uses that function as the normalization layer. The code also checks if the attention type is 'divided_space_time'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":138-165",
+ "content": " qk_scale=None,\n drop=0.0,\n attn_drop=0.0,\n drop_path=0.1,\n act_layer=nn.GELU,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n attention_type='divided_space_time'):\n super().__init__()\n self.attention_type = attention_type\n if isinstance(norm_layer, str):\n self.norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop)\n # Temporal Attention Parameters\n if self.attention_type == 'divided_space_time':"
+ },
+ {
+ "comment": "This code initializes the temporal normalization layer and attention mechanism for a Vision Transformer backbone. It also creates a linear layer and drop path, based on provided configurations. The norm_layer parameter can be a string representing the desired normalization layer or a Callable object. If not a valid type, it raises a TypeError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":166-184",
+ "content": " if isinstance(norm_layer, str):\n self.temporal_norm1 = eval(norm_layer)(dim, epsilon=epsilon)\n elif isinstance(norm_layer, Callable):\n self.temporal_norm1 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n self.temporal_attn = Attention(dim,\n num_heads=num_heads,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,\n attn_drop=attn_drop,\n proj_drop=drop)\n self.temporal_fc = nn.Linear(dim, dim)\n # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here\n self.drop_path = DropPath(drop_path) if drop_path > 0. else Identity()\n if isinstance(norm_layer, str):\n self.norm2 = eval(norm_layer)(dim, epsilon=epsilon)"
+ },
+ {
+ "comment": "The code defines a class and its forward method. It sets the normalization layer, calculates the number of spatial tokens, checks the attention type, applies normalization and MLP layers to the input, and performs divided space-time attention.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":185-209",
+ "content": " elif isinstance(norm_layer, Callable):\n self.norm2 = norm_layer(dim, epsilon=epsilon)\n else:\n raise TypeError(\n \"The norm_layer must be str or paddle.nn.layer.Layer class\")\n mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = Mlp(in_features=dim,\n hidden_features=mlp_hidden_dim,\n act_layer=act_layer,\n drop=drop)\n def forward(self, x, B, T, W):\n num_spatial_tokens = (x.shape[1] - 1) // T\n H = num_spatial_tokens // W\n if self.attention_type in ['space_only', 'joint_space_time']:\n x = x + self.drop_path(self.attn(self.norm1(x)))\n x = x + self.drop_path(self.mlp(self.norm2(x)))\n return x\n elif self.attention_type == 'divided_space_time':\n ########## Temporal ##########\n xt = x[:, 1:, :]\n _, _, _, _t, _m = B, H, W, T, xt.shape[-1]\n xt = xt.reshape([-1, _t, _m])\n res_temporal = self.drop_path("
+ },
+ {
+ "comment": "This code performs spatial attention in the Vision Transformer model. It creates a cls_token, reshapes the input, concatenates it with the cls_token, and then passes it through a drop path and an attention layer. Finally, it extracts the cls_token for further use.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":210-234",
+ "content": " self.temporal_attn(self.temporal_norm1(xt)))\n _, _h, _w, _t, _m = B, H, W, T, res_temporal.shape[-1]\n res_temporal = res_temporal.reshape([-1, _h * _w * _t, _m])\n res_temporal = self.temporal_fc(res_temporal)\n xt = x[:, 1:, :] + res_temporal\n ########## Spatial ##########\n init_cls_token = x[:, 0, :].unsqueeze(1)\n cls_token = init_cls_token.tile((1, T, 1))\n _b, _t, _m = cls_token.shape\n cls_token = cls_token.reshape([-1, _m]).unsqueeze(1)\n xs = xt\n _, _h, _w, _t, _m = B, H, W, T, xs.shape[-1]\n xs = xs.reshape([-1, _h, _w, _t, _m]).transpose(\n (0, 3, 1, 2, 4)).reshape([-1, _h * _w, _m])\n xs = paddle.concat((cls_token, xs), axis=1)\n res_spatial = self.drop_path(self.attn(self.norm1(xs)))\n # Taking care of CLS token\n cls_token = res_spatial[:, 0, :]\n _, _t, _m = B, T, cls_token.shape[-1]\n cls_token = cls_token.reshape([-1, _t, _m])"
+ },
+ {
+ "comment": "This code performs averaging across frames, reshapes the spatial features, concatenates initial class token and input sequence, adds a drop path and MLP layer, and returns the output. It also defines PatchEmbed for image to patch embedding.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":235-266",
+ "content": " # averaging for every frame\n cls_token = paddle.mean(cls_token, axis=1, keepdim=True)\n res_spatial = res_spatial[:, 1:, :]\n _, _t, _h, _w, _m = B, T, H, W, res_spatial.shape[-1]\n res_spatial = res_spatial.reshape([-1, _t, _h, _w, _m]).transpose(\n (0, 2, 3, 1, 4)).reshape([-1, _h * _w * _t, _m])\n res = res_spatial\n x = xt\n x = paddle.concat((init_cls_token, x), axis=1) + paddle.concat(\n (cls_token, res), axis=1)\n # Mlp\n x = x + self.drop_path(self.mlp(self.norm2(x)))\n return x\n else:\n raise NotImplementedError\nclass PatchEmbed(nn.Layer):\n \"\"\" Image to Patch Embedding\n \"\"\"\n def __init__(self,\n img_size=224,\n patch_size=16,\n in_channels=3,\n embed_dim=768):\n super().__init__()\n img_size = to_2tuple(img_size)\n patch_size = to_2tuple(patch_size)\n num_patches = (img_size[1] // patch_size[1]) * (img_size[0] //"
+ },
+ {
+ "comment": "The code defines a VisionTransformer class that takes input patches of an image. It initializes the model parameters such as img_size, patch_size and num_patches. The forward function performs the transformation by projecting the input into embedding space using a convolutional layer. If the input image size does not match the expected model size, it raises an assertion error. This class is registered with BACKBONES for future use.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":267-297",
+ "content": " patch_size[0])\n self.img_size = img_size\n self.patch_size = patch_size\n self.num_patches = num_patches\n self.proj = nn.Conv2D(in_channels,\n embed_dim,\n kernel_size=patch_size,\n stride=patch_size)\n def forward(self, x):\n B, C, T, H, W = x.shape\n assert H == self.img_size[0] and W == self.img_size[1], \\\n f\"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]}).\"\n x = x.transpose((0, 2, 1, 3, 4))\n x = x.reshape([-1, C, H, W])\n x = self.proj(x)\n W = x.shape[-1]\n x = x.flatten(2).transpose((0, 2, 1))\n return x, T, W\n@BACKBONES.register()\nclass VisionTransformer(nn.Layer):\n \"\"\" Vision Transformer with support for patch input\n \"\"\"\n def __init__(self,\n pretrained=None,\n img_size=224,\n patch_size=16,"
+ },
+ {
+ "comment": "This code initializes a Vision Transformer (ViT) backbone model with specified parameters such as input dimensions, embedding dimension, depth, number of heads, mlp ratio, and attention type. The code sets up the patch embedding layer, creates a class token, and defines the number of patches based on the input size provided.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":298-326",
+ "content": " in_channels=3,\n embed_dim=768,\n depth=12,\n num_heads=12,\n mlp_ratio=4,\n qkv_bias=False,\n qk_scale=None,\n drop_rate=0.,\n attn_drop_rate=0.,\n drop_path_rate=0.1,\n norm_layer='nn.LayerNorm',\n epsilon=1e-5,\n num_seg=8,\n attention_type='divided_space_time',\n **args):\n super().__init__()\n self.pretrained = pretrained\n self.num_seg = num_seg\n self.attention_type = attention_type\n self.num_features = self.embed_dim = embed_dim\n self.patch_embed = PatchEmbed(img_size=img_size,\n patch_size=patch_size,\n in_channels=in_channels,\n embed_dim=embed_dim)\n num_patches = self.patch_embed.num_patches\n # Positional Embeddings\n self.cls_token = self.create_parameter(shape=(1, 1, embed_dim),"
+ },
+ {
+ "comment": "This code initializes various components of a vision transformer model, including positional embeddings (pos_embed), classification token (cls_token), and dropout layers (pos_drop, time_drop). It also creates a LayerList of blocks with specified dimensions and parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":327-349",
+ "content": " default_initializer=zeros_)\n self.pos_embed = self.create_parameter(shape=(1, num_patches + 1,\n embed_dim),\n default_initializer=zeros_)\n self.pos_drop = nn.Dropout(p=drop_rate)\n if self.attention_type != 'space_only':\n self.time_embed = self.create_parameter(shape=(1, num_seg,\n embed_dim),\n default_initializer=zeros_)\n self.time_drop = nn.Dropout(p=drop_rate)\n self.add_parameter(\"pos_embed\", self.pos_embed)\n self.add_parameter(\"cls_token\", self.cls_token)\n dpr = np.linspace(0, drop_path_rate, depth)\n self.blocks = nn.LayerList([\n Block(dim=embed_dim,\n num_heads=num_heads,\n mlp_ratio=mlp_ratio,\n qkv_bias=qkv_bias,\n qk_scale=qk_scale,"
+ },
+ {
+ "comment": "This code initializes a Vision Transformer (ViT) model. It creates a series of blocks with specified dimensions, applies normalization layers, and initializes the weight values using truncated normal distribution. Additionally, if pre-trained weights are provided, it loads them into the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":350-378",
+ "content": " drop=drop_rate,\n attn_drop=attn_drop_rate,\n drop_path=dpr[i],\n norm_layer=norm_layer,\n epsilon=epsilon,\n attention_type=self.attention_type) for i in range(depth)\n ])\n self.norm = eval(norm_layer)(embed_dim, epsilon=epsilon)\n def init_weights(self):\n \"\"\"First init model's weight\"\"\"\n trunc_normal_(self.pos_embed, std=0.02)\n trunc_normal_(self.cls_token, std=0.02)\n self.apply(self._init_fn)\n if self.attention_type == 'divided_space_time':\n i = 0\n for m in self.blocks.sublayers(include_self=True):\n m_str = str(m)\n if 'Block' in m_str:\n if i > 0:\n zeros_(m.temporal_fc.weight)\n zeros_(m.temporal_fc.bias)\n i += 1\n \"\"\"Second, if provide pretrained ckpt, load it\"\"\"\n if isinstance(\n self.pretrained, str\n ) and self.pretrained.strip() != \"\": # load pretrained weights"
+ },
+ {
+ "comment": "This code initializes the forward function of a Vision Transformer (ViT) model. It extracts features from input images, adds positional embeddings, and handles batch size changes. The trunc_normal_ and zeros_ functions are used to initialize weights and biases for layers like Linear and LayerNorm, respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":379-404",
+ "content": " load_ckpt(self,\n self.pretrained,\n num_patches=self.patch_embed.num_patches,\n num_seg=self.num_seg,\n attention_type=self.attention_type)\n def _init_fn(self, m):\n if isinstance(m, nn.Linear):\n trunc_normal_(m.weight)\n if m.bias is not None:\n zeros_(m.bias)\n elif isinstance(m, nn.LayerNorm):\n ones_(m.weight)\n zeros_(m.bias)\n def forward_features(self, x):\n # B = x.shape[0]\n B = paddle.shape(x)[0]\n x, T, W = self.patch_embed(x) # [BT,nH*nW,F]\n cls_tokens = self.cls_token.expand((B * T, -1, -1)) # [1,1,F]->[BT,1,F]\n x = paddle.concat((cls_tokens, x), axis=1)\n pos_interp = (x.shape[1] != self.pos_embed.shape[1])\n if pos_interp:\n pos_embed = self.pos_embed\n cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)\n other_pos_embed = pos_embed[0, 1:, :].unsqueeze(0).transpose("
+ },
+ {
+ "comment": "The code is applying relative position embeddings to the input features (x) for a vision transformer model. It first checks if a specific flag is set, then interpolates other position embeddings based on the size of the input and adds them to class position embeddings. If the flag is not set, it simply adds the position embeddings from the model. Afterward, the code applies time embeddings if the attention type is not \"space_only\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":405-429",
+ "content": " (0, 2, 1))\n P = int(other_pos_embed.shape[2]**0.5)\n H = x.shape[1] // W\n other_pos_embed = other_pos_embed.reshape([1, x.shape[2], P, P])\n new_pos_embed = F.interpolate(other_pos_embed,\n size=(H, W),\n mode='nearest')\n new_pos_embed = new_pos_embed.flatten(2)\n new_pos_embed = new_pos_embed.transpose((0, 2, 1))\n new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed),\n axis=1)\n x = x + new_pos_embed\n else:\n x = x + self.pos_embed\n x = self.pos_drop(x)\n # Time Embeddings\n if self.attention_type != 'space_only':\n cls_tokens = x[:B, 0, :].unsqueeze(1) if B > 0 else x.split(\n T)[0].index_select(paddle.to_tensor([0]), axis=1)\n x = x[:, 1:]\n _, _n, _m = x.shape\n _t = T\n x = x.reshape([-1, _t, _n, _m]).transpose("
+ },
+ {
+ "comment": "This code performs time embeddings resizing and adds them to the input feature maps. It then flattens the tensor, concatenates class tokens, processes through attention blocks, and finally, for space-only attention type, it makes predictions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":430-454",
+ "content": " (0, 2, 1, 3)).reshape([-1, _t, _m])\n # Resizing time embeddings in case they don't match\n time_interp = (T != self.time_embed.shape[1])\n if time_interp: # T' != T\n time_embed = self.time_embed.transpose((0, 2, 1)).unsqueeze(0)\n new_time_embed = F.interpolate(time_embed,\n size=(T, x.shape[-1]),\n mode='nearest').squeeze(0)\n new_time_embed = new_time_embed.transpose((0, 2, 1))\n x = x + new_time_embed\n else:\n x = x + self.time_embed\n x = self.time_drop(x)\n _, _t, _m = x.shape\n x = x.reshape([-1, W * W * T, _m])\n x = paddle.concat((cls_tokens, x), axis=1)\n # Attention blocks\n for blk in self.blocks:\n x = blk(x, B, T, W)\n # Predictions for space-only baseline\n if self.attention_type == 'space_only':\n _, _n, _m = x.shape"
+ },
+ {
+ "comment": "This code snippet is part of a Vision Transformer (ViT) model implementation. The function averages predictions for every frame and applies normalization before returning the embeddings for each image in the input sequence.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/vit.py\":455-464",
+ "content": " _t = T\n x = x.reshape([-1, _t, _n, _m])\n x = paddle.mean(x, 1) # averaging predictions for every frame\n x = self.norm(x)\n return x[:, 0] # [B, embed_dim]\n def forward(self, x):\n x = self.forward_features(x)\n return x"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/225fb15b-381f-479b-a3a1-53e2129c8cd8.json b/docs/doc/225fb15b-381f-479b-a3a1-53e2129c8cd8.json
new file mode 100644
index 000000000..5be528069
--- /dev/null
+++ b/docs/doc/225fb15b-381f-479b-a3a1-53e2129c8cd8.json
@@ -0,0 +1,35 @@
+{
+ "summary": "TSMRecHead is a TSNHead-based classifier head for Temporal Segment Networks, performing average pooling, optional dropout, reshaping, mean operation, and applying a fully connected layer. It uses defined loss function to compare with labels, and calculates loss based on provided labels using label smoothing and weighted average.",
+ "details": [
+ {
+ "comment": "TSMRecHead is a TSNHead-based classifier head for Temporal Segment Networks (TSMs) with specified number of classes, input channels and registered under PaddlePaddle's HEADS registry.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/heads/tsm_rec_head.py\":0-32",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport math\nimport paddle\nimport paddle.nn.functional as F\nfrom paddle.nn import AdaptiveAvgPool2D, Linear, Dropout\nfrom .base import BaseHead\nfrom .tsn_head import TSNHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass TSMRecHead(TSNHead):\n \"\"\" TSM Rec Head\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature."
+ },
+ {
+ "comment": "This function initializes the weights of the FC layer using a uniform distribution, and sets the standard deviation for normal initialization. The loss_cfg argument determines the type of loss function to use, and drop_ratio is the probability of dropping connections between layers during training. The stdv value is set based on the number of input channels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/heads/tsm_rec_head.py\":33-61",
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n drop_ratio(float): drop ratio. Default: 0.8.\n std(float): Std(Scale) value in normal initilizar. Default: 0.001.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='L1Loss'),\n drop_ratio=0.8,\n std=0.01,\n data_format=\"NCHW\",\n **kwargs):\n super().__init__(num_classes,\n in_channels,\n loss_cfg,\n drop_ratio=drop_ratio,\n std=std,\n data_format=data_format,\n **kwargs)\n self.stdv = 1.0 / math.sqrt(self.in_channels * 1.0)\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'Uniform',"
+ },
+ {
+ "comment": "This code defines a head for TSM (Temporal Shift Module) Recognition task. It includes initialization of weights, setting learning rate, and applying L2 decay regularizer. The forward method performs average pooling, optional dropout, reshaping, mean operation, and finally passes the result through fully connected layer to obtain classification scores.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/heads/tsm_rec_head.py\":62-90",
+ "content": " 'fc_0.w_0',\n 'fc_0.b_0',\n low=-self.stdv,\n high=self.stdv)\n self.fc.bias.learning_rate = 2.0\n self.fc.bias.regularizer = paddle.regularizer.L2Decay(0.)\n def forward(self, x, num_seg):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x)\n # [N * num_segs, in_channels, 1, 1]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N * num_seg, in_channels, 1, 1]\n x = paddle.reshape(x, [-1, num_seg, x.shape[1]])\n # [N, num_seg, in_channels]\n x = paddle.mean(x, axis=1)\n # [N, 1, in_channels]\n x = paddle.reshape(x, shape=[-1, self.in_channels])\n # [N, in_channels]\n score = self.fc(x)"
+ },
+ {
+ "comment": "The code defines a loss function for a model that predicts scores and compares them with the given labels. It calculates the loss between the predicted scores and the target labels, considering cases where there is only one label. The losses are returned in a dictionary format with 'loss' as the mandatory field.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/heads/tsm_rec_head.py\":91-121",
+ "content": " # [N, num_class]\n #m = paddle.nn.Sigmoid()\n #score = m(score)\n return score\n def loss(self, scores, labels, valid_mode=False, **kwargs):\n \"\"\"Calculate the loss accroding to the model output ```scores```,\n and the target ```labels```.\n Args:\n scores (paddle.Tensor): The output of the model.\n labels (paddle.Tensor): The target output of the model.\n Returns:\n losses (dict): A dict containing field 'loss'(mandatory).\n \"\"\"\n if len(labels) == 1: #commonly case\n output = []\n label = []\n labels = labels[0]\n losses = dict()\n loss = self.loss_func(scores, labels, **kwargs)\n score_list = paddle.tolist(scores)\n label_list = paddle.tolist(labels)\n score_list_len = len(score_list)\n for i in range(score_list_len):\n output.append(score_list[i][0])\n label.append(label_list[i][0])\n losses['loss'] = loss"
+ },
+ {
+ "comment": "This function calculates the loss based on the number of labels provided. If one label is given, it returns the output and label as losses. If three labels are given (a, b, lam), it casts the labels to float32, applies label smoothing or standard loss depending on epsilon, then calculates the weighted average loss for a and b. It returns the loss, output, and label in a dictionary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/heads/tsm_rec_head.py\":122-148",
+ "content": " losses['output'] = output\n losses['label'] = label\n return losses\n elif len(labels) == 3:\n labels_a, labels_b, lam = labels\n labels_a = paddle.cast(labels_a, dtype='float32')\n labels_b = paddle.cast(labels_b, dtype='float32')\n lam = lam[0] # get lam value\n losses = dict()\n if self.ls_eps != 0:\n loss_a = self.label_smooth_loss(scores, labels_a, **kwargs)\n loss_b = self.label_smooth_loss(scores, labels_b, **kwargs)\n else:\n loss_a = self.loss_func(scores, labels_a, **kwargs)\n loss_b = self.loss_func(scores, labels_a, **kwargs)\n loss = lam * loss_a + (1 - lam) * loss_b\n losses['loss'] = loss\n losses['output'] = output\n losses['label'] = label\n return losses\n else:\n raise NotImplementedError\n def label_smooth_loss(self, scores, labels, **kwargs):\n \"\"\"label smooth loss\"\"\""
+ },
+ {
+ "comment": "Applies label smoothing to the input labels, squeezes the labels along a specified axis, and calculates the loss using a provided loss function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/heads/tsm_rec_head.py\":149-152",
+ "content": " labels = F.label_smooth(labels, epsilon=self.ls_eps)\n labels = paddle.squeeze(labels, axis=1)\n loss = self.loss_func(scores, labels, **kwargs)\n return loss"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/22d919b3-064e-43b2-8d4a-c40b0e1afd3b.json b/docs/doc/22d919b3-064e-43b2-8d4a-c40b0e1afd3b.json
new file mode 100644
index 000000000..92504aa4f
--- /dev/null
+++ b/docs/doc/22d919b3-064e-43b2-8d4a-c40b0e1afd3b.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code snippet appears to import the \"build_optimizer\" and \"build_lr\" functions from their respective modules within the \"paddlevideo.solver\" package. The comments at the top of the file indicate that this code is protected by copyright and licensed under the Apache License, Version 2.0.",
+ "details": [
+ {
+ "comment": "This code snippet appears to import the \"build_optimizer\" and \"build_lr\" functions from their respective modules within the \"paddlevideo.solver\" package. The comments at the top of the file indicate that this code is protected by copyright and licensed under the Apache License, Version 2.0.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/solver/__init__.py\":0-15",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .optimizer import build_optimizer\nfrom .lr import build_lr"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/23ba667d-92f7-4c25-a37d-1fcd0aa5ab20.json b/docs/doc/23ba667d-92f7-4c25-a37d-1fcd0aa5ab20.json
new file mode 100644
index 000000000..e9b7eae05
--- /dev/null
+++ b/docs/doc/23ba667d-92f7-4c25-a37d-1fcd0aa5ab20.json
@@ -0,0 +1,35 @@
+{
+ "summary": "PaddleVideo Quick Start guide covers installation, usage details, and action recognition model for classifying video files. Highlights top-5 classes with high confidence using an example command, also suggesting alternative OpenCV installation method.",
+ "details": [
+ {
+ "comment": "Code is an English version of the Quick Start guide for PaddleVideo. It provides information on how to install the necessary packages, use PaddleVideo by command line and Python code, describes arguments, and answers frequently asked questions. The code also includes instructions for installing PaddlePaddle with or without a GPU, as well as the option to install the PaddleVideo Whl Package from pypi.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/quick_start.md\":0-35",
+ "content": "English | [\u7b80\u4f53\u4e2d\u6587](../zh-CN/quick_start.md)\n# PaddleVide Quick Start\n- [1. Installation](#1)\n - [1.1 Install PaddlePaddle](#11)\n - [1.2 Install PaddleVideo Whl Package](#12)\n- [2. Easy-to-Use](#2)\n - [2.1 Use by Command Line](#21)\n - [2.2 Use by Python Code](#22)\n- [3. Arguments description](#3)\n- [4.QA](#4)\n## 1. Installation\n\n### 1.1 Install PaddlePaddle\n- If you have CUDA 9 or CUDA 10 installed on your machine, please run the following command to install\n ```bash\n python3.7 -m pip install paddlepaddle-gpu -i https://mirror.baidu.com/pypi/simple\n ```\n- If you have no available GPU on your machine, please run the following command to install the CPU version\n ```bash\n python3.7 -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple\n ```\nFor more software version requirements, please refer to the instructions in [Installation Document](https://www.paddlepaddle.org.cn/install/quick) for operation.\n\n### 1.2 Install PaddleVideo Whl Package\n- option1: use pypi\uff08recommand\uff09"
+ },
+ {
+ "comment": "Install package using pip:\n```bash\npip3.7 install ppvideo==2.3.0\n```\nAlternatively, build and install locally:\n```bash\npython3.7 setup.py bdist_wheel\npython3.7 -m pip install dist/ppvideo-2.3.0-py3-none-any.whl\n```\nCommand to use by command line:\n```bash\nppvideo --model_name='ppTSM_v2' --use_gpu=False --video_file='data/example.avi'\n```\nThis command uses PP-TSM_v2 model on CPU for inference on data/example.avi file, divided into 16 segments and frames combined before feeding into network. Results show top-1 prediction class_id as 5, scores as 1.0, and class name as 'archery'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/quick_start.md\":37-75",
+ "content": "```bash\npip3.7 install ppvideo==2.3.0\n```\n- option2: build and install locally\n```bash\npython3.7 setup.py bdist_wheel\npython3.7 -m pip install dist/ppvideo-2.3.0-py3-none-any.whl\n```\n## 2. Easy-to-Use\n\n### 2.1 Use by Command Line\nRun shell command\uff1a\n```bash\nppvideo --model_name='ppTSM_v2' --use_gpu=False --video_file='data/example.avi'\n```\n- This command use `PP-TSM_v2` model to infer `data/example.avi` file in `CPU`.\n- The length of the example video is about 10s. When inference, the video is first divided into 16 segments according to the time axis, then extract one frame from each segment. Finally all frames are combined and feeded into the network.\nResults\uff1a\n```\nCurrent video file: data/example.avi\n top-1 classes: [5]\n top-1 scores: [1.]\n top-1 label names: ['archery']\n```\nAs you can see, use `PP-TSM_v2` trained on Kinetics-400 to predict `data/example.avi` video\uff0ctop1 prediction class_id is `5`, scores is `1.0`, class name is `archery`.\n\n### 2.2 Use by Python Code"
+ },
+ {
+ "comment": "This code uses the PaddleVideo library with the PP-TSM_v2 model for video inference on a CPU. It predicts the top-1 class, score, and label name of the provided 'data/example.avi' video file. The model is trained on Kinetics-400 dataset. Arguments include model name (PP-TSM or PP-TSM_v2), video file path, GPU usage, and other optional parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/quick_start.md\":77-106",
+ "content": "Run python code\uff1a\n```python\nfrom ppvideo import PaddleVideo\nclas = PaddleVideo(model_name='ppTSM_v2', use_gpu=False)\nvideo_file='data/example.avi'\nclas.predict(video_file)\n```\n- This code use `PP-TSM_v2` model to infer `data/example.avi` file in `CPU`.\nResults:\n```\nCurrent video file: data/example.avi\n top-1 classes: [5]\n top-1 scores: [1.]\n top-1 label names: ['archery']\n```\nAs you can see, use `PP-TSM_v2` trained on Kinetics-400 to predict `data/example.avi` video\uff0ctop1 prediction class_id is `5`, scores is `1.0`, class name is `archery`.\n\n## 3. Arguments description\n| name | type | description |\n| :---: | :---: | :--- |\n| model_name | str | optional, model name, `'ppTSM'` or `'ppTSM_v2'`. If None, please specify the path of your inference model by args `model_file` and `params_file`. |\n| video_file | str | required, Video file path, supported format: single video file path, or folder containing multiple videos. |\n| use_gpu | bool | whether to use GPU\uff0cdefault True\u3002 |\n| nu"
+ },
+ {
+ "comment": "The code defines several parameters for the PaddleVideo model including the number of segments, short and target frame sizes, model file paths, batch size, use of float16, TensorRT, MKLDNN, top_k, and label name path. It also provides a command example usage of the model with specified parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/quick_start.md\":106-121",
+ "content": "m_seg | int | The number of segments used in the TSM model, which is also the number of frames extracted from the video. 8 for `ppTSM`, 16 for `ppTSM_v2`, default 16. |\n| short_size | int | short size of frame, default 256.|\n| target_size | int | target size of frame, default 224.|\n| model_file | str | optional\uff0cinference model(`.pdmodel`)path. |\n| params_file | str | optional, inference modle(`.pdiparams`) path. |\n| batch_size | int | Batch size, default 1.|\n| use_fp16 | bool | whether to use float16\uff0cdefault False.|\n| use_tensorrt | bool| whether to use Tensorrt, default False.|\n| gpu_mem | int | use GPU memory, default 8000.|\n| enable_mkldnn | bool | whether to use MKLDNN, default False.|\n| top_k | int | top_k, default 1. |\n| label_name_path | str | This file consists the relation of class_id and class_name. Default use `data/k400/Kinetics-400_label_list.txt` of Kinetics-400. You can replace it with your own label file. |\ncommand example1\uff1a\n```bash\nppvideo --model_name='ppTSM_v2' --num_seg=16 --video_file=\"data/mp4\" --batch_size=2 --top_k=5"
+ },
+ {
+ "comment": "The code displays the top-5 classes, scores, and label names for five different video files. It shows that the classifier consistently identifies the same top-5 classes with high confidence for each video file, indicating a reliable classification performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/quick_start.md\":122-141",
+ "content": "```\nResults\uff1a\n```txt\nCurrent video file: data/mp4/example3.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]\n top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\nCurrent video file: data/mp4/example2.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]\n top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\nCurrent video file: data/mp4/example.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]\n top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\nCurrent video file: data/mp4/example1.avi\n top-5 classes: [ 5 345 311 159 327]\n top-5 scores: [1.0000000e+00 1.0152016e-11 8.2871061e-14 6.7713670e-14 5.0752070e-14]"
+ },
+ {
+ "comment": "The code provides a list of top-5 label names for PaddleVideo's action recognition model. The command example demonstrates how to run the model with specific parameters, such as the model name, number of video segments, input video file, and batch size. Additionally, it suggests an alternative installation method for OpenCV-python if the regular installation is slow.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/quick_start.md\":142-156",
+ "content": " top-5 label names: ['archery', 'sword_fighting', 'skipping_rope', 'hula_hooping', 'spray_painting']\n```\ncommand example1\uff1a\n```bash\nppvideo --model_name='ppTSM' --num_seg=8 --video_file=\"data/mp4\" --batch_size=2 --top_k=5\n```\n\n## 4. QA\n1. opecv-python Installation maybe slow, you can try:\n```\npython3.7 -m pip install opencv-python==4.2.0.32 -i https://pypi.doubanio.com/simple\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/23c777e0-a6ea-4c5b-90a7-b744b8660384.json b/docs/doc/23c777e0-a6ea-4c5b-90a7-b744b8660384.json
new file mode 100644
index 000000000..9f38e4b1d
--- /dev/null
+++ b/docs/doc/23c777e0-a6ea-4c5b-90a7-b744b8660384.json
@@ -0,0 +1,130 @@
+{
+ "summary": "This code defines image augmentation and resizing functions for AVA dataset in PaddleVideo library using operations like resizing, lazy initialization, and RandomRescale/Resize transforms. It creates classes for ground truth bounding boxes and proposals, cropping and flipping entity boxes, and includes Flip and Normalize classes for image processing and normalization.",
+ "details": [
+ {
+ "comment": "This code is part of the PaddleVideo library and contains a module for image augmentations in the AVA dataset. It imports necessary libraries, defines conversion dictionaries for interpolation methods between PIL and OpenCV, and sets up registry entries for the PIPELINES module.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":0-33",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport numpy as np\nimport math\nfrom PIL import Image\nfrom ..registry import PIPELINES\nfrom collections.abc import Sequence\nimport cv2\npillow_interp_codes = {\n 'nearest': Image.NEAREST,\n 'bilinear': Image.BILINEAR,\n 'bicubic': Image.BICUBIC,\n 'box': Image.BOX,\n 'lanczos': Image.LANCZOS,\n 'hamming': Image.HAMMING\n}\ncv2_interp_codes = {\n 'nearest': cv2.INTER_NEAREST,\n 'bilinear': cv2.INTER_LINEAR,"
+ },
+ {
+ "comment": "This function initializes the lazy operation properly, ensuring a non-lazy operation is not accidentally mixed in. If 'img_shape' is not in results, it adds 'img_shape'. If 'lazy' is set to True and 'lazy' does not exist in results, it creates a new dictionary for lazy operation containing 'original_shape', 'img_shape', 'crop_bbox', 'flip', 'flip_direction', and 'interpolation'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":34-63",
+ "content": " 'bicubic': cv2.INTER_CUBIC,\n 'area': cv2.INTER_AREA,\n 'lanczos': cv2.INTER_LANCZOS4\n}\ndef _init_lazy_if_proper(results, lazy):\n \"\"\"Initialize lazy operation properly.\n Make sure that a lazy operation is properly initialized,\n and avoid a non-lazy operation accidentally getting mixed in.\n Required keys in results are \"imgs\" if \"img_shape\" not in results,\n otherwise, Required keys in results are \"img_shape\", add or modified keys\n are \"img_shape\", \"lazy\".\n Add or modified keys in \"lazy\" are \"original_shape\", \"crop_bbox\", \"flip\",\n \"flip_direction\", \"interpolation\".\n Args:\n results (dict): A dict stores data pipeline result.\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n if 'img_shape' not in results:\n results['img_shape'] = results['imgs'][0].shape[:2]\n if lazy:\n if 'lazy' not in results:\n img_h, img_w = results['img_shape']\n lazyop = dict()\n lazyop['original_shape'] = results['img_shape']"
+ },
+ {
+ "comment": "This code defines functions for image augmentations in the AVA dataset pipeline. The \"_scale_size\" function scales a size by a ratio, while \"rescale_size\" calculates the new size to be rescaled based on an input scale factor or maximum size. The code also includes the initialization of crop parameters and flipping options for lazy operations in the results dictionary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":64-95",
+ "content": " lazyop['crop_bbox'] = np.array([0, 0, img_w, img_h],\n dtype=np.float32)\n lazyop['flip'] = False\n lazyop['flip_direction'] = None\n lazyop['interpolation'] = None\n results['lazy'] = lazyop\n else:\n assert 'lazy' not in results, 'Use Fuse after lazy operations'\ndef _scale_size(size, scale):\n \"\"\"Rescale a size by a ratio.\n Args:\n size (tuple[int]): (w, h).\n scale (float): Scaling factor.\n Returns:\n tuple[int]: scaled size.\n \"\"\"\n w, h = size\n return int(w * float(scale) + 0.5), int(h * float(scale) + 0.5)\ndef rescale_size(old_size, scale, return_scale=False):\n \"\"\"Calculate the new size to be rescaled to.\n Args:\n old_size (tuple[int]): The old size (w, h) of image.\n scale (float | tuple[int]): The scaling factor or maximum size.\n If it is a float number, then the image will be rescaled by this\n factor, else if it is a tuple of 2 integers, then the image will"
+ },
+ {
+ "comment": "Function \"imresize\" resizes an image based on the provided scale factor. If the scale is a number, it's used directly as the scaling factor. If it's a tuple of ints, it sets max and min edge sizes for resizing. Returns new resized size or both size and scaling factor if requested.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":96-129",
+ "content": " be rescaled as large as possible within the scale.\n return_scale (bool): Whether to return the scaling factor besides the\n rescaled image size.\n Returns:\n tuple[int]: The new rescaled image size.\n \"\"\"\n w, h = old_size\n if isinstance(scale, (float, int)):\n if scale <= 0:\n raise ValueError(f'Invalid scale {scale}, must be positive.')\n scale_factor = scale\n elif isinstance(scale, tuple):\n max_long_edge = max(scale)\n max_short_edge = min(scale)\n scale_factor = min(max_long_edge / max(h, w),\n max_short_edge / min(h, w))\n else:\n raise TypeError(\n f'Scale must be a number or tuple of int, but got {type(scale)}')\n new_size = _scale_size((w, h), scale_factor)\n if return_scale:\n return new_size, scale_factor\n else:\n return new_size\ndef imresize(img,\n size,\n return_scale=False,\n interpolation='bilinear',\n out=None,"
+ },
+ {
+ "comment": "This code defines a function for resizing an image to a given size. It supports two backends: 'cv2' and 'pillow'. If the backend is not specified, it defaults to 'cv2'. The function first gets the original image's height and width, checks if the backend is valid, handles unsupported backends, asserts the image type for 'pillow' backend, resizes the image using either OpenCV or Pillow library based on the backend, and finally returns the resized image along with scale factors if return_scale is True. The EntityBoxRescale class registers a pipeline to rescale entity boxes and proposals according to the image shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":130-159",
+ "content": " backend=None):\n \"\"\"Resize image to a given size. \"\"\"\n h, w = img.shape[:2]\n if backend is None:\n backend = 'cv2'\n if backend not in ['cv2', 'pillow']:\n raise ValueError(f'backend: {backend} is not supported for resize.'\n f\"Supported backends are 'cv2', 'pillow'\")\n if backend == 'pillow':\n assert img.dtype == np.uint8, 'Pillow backend only support uint8 type'\n pil_image = Image.fromarray(img)\n pil_image = pil_image.resize(size, pillow_interp_codes[interpolation])\n resized_img = np.array(pil_image)\n else:\n resized_img = cv2.resize(\n img, size, dst=out, interpolation=cv2_interp_codes[interpolation])\n if not return_scale:\n return resized_img\n else:\n w_scale = size[0] / w\n h_scale = size[1] / h\n return resized_img, w_scale, h_scale\n@PIPELINES.register()\nclass EntityBoxRescale:\n \"\"\"Rescale the entity box and proposals according to the image shape.\n Required keys are \"proposals\", \"gt_bboxes\", added or modified keys are"
+ },
+ {
+ "comment": "The code defines a class called EntityBoxCrop that scales the ground truth bounding boxes (gt_bboxes) and proposals, if present, by a given scale factor. It ensures that the number of columns in the proposals is 4. This class can be registered as a pipeline for video augmentation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":160-192",
+ "content": " \"gt_bboxes\". If original \"proposals\" is not None, \"proposals\" and\n will be added or modified.\n Args:\n scale_factor (np.ndarray): The scale factor used entity_box rescaling.\n \"\"\"\n def __init__(self, scale_factor):\n self.scale_factor = scale_factor\n def __call__(self, results):\n scale_factor = np.concatenate([self.scale_factor, self.scale_factor])\n if 'gt_bboxes' in results:\n gt_bboxes = results['gt_bboxes']\n results['gt_bboxes'] = gt_bboxes * scale_factor\n if 'proposals' in results:\n proposals = results['proposals']\n if proposals is not None:\n assert proposals.shape[1] == 4, (\n 'proposals shape should be in '\n f'(n, 4), but got {proposals.shape}')\n results['proposals'] = proposals * scale_factor\n return results\n def __repr__(self):\n return f'{self.__class__.__name__}(scale_factor={self.scale_factor})'\n@PIPELINES.register()\nclass EntityBoxCrop:"
+ },
+ {
+ "comment": "This code initializes an object that crops the entity boxes and proposals according to the cropped images. The required keys are \"proposals\" and \"gt_bboxes\", while \"gt_bboxes\" is added or modified. If original \"proposals\" is not None, \"proposals\" will be modified. The crop_bbox argument specifies the bbox used to crop the original image.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":193-223",
+ "content": " \"\"\"Crop the entity boxes and proposals according to the cropped images.\n Required keys are \"proposals\", \"gt_bboxes\", added or modified keys are\n \"gt_bboxes\". If original \"proposals\" is not None, \"proposals\" will be\n modified.\n Args:\n crop_bbox(np.ndarray | None): The bbox used to crop the original image.\n \"\"\"\n def __init__(self, crop_bbox):\n self.crop_bbox = crop_bbox\n def __call__(self, results):\n proposals = results['proposals']\n gt_bboxes = results['gt_bboxes']\n if self.crop_bbox is None:\n return results\n x1, y1, x2, y2 = self.crop_bbox\n img_w, img_h = x2 - x1, y2 - y1\n assert gt_bboxes.shape[-1] == 4\n gt_bboxes_ = gt_bboxes.copy()\n gt_bboxes_[..., 0::2] = np.clip(gt_bboxes[..., 0::2] - x1, 0, img_w - 1)\n gt_bboxes_[..., 1::2] = np.clip(gt_bboxes[..., 1::2] - y1, 0, img_h - 1)\n results['gt_bboxes'] = gt_bboxes_\n if proposals is not None:\n assert proposals.shape[-1] == 4"
+ },
+ {
+ "comment": "This code defines two classes, \"EntityBoxFlip\" and a nameless class. The nameless class performs cropping operations on proposals based on given coordinates (x1, y1). It also updates the results['proposals'] with the modified proposals. The EntityBoxFlip class flips the entity boxes and proposals horizontally with a certain probability. It adds or modifies keys \"gt_bboxes\" in the results dictionary. If \"proposals\" is not None, it will also modify them. The img_shape tuple represents the image shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":224-248",
+ "content": " proposals_ = proposals.copy()\n proposals_[..., 0::2] = np.clip(proposals[..., 0::2] - x1, 0,\n img_w - 1)\n proposals_[..., 1::2] = np.clip(proposals[..., 1::2] - y1, 0,\n img_h - 1)\n results['proposals'] = proposals_\n return results\n def __repr__(self):\n return f'{self.__class__.__name__}(crop_bbox={self.crop_bbox})'\n@PIPELINES.register()\nclass EntityBoxFlip:\n \"\"\"Flip the entity boxes and proposals with a probability.\n Reverse the order of elements in the given bounding boxes and proposals\n with a specific direction. The shape of them are preserved, but the\n elements are reordered. Only the horizontal flip is supported (seems\n vertical flipping makes no sense). Required keys are \"proposals\",\n \"gt_bboxes\", added or modified keys are \"gt_bboxes\". If \"proposals\"\n is not None, it will also be modified.\n Args:\n img_shape (tuple[int]): The img shape."
+ },
+ {
+ "comment": "This code defines a pipeline for resizing images to a specific size. It first initializes the pipeline with an image shape and then, in the __call__ method, it adjusts the ground truth bounding boxes and proposal bounding boxes by subtracting their width values from the total image width minus 1. If there are no proposals, it sets proposals_ to None. The __repr__ method provides a string representation of the pipeline.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":249-283",
+ "content": " \"\"\"\n def __init__(self, img_shape):\n self.img_shape = img_shape\n def __call__(self, results):\n proposals = results['proposals']\n gt_bboxes = results['gt_bboxes']\n img_h, img_w = self.img_shape\n assert gt_bboxes.shape[-1] == 4\n gt_bboxes_ = gt_bboxes.copy()\n gt_bboxes_[..., 0::4] = img_w - gt_bboxes[..., 2::4] - 1\n gt_bboxes_[..., 2::4] = img_w - gt_bboxes[..., 0::4] - 1\n if proposals is not None:\n assert proposals.shape[-1] == 4\n proposals_ = proposals.copy()\n proposals_[..., 0::4] = img_w - proposals[..., 2::4] - 1\n proposals_[..., 2::4] = img_w - proposals[..., 0::4] - 1\n else:\n proposals_ = None\n results['proposals'] = proposals_\n results['gt_bboxes'] = gt_bboxes_\n return results\n def __repr__(self):\n repr_str = f'{self.__class__.__name__}(img_shape={self.img_shape})'\n return repr_str\n@PIPELINES.register()\nclass Resize:\n \"\"\"Resize images to a specific size."
+ },
+ {
+ "comment": "This code defines a function for image augmentation in PaddleVideo. The function takes arguments like scale, keep_ratio, interpolation, and lazy. If keep_ratio is True, it scales the image by the given factor or resizes to the maximum size specified by the tuple. It uses bilinear interpolation by default. Lazy operation can be determined if lazy argument is set to True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":285-302",
+ "content": " Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified\n keys are \"imgs\", \"img_shape\", \"keep_ratio\", \"scale_factor\", \"lazy\",\n \"resize_size\". Required keys in \"lazy\" is None, added or modified key is\n \"interpolation\".\n Args:\n scale (float | Tuple[int]): If keep_ratio is True, it serves as scaling\n factor or maximum size:\n If it is a float number, the image will be rescaled by this\n factor, else if it is a tuple of 2 integers, the image will\n be rescaled as large as possible within the scale.\n Otherwise, it serves as (w, h) of output size.\n keep_ratio (bool): If set to True, Images will be resized without\n changing the aspect ratio. Otherwise, it will resize images to a\n given size. Default: True.\n interpolation (str): Algorithm used for interpolation:\n \"nearest\" | \"bilinear\". Default: \"bilinear\".\n lazy (bool): Determine whether to apply lazy operation. Default: False."
+ },
+ {
+ "comment": "Initializes a resize augmentation object with scale, keeps ratio (True or False), interpolation method ('bilinear' default), and lazy flag (False).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":303-333",
+ "content": " \"\"\"\n def __init__(self,\n scale,\n keep_ratio=True,\n interpolation='bilinear',\n lazy=False):\n if isinstance(scale, str):\n scale = eval(scale)\n if isinstance(scale, float):\n if scale <= 0:\n raise ValueError(f'Invalid scale {scale}, must be positive.')\n elif isinstance(scale, tuple):\n max_long_edge = max(scale)\n max_short_edge = min(scale)\n if max_short_edge == -1:\n # assign np.inf to long edge for rescaling short edge later.\n scale = (np.inf, max_long_edge)\n else:\n raise TypeError(\n f'Scale must be float or tuple of int, but got {type(scale)}')\n self.scale = scale\n self.keep_ratio = keep_ratio\n self.interpolation = interpolation\n self.lazy = lazy\n def __call__(self, results):\n \"\"\"Performs the Resize augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed"
+ },
+ {
+ "comment": "This code resizes images and keypoints based on the 'scale_factor' or scale provided. If 'scale_factor' is not already in the results, it initializes it with a default value of [1, 1]. The code then calculates new image width (new_w) and height (new_h) based on the scale or keep_ratio setting. It updates 'img_shape', 'keep_ratio', and 'scale_factor' in the results dictionary, and if not lazy, it resizes images and keypoints accordingly using the imresize function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":334-362",
+ "content": " to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n if 'scale_factor' not in results:\n results['scale_factor'] = np.array([1, 1], dtype=np.float32)\n img_h, img_w = results['img_shape']\n if self.keep_ratio:\n new_w, new_h = rescale_size((img_w, img_h), self.scale)\n else:\n new_w, new_h = self.scale\n self.scale_factor = np.array([new_w / img_w, new_h / img_h],\n dtype=np.float32)\n results['img_shape'] = (new_h, new_w)\n results['keep_ratio'] = self.keep_ratio\n results['scale_factor'] = results['scale_factor'] * self.scale_factor\n if not self.lazy:\n if 'imgs' in results:\n results['imgs'] = [\n imresize(\n img, (new_w, new_h), interpolation=self.interpolation)\n for img in results['imgs']\n ]\n if 'keypoint' in results:\n results['keypoint'] = results['keypoint'] * self.scale_factor"
+ },
+ {
+ "comment": "This code defines a class RandomRescale that performs random rescaling on images, maintaining the aspect ratio. It takes in a range for the short edge size, and an optional interpolation method. The class also has a lazy attribute to control whether the transformation is applied lazily or not. The class also includes an EntityBoxRescale function to rescale bounding boxes. The code ends with registering RandomRescale as a pipeline module.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":363-392",
+ "content": " else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Put Flip at last for now')\n lazyop['interpolation'] = self.interpolation\n #if 'gt_bboxes' in results:\n assert not self.lazy\n entity_box_rescale = EntityBoxRescale(self.scale_factor)\n results = entity_box_rescale(results)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'scale={self.scale}, keep_ratio={self.keep_ratio}, '\n f'interpolation={self.interpolation}, '\n f'lazy={self.lazy})')\n return repr_str\n@PIPELINES.register()\nclass RandomRescale:\n \"\"\"Randomly resize images so that the short_edge is resized to a specific\n size in a given range. The scale ratio is unchanged after resizing.\n \"\"\"\n def __init__(self, scale_range, interpolation='bilinear'):\n scale_range = eval(scale_range)\n self.scale_range = scale_range"
+ },
+ {
+ "comment": "This code defines a Resize augmentation transform with random scaling range, keeps aspect ratio and applies specified interpolation. It also includes a __repr__ method to provide class name, scale range and short edge value for debugging purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":394-422",
+ "content": " assert len(scale_range) == 2\n assert scale_range[0] < scale_range[1]\n assert np.all([x > 0 for x in scale_range])\n self.keep_ratio = True\n self.interpolation = interpolation\n def __call__(self, results):\n \"\"\"Performs the Resize augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n short_edge = np.random.randint(self.scale_range[0],\n self.scale_range[1] + 1)\n resize = Resize((-1, short_edge),\n keep_ratio=True,\n interpolation=self.interpolation,\n lazy=False)\n results = resize(results)\n results['short_edge'] = short_edge\n return results\n def __repr__(self):\n scale_range = self.scale_range\n repr_str = (f'{self.__class__.__name__}('\n f'scale_range=({scale_range[0]}, {scale_range[1]}), '"
+ },
+ {
+ "comment": "This code defines a Rescale augmentation class for image processing in the PaddleVideo framework. It resizes images so that the short edge length is within a specified range, while maintaining the aspect ratio. The interpolation method can be set to 'nearest' or 'bilinear'. This augmentation modifies the 'imgs', 'img_shape', 'keep_ratio', 'scale_factor', 'resize_size', and 'short_edge' keys in the results dictionary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":423-454",
+ "content": " f'interpolation={self.interpolation})')\n return repr_str\n@PIPELINES.register()\nclass Rescale:\n \"\"\"resize images so that the short_edge is resized to a specific\n size in a given range. The scale ratio is unchanged after resizing.\n Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified\n keys are \"imgs\", \"img_shape\", \"keep_ratio\", \"scale_factor\", \"resize_size\",\n \"short_edge\".\n Args:\n scale_range (tuple[int]): The range of short edge length. A closed\n interval.\n interpolation (str): Algorithm used for interpolation:\n \"nearest\" | \"bilinear\". Default: \"bilinear\".\n \"\"\"\n def __init__(self, scale_range, interpolation='bilinear'):\n scale_range = eval(scale_range)\n self.scale_range = scale_range\n self.keep_ratio = True\n self.interpolation = interpolation\n def __call__(self, results):\n \"\"\"Performs the Resize augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed"
+ },
+ {
+ "comment": "This code defines a Resize transform and a RandomCrop_v2 class for image processing pipelines in PaddleVideo. The Resize transform scales images within a specified range and with optional interpolation, while the RandomCrop_v2 performs square random cropping on images to a specific output size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":455-486",
+ "content": " to the next transform in pipeline.\n \"\"\"\n resize = Resize(\n self.scale_range,\n keep_ratio=True,\n interpolation=self.interpolation,\n lazy=False)\n results = resize(results)\n return results\n def __repr__(self):\n scale_range = self.scale_range\n repr_str = (f'{self.__class__.__name__}('\n f'scale_range=({scale_range[0]}, {scale_range[1]}), '\n f'interpolation={self.interpolation})')\n return repr_str\n@PIPELINES.register()\nclass RandomCrop_v2:\n \"\"\"Vanilla square random crop that specifics the output size.\n Required keys in results are \"imgs\" and \"img_shape\", added or\n modified keys are \"imgs\", \"lazy\"; Required keys in \"lazy\" are \"flip\",\n \"crop_bbox\", added or modified key is \"crop_bbox\".\n Args:\n size (int): The output size of the images.\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n def __init__(self, size, lazy=False):"
+ },
+ {
+ "comment": "This code defines a class with an __init__ method that checks if the input 'size' is an integer, and a __call__ method to perform random cropping. The __call__ method takes a dictionary of results and performs random cropping based on the size attribute of the class instance. It asserts that the size is less than or equal to the image height and width, then randomly selects y and x offsets for cropping. If 'crop_quadruple' is not in the results dictionary, it adds a new entry with initial values. Finally, it calculates ratios for cropping based on the input size and image dimensions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":487-516",
+ "content": " if not isinstance(size, int):\n raise TypeError(f'Size must be an int, but got {type(size)}')\n self.size = size\n self.lazy = lazy\n def __call__(self, results):\n \"\"\"Performs the RandomCrop augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n img_h, img_w = results['img_shape']\n assert self.size <= img_h and self.size <= img_w\n y_offset = 0\n x_offset = 0\n if img_h > self.size:\n y_offset = int(np.random.randint(0, img_h - self.size))\n if img_w > self.size:\n x_offset = int(np.random.randint(0, img_w - self.size))\n if 'crop_quadruple' not in results:\n results['crop_quadruple'] = np.array(\n [0, 0, 1, 1], # x, y, w, h\n dtype=np.float32)\n x_ratio, y_ratio = x_offset / img_w, y_offset / img_h\n w_ratio, h_ratio = self.size / img_w, self.size / img_h"
+ },
+ {
+ "comment": "This code segment is adjusting the crop quadruple, calculating a new crop bounding box, and updating the image shape based on provided offsets. It also handles lazy loading if enabled.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":518-543",
+ "content": " old_crop_quadruple = results['crop_quadruple']\n old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]\n old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]\n new_crop_quadruple = [\n old_x_ratio + x_ratio * old_w_ratio,\n old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,\n h_ratio * old_x_ratio\n ]\n results['crop_quadruple'] = np.array(\n new_crop_quadruple, dtype=np.float32)\n new_h, new_w = self.size, self.size\n results['crop_bbox'] = np.array(\n [x_offset, y_offset, x_offset + new_w, y_offset + new_h])\n results['img_shape'] = (new_h, new_w)\n if not self.lazy:\n results['imgs'] = [\n img[y_offset:y_offset + new_h, x_offset:x_offset + new_w]\n for img in results['imgs']\n ]\n else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Put Flip at last for now')"
+ },
+ {
+ "comment": "This code section is responsible for applying augmentations to video frames, specifically crop and flip operations. It takes input parameters such as image size, whether it should be applied lazily or not, and the direction of flipping (if applicable). The code adjusts the crop region based on the specified offset values and stores them in the 'crop_bbox' field of the lazy operation dictionary. Additionally, if there are entity boxes present, they will also be processed according to the applied crop and flip operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":545-570",
+ "content": " # record crop_bbox in lazyop dict to ensure only crop once in Fuse\n lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']\n left = x_offset * (lazy_right - lazy_left) / img_w\n right = (x_offset + new_w) * (lazy_right - lazy_left) / img_w\n top = y_offset * (lazy_bottom - lazy_top) / img_h\n bottom = (y_offset + new_h) * (lazy_bottom - lazy_top) / img_h\n lazyop['crop_bbox'] = np.array(\n [(lazy_left + left), (lazy_top + top), (lazy_left + right),\n (lazy_top + bottom)],\n dtype=np.float32)\n # Process entity boxes\n if 'gt_bboxes' in results:\n assert not self.lazy\n entity_box_crop = EntityBoxCrop(results['crop_bbox'])\n results = entity_box_crop(results)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}(size={self.size}, '\n f'lazy={self.lazy})')\n return repr_str\ndef imflip_(img, direction='horizontal'):"
+ },
+ {
+ "comment": "The provided code contains three functions: `inplace_flip`, `iminvert`, and a pipeline class called `Flip`. \n\n`inplace_flip` takes an image (`ndarray`) and the direction for flipping (horizontal, vertical or diagonal), asserts that the direction is valid, and returns the flipped image in-place. If the direction is horizontal, it uses `cv2.flip()` with parameter 1 to flip horizontally; if the direction is vertical, it uses `cv2.flip()` with parameter 0 to flip vertically; if the direction is diagonal, it uses `cv2.flip()` with parameter -1 for diagonal flipping.\n\n`iminvert` takes an image (`ndarray`) and returns its negative (inverted) version by subtracting the original image from a numpy array of full value 255 (the maximum possible value for an 8-bit image). This effectively reverses all pixel intensities in the image.\n\nThe `Flip` class is a pipeline module that flips the input images with a certain probability. It requires keys \"imgs\", \"img_shape\", and \"modality\" (although it does not modify them) and adds no new keys.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":571-608",
+ "content": " \"\"\"Inplace flip an image horizontally or vertically.\n Args:\n img (ndarray): Image to be flipped.\n direction (str): The flip direction, either \"horizontal\" or\n \"vertical\" or \"diagonal\".\n Returns:\n ndarray: The flipped image (inplace).\n \"\"\"\n assert direction in ['horizontal', 'vertical', 'diagonal']\n if direction == 'horizontal':\n return cv2.flip(img, 1, img)\n elif direction == 'vertical':\n return cv2.flip(img, 0, img)\n else:\n return cv2.flip(img, -1, img)\ndef iminvert(img):\n \"\"\"Invert (negate) an image.\n Args:\n img (ndarray): Image to be inverted.\n Returns:\n ndarray: The inverted image.\n \"\"\"\n return np.full_like(img, 255) - img\n@PIPELINES.register()\nclass Flip:\n \"\"\"Flip the input images with a probability.\n Reverse the order of elements in the given imgs with a specific direction.\n The shape of the imgs is preserved, but the elements are reordered.\n Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified"
+ },
+ {
+ "comment": "This code defines a Flip augmentation class for image processing in PaddleVideo. It takes flip ratio, direction (horizontal or vertical), and lazy operation as parameters. The flip_ratio determines the probability of applying the flip transformation, while direction specifies whether to flip horizontally or vertically. If the 'lazy' parameter is True, the transformation will be applied lazily. This augmentation should be placed after cropping/reshaping transformations for proper crop_quadruple calculation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":609-630",
+ "content": " keys are \"imgs\", \"lazy\" and \"flip_direction\". Required keys in \"lazy\" is\n None, added or modified key are \"flip\" and \"flip_direction\". The Flip\n augmentation should be placed after any cropping / reshaping augmentations,\n to make sure crop_quadruple is calculated properly.\n Args:\n flip_ratio (float): Probability of implementing flip. Default: 0.5.\n direction (str): Flip imgs horizontally or vertically. Options are\n \"horizontal\" | \"vertical\". Default: \"horizontal\".\n lazy (bool): Determine whether to apply lazy operation. Default: False.\n \"\"\"\n _directions = ['horizontal', 'vertical']\n def __init__(self, flip_ratio=0.5, direction='horizontal', lazy=False):\n if direction not in self._directions:\n raise ValueError(f'Direction {direction} is not supported. '\n f'Currently support ones are {self._directions}')\n self.flip_ratio = flip_ratio\n self.direction = direction\n self.lazy = lazy\n def __call__(self, results):"
+ },
+ {
+ "comment": "The code snippet performs a Flip augmentation on images, randomly flipping them horizontally based on a given flip ratio. It also sets the 'flip' and 'flip_direction' keys in the results dictionary. If the 'lazy' option is not used (self.lazy), it iterates through the images, applying the flip transformation if necessary. It also handles 'gt_bboxes' if they exist in the results dictionary, ensuring horizontal flips are applied correctly without any issues.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":631-659",
+ "content": " \"\"\"Performs the Flip augmentation.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n _init_lazy_if_proper(results, self.lazy)\n flip = np.random.rand() < self.flip_ratio\n results['flip'] = flip\n results['flip_direction'] = self.direction\n if not self.lazy:\n if flip:\n for i, img in enumerate(results['imgs']):\n imflip_(img, self.direction)\n lt = len(results['imgs'])\n else:\n results['imgs'] = list(results['imgs'])\n else:\n lazyop = results['lazy']\n if lazyop['flip']:\n raise NotImplementedError('Use one Flip please')\n lazyop['flip'] = flip\n lazyop['flip_direction'] = self.direction\n if 'gt_bboxes' in results and flip:\n assert not self.lazy and self.direction == 'horizontal'\n entity_box_flip = EntityBoxFlip(results['img_shape'])"
+ },
+ {
+ "comment": "This code contains a class for image augmentation, including flip ratio and direction, with a method to normalize an image using mean and std values. It also includes an inplace normalization function that converts BGR to RGB if necessary. The `__repr__` method returns a string representation of the class attributes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":660-692",
+ "content": " results = entity_box_flip(results)\n return results\n def __repr__(self):\n repr_str = (\n f'{self.__class__.__name__}('\n f'flip_ratio={self.flip_ratio}, direction={self.direction}, '\n f'lazy={self.lazy})')\n return repr_str\ndef imnormalize_(img, mean, std, to_rgb=True):\n \"\"\"Inplace normalize an image with mean and std.\n Args:\n img (ndarray): Image to be normalized.\n mean (ndarray): The mean to be used for normalize.\n std (ndarray): The std to be used for normalize.\n to_rgb (bool): Whether to convert to rgb.\n Returns:\n ndarray: The normalized image.\n \"\"\"\n # cv2 inplace normalization does not accept uint8\n assert img.dtype != np.uint8\n mean = np.float64(mean.reshape(1, -1))\n stdinv = 1 / np.float64(std.reshape(1, -1))\n if to_rgb:\n cv2.cvtColor(img, cv2.COLOR_BGR2RGB, img) # inplace\n cv2.subtract(img, mean, img) # inplace\n cv2.multiply(img, stdinv, img) # inplace\n return img"
+ },
+ {
+ "comment": "This code defines a class called \"Normalize\" that normalizes images based on given mean and std values. It can also convert channels from RGB to BGR if necessary. Additionally, it adjusts flow magnitude when modality is 'Flow' with an optional adjust_magnitude parameter. The class requires keys \"imgs\", \"img_shape\", \"modality\" with additional keys \"imgs\" and \"img_norm_cfg\" being added or modified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":695-719",
+ "content": "@PIPELINES.register()\nclass Normalize:\n \"\"\"Normalize images with the given mean and std value.\n Required keys are \"imgs\", \"img_shape\", \"modality\", added or modified\n keys are \"imgs\" and \"img_norm_cfg\". If modality is 'Flow', additional\n keys \"scale_factor\" is required\n Args:\n mean (Sequence[float]): Mean values of different channels.\n std (Sequence[float]): Std values of different channels.\n to_bgr (bool): Whether to convert channels from RGB to BGR.\n Default: False.\n adjust_magnitude (bool): Indicate whether to adjust the flow magnitude\n on 'scale_factor' when modality is 'Flow'. Default: False.\n \"\"\"\n def __init__(self, mean, std, to_bgr=False, adjust_magnitude=False):\n if not isinstance(mean, Sequence):\n raise TypeError(\n f'Mean must be list, tuple or np.ndarray, but got {type(mean)}')\n if not isinstance(std, Sequence):\n raise TypeError(\n f'Std must be list, tuple or np.ndarray, but got {type(std)}')"
+ },
+ {
+ "comment": "This code defines an augmentation pipeline for image normalization in AVA. It initializes mean, std, and to_bgr values, and then applies the normalization transformation to each input image. The normalized images are stored in 'imgs' and the configuration is saved in 'img_norm_cfg'. The __repr__ method provides a string representation of the object's state.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/augmentations_ava.py\":721-748",
+ "content": " self.mean = np.array(mean, dtype=np.float32)\n self.std = np.array(std, dtype=np.float32)\n self.to_bgr = to_bgr\n self.adjust_magnitude = adjust_magnitude\n def __call__(self, results):\n n = len(results['imgs'])\n h, w, c = results['imgs'][0].shape\n imgs = np.empty((n, h, w, c), dtype=np.float32)\n for i, img in enumerate(results['imgs']):\n imgs[i] = img\n for img in imgs:\n imnormalize_(img, self.mean, self.std, self.to_bgr)\n results['imgs'] = imgs\n results['img_norm_cfg'] = dict(\n mean=self.mean, std=self.std, to_bgr=self.to_bgr)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'mean={self.mean}, '\n f'std={self.std}, '\n f'to_bgr={self.to_bgr}, '\n f'adjust_magnitude={self.adjust_magnitude})')\n return repr_str"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/23cc0f4d-18b7-4a32-a685-092f8cd2c6b8.json b/docs/doc/23cc0f4d-18b7-4a32-a685-092f8cd2c6b8.json
new file mode 100644
index 000000000..9d815c764
--- /dev/null
+++ b/docs/doc/23cc0f4d-18b7-4a32-a685-092f8cd2c6b8.json
@@ -0,0 +1,15 @@
+{
+ "summary": "FastRCNN is a two-stage object detection class inheriting from TwoStageDetector, created with specified head, train and test configurations, and optional pretrained weights.",
+ "details": [
+ {
+ "comment": "Defines the FastRCNN class, a two-stage detector that inherits from TwoStageDetector. It takes backbone, head, train_cfg, test_cfg, neck, and pretrained as parameters for object detection.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/detectors/fast_rcnn.py\":0-29",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .two_stage import TwoStageDetector\nfrom ...registry import DETECTORS\n@DETECTORS.register()\nclass FastRCNN(TwoStageDetector):\n def __init__(self,\n backbone,\n head=None,\n train_cfg=None,\n test_cfg=None,\n neck=None,\n pretrained=None):\n super(FastRCNN, self).__init__(\n backbone=backbone,\n neck=neck,"
+ },
+ {
+ "comment": "Creates a Fast R-CNN detector with specified head, train and test configurations, and optionally pretrained weights.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/detectors/fast_rcnn.py\":30-33",
+ "content": " roi_head=head,\n train_cfg=train_cfg,\n test_cfg=test_cfg,\n pretrained=pretrained)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/24375566-77e4-4025-9c4f-d6ce173bd1c8.json b/docs/doc/24375566-77e4-4025-9c4f-d6ce173bd1c8.json
new file mode 100644
index 000000000..ef65b3bdd
--- /dev/null
+++ b/docs/doc/24375566-77e4-4025-9c4f-d6ce173bd1c8.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code defines a localizer model for PaddleVideo with forward network and methods for training, validating, testing, and inferring. It uses input data to predict bounding boxes, start position, and end position while calculating loss using ground truth values.",
+ "details": [
+ {
+ "comment": "This code is part of the PaddleVideo library and defines a BMNLocalizer class, which is a localization framework. It includes a forward_net method for calling the backbone's forward function and a train_step method for handling training steps with input data. The gt_iou_map, gt_start, and gt_end are provided as part of the data batch to be used in the training step.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/localizers/bmn_localizer.py\":0-35",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import LOCALIZERS\nfrom .base import BaseLocalizer\nimport paddle\n@LOCALIZERS.register()\nclass BMNLocalizer(BaseLocalizer):\n \"\"\"BMN Localization framework\n \"\"\"\n def forward_net(self, imgs):\n \"\"\"Call backbone forward.\n \"\"\"\n preds = self.backbone(imgs)\n return preds\n def train_step(self, data_batch):\n \"\"\"Training step.\n \"\"\"\n x_data = data_batch[0]\n gt_iou_map = data_batch[1]\n gt_start = data_batch[2]\n gt_end = data_batch[3]\n gt_iou_map.stop_gradient = True"
+ },
+ {
+ "comment": "This code defines a localizer model for PaddleVideo. It includes functions for training, validating, testing, and inferring steps. The localizer has a forward network which takes input data and returns predictions for bounding boxes (pred_bm), start position (pred_start), and end position (pred_end). Loss is calculated using the provided ground truth values (gt_iou_map, gt_start, gt_end) and averaged over the batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/localizers/bmn_localizer.py\":36-68",
+ "content": " gt_start.stop_gradient = True\n gt_end.stop_gradient = True\n # call Model forward\n pred_bm, pred_start, pred_end = self.forward_net(x_data)\n # call Loss forward\n loss = self.loss(pred_bm, pred_start, pred_end, gt_iou_map, gt_start,\n gt_end)\n avg_loss = paddle.mean(loss)\n loss_metrics = dict()\n loss_metrics['loss'] = avg_loss\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"Validating setp.\n \"\"\"\n return self.train_step(data_batch)\n def test_step(self, data_batch):\n \"\"\"Test step.\n \"\"\"\n x_data = data_batch[0]\n pred_bm, pred_start, pred_end = self.forward_net(x_data)\n return pred_bm, pred_start, pred_end\n def infer_step(self, data_batch):\n \"\"\"Infer step\n \"\"\"\n x_data = data_batch[0]\n # call Model forward\n pred_bm, pred_start, pred_end = self.forward_net(x_data)\n return pred_bm, pred_start, pred_end"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/249d4907-6cf9-47a6-8199-4eb5c9f53708.json b/docs/doc/249d4907-6cf9-47a6-8199-4eb5c9f53708.json
new file mode 100644
index 000000000..9a892a02a
--- /dev/null
+++ b/docs/doc/249d4907-6cf9-47a6-8199-4eb5c9f53708.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code imports libraries, initializes an experiment, defines functions for training a video analysis model, handles command-line arguments and ensures checkpoints are saved before running the training process.",
+ "details": [
+ {
+ "comment": "The code imports necessary libraries, modules and packages for the PaddleVideo project. It also handles copyright and license information, sets seeds to ensure reproducibility, and includes utility functions for logging, model training, and data loading. It defines a Trainer class and an evaluation function, as well as parsing configuration files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/train.py\":0-34",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport os\nimport time\nimport copy\nimport socket\nimport paddle\nimport argparse\nimport warnings\nimport numpy as np\nimport model.loss as module_loss\nimport model.model as module_arch\nimport model.metric as module_metric\nimport data_loader.data_loaders as module_data\nfrom pathlib import Path\nfrom utils import set_seeds\nfrom trainer import Trainer\nfrom test import evaluation\nfrom mergedeep import merge, Strategy\nfrom parse_config import ConfigParser\nfrom logger.log_parser import log_summary\nfrom utils import compute_dims, compute_trn_config"
+ },
+ {
+ "comment": "This code snippet defines a function `run_exp()` that initializes an experiment. It sets the random seed, initializes the model (arch) and data loaders based on the given configuration. The seeds are obtained from command line arguments, and for each seed, it logs information about the setting and proceeds with the experiment initialization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/train.py\":36-66",
+ "content": "def run_exp(config):\n warnings.filterwarnings('ignore')\n logger = config.get_logger('train')\n expert_dims, raw_input_dims = compute_dims(config, logger)\n trn_config = compute_trn_config(config)\n if config._args.group_seed:\n seeds = [int(config._args.group_seed)]\n else:\n seeds = [int(x) for x in config._args.seeds.split(\",\")]\n for ii, seed in enumerate(seeds):\n tic = time.time()\n logger.info(f\"{ii + 1}/{len(seeds)} Setting experiment random seed to {seed}\")\n set_seeds(seed)\n config[\"seed\"] = seed\n model = config.init(\n name='arch',\n module=module_arch,\n expert_dims=expert_dims,\n text_dim=config[\"experts\"][\"text_dim\"],\n ce_shared_dim=config[\"experts\"].get(\"ce_shared_dim\", None),\n feat_aggregation=config[\"data_loader\"][\"args\"][\"feat_aggregation\"],\n )\n logger.info(model)\n data_loaders = config.init(\n name='data_loader',\n module=module_data,"
+ },
+ {
+ "comment": "Initializing a model with specific configurations and defining the loss function, metrics to track progress, learning rate scheduler for dynamic adjustments, and an optimizer (AdamW) to update model parameters. Also creating a Trainer instance which combines all these components for training the model on given data loaders.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/train.py\":67-91",
+ "content": " logger=logger,\n raw_input_dims=raw_input_dims,\n text_feat=config[\"experts\"][\"text_feat\"],\n text_dim=config[\"experts\"][\"text_dim\"],\n text_agg=config[\"experts\"][\"text_agg\"],\n use_zeros_for_missing=config[\"experts\"].get(\"use_zeros_for_missing\", False),\n eval_only=False,\n )\n loss = config.init(name=\"loss\", module=module_loss)\n metrics = [getattr(module_metric, met) for met in config['metrics']]\n lr_scheduler = paddle.optimizer.lr.StepDecay(learning_rate=0.0001, step_size=5, gamma=0.9)\n optimizer = paddle.optimizer.AdamW(learning_rate=lr_scheduler, weight_decay=1e-4, parameters=model.parameters(), grad_clip=paddle.nn.ClipGradByGlobalNorm(2))\n trainer = Trainer(\n model,\n loss,\n metrics,\n optimizer,\n config=config,\n data_loaders=data_loaders,\n lr_scheduler=lr_scheduler,\n mini_train=config._args.mini_train,\n visualizer=None,"
+ },
+ {
+ "comment": "This code sets up a trainer with specified configuration, trains the model, saves the best model at 'best_model_path', logs training duration, reports relevant statistics if multiple runs were conducted, and prints the log file location.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/train.py\":92-114",
+ "content": " val_freq=config[\"trainer\"].get(\"val_freq\", 1),\n force_cpu_val=config.get(\"force_cpu_val\", False),\n skip_first_n_saves=config[\"trainer\"].get(\"skip_first_n_saves\", 0),\n include_optim_in_save_model=config[\"trainer\"].get(\"include_optim_in_save_model\", 1),\n cache_targets=set(config.get(\"cache_targets\", [])),\n )\n trainer.train()\n best_model_path = config.save_dir / \"trained_model.pdparams\"\n duration = time.strftime('%Hh%Mm%Ss', time.gmtime(time.time() - tic))\n logger.info(f\"Training took {duration}\")\n # If multiple runs were conducted, report relevant statistics\n if len(seeds) > 1:\n log_summary(\n logger=logger,\n log_path=config.log_path,\n eval_mode=config[\"eval_mode\"],\n fixed_num_epochs=config[\"trainer\"][\"epochs\"],\n )\n print(f\"Log file stored at {config.log_path}\")\n # Report the location of the \"best\" model of the final seeded run (here\n # \"best\" corresponds to the model with the highest geometric mean over the"
+ },
+ {
+ "comment": "This code defines the command-line arguments for the training script of a video analysis application. The arguments include config file path, resuming from a previous model, mini-batch training option, grouping experiments by ID, disabling workers, refreshing LRU cache, training a single epoch, purging existing experiments, and debugging options.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/train.py\":115-132",
+ "content": " # R@1, R@5 and R@10 metrics when a validation set is used, or simply the final\n # epoch of training for fixed-length schedules).\n print(f\"The best performing model can be found at {str(best_model_path)}\")\ndef main():\n args = argparse.ArgumentParser(description='Main entry point for training')\n args.add_argument('--config', help='config file path')\n args.add_argument('--resume', help='path to latest model (default: None)')\n args.add_argument('--mini_train', action=\"store_true\")\n args.add_argument('--group_id', help=\"if supplied, group these experiments\")\n args.add_argument('--disable_workers', action=\"store_true\")\n args.add_argument('--refresh_lru_cache', action=\"store_true\")\n args.add_argument('--train_single_epoch', action=\"store_true\")\n args.add_argument('--purge_exp_dir', action=\"store_true\",\n help=\"remove all previous experiments with the given config\")\n args.add_argument(\"--dbg\", default=\"ipdb.set_trace\")\n args.add_argument(\"--custom_args\", help=\"qualified key,val pairs\")"
+ },
+ {
+ "comment": "This code is parsing command-line arguments for seeds, setting environment variables, and asserting that the number of training epochs is greater than the save period to ensure checkpoints are saved. The function run_exp is then called with these configuration settings, and the main function is executed if the script is run directly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/train.py\":134-150",
+ "content": " # Seeds can either be passed directly as a comma separated list at the command line,\n # or individually for separate experiments as a group (used for slurm experiments)\n seed_args = args.add_mutually_exclusive_group()\n seed_args.add_argument('--seeds', default=\"0\", help=\"comma separated list of seeds\")\n seed_args.add_argument('--group_seed', help=\"seed for group member\")\n args = ConfigParser(args)\n os.environ[\"PYTHONBREAKPOINT\"] = args._args.dbg\n args[\"data_loader\"][\"args\"][\"refresh_lru_cache\"] = args._args.refresh_lru_cache\n msg = (f\"Expected the number of training epochs ({args['trainer']['epochs']})\"\n f\"to exceed the save period ({args['trainer']['save_period']}), otherwise\"\n \" no checkpoints will be saved.\")\n assert args[\"trainer\"][\"epochs\"] >= args[\"trainer\"][\"save_period\"], msg\n run_exp(config=args)\nif __name__ == '__main__':\n main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/24e4e098-d91f-4056-ab8a-129403184220.json b/docs/doc/24e4e098-d91f-4056-ab8a-129403184220.json
new file mode 100644
index 000000000..adbb80f5d
--- /dev/null
+++ b/docs/doc/24e4e098-d91f-4056-ab8a-129403184220.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code provides a logger class for PaddleVideo's Video Quality Assessment app, enabling logging for distributed apps with rank-based output to file or console. It initializes loggers and disables log event propagation when verbosity level is set to \"DEBUG\".",
+ "details": [
+ {
+ "comment": "This code snippet is from PaddleVideo's Video Quality Assessment application, and it contains a logger class for logging messages. The logger imports necessary modules like logging, os, sys, datetime, and ParallelEnv, along with defining color codes and a function for colored output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/logger.py\":0-39",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport logging\nimport os\nimport sys\nimport datetime\nfrom paddle.distributed import ParallelEnv\nColor = {\n 'RED': '\\033[31m',\n 'HEADER': '\\033[35m', # deep purple\n 'PURPLE': '\\033[95m', # purple\n 'OKBLUE': '\\033[94m',\n 'OKGREEN': '\\033[92m',\n 'WARNING': '\\033[93m',\n 'FAIL': '\\033[91m',\n 'ENDC': '\\033[0m'\n}\ndef coloring(message, color=\"OKGREEN\"):\n \"\"\"coloring\"\"\"\n assert color in Color.keys()"
+ },
+ {
+ "comment": "Function `setup_logger` initializes the paddlevideo logger and sets its verbosity level to \"INFO\". It takes optional arguments for output file name or directory, and root module name. If the verbosity level is set to \"DEBUG\", the logger will have a lower threshold for logging messages. The function also disables propagation of log events to the root logger.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/logger.py\":40-73",
+ "content": " if os.environ.get('COLORING', True):\n return Color[color] + str(message) + Color[\"ENDC\"]\n else:\n return message\nlogger_initialized = []\ndef setup_logger(output=None, name=\"paddlevideo\", level=\"INFO\"):\n \"\"\"\n Initialize the paddlevideo logger and set its verbosity level to \"INFO\".\n Args:\n output (str): a file name or a directory to save log. If None, will not save log file.\n If ends with \".txt\" or \".log\", assumed to be a file name.\n Otherwise, logs will be saved to `output/log.txt`.\n name (str): the root module name of this logger\n Returns:\n logging.Logger: a logger\n \"\"\"\n def time_zone(sec, fmt):\n real_time = datetime.datetime.now()\n return real_time.timetuple()\n logging.Formatter.converter = time_zone\n logger = logging.getLogger(name)\n if level == \"INFO\":\n logger.setLevel(logging.INFO)\n elif level==\"DEBUG\":\n logger.setLevel(logging.DEBUG)\n logger.propagate = False\n if level == \"DEBUG\":\n plain_formatter = logging.Formatter("
+ },
+ {
+ "comment": "This code sets up logging configuration for a distributed application. It uses a logger to handle log messages, and based on the local rank of each process, it determines whether to output logs to standard out, standard err, or a file. If no output is provided, it defaults to a \".log.txt\" file in the specified directory. The log files for different ranks are distinguished by appending the rank number. If the directory doesn't exist, it creates one before writing the logs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/logger.py\":74-102",
+ "content": " \"[%(asctime)s] %(name)s %(levelname)s: %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n else:\n plain_formatter = logging.Formatter(\n \"[%(asctime)s] %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n # stdout logging: master only\n local_rank = ParallelEnv().local_rank\n if local_rank == 0:\n ch = logging.StreamHandler(stream=sys.stdout)\n ch.setLevel(logging.DEBUG)\n formatter = plain_formatter\n ch.setFormatter(formatter)\n logger.addHandler(ch)\n # file logging: all workers\n if output is not None:\n if output.endswith(\".txt\") or output.endswith(\".log\"):\n filename = output\n else:\n filename = os.path.join(output, \".log.txt\")\n if local_rank > 0:\n filename = filename + \".rank{}\".format(local_rank)\n # PathManager.mkdirs(os.path.dirname(filename))\n os.makedirs(os.path.dirname(filename), exist_ok=True)\n # fh = logging.StreamHandler(_cached_log_stream(filename)\n fh = logging.FileHandler(filename, mode='a')"
+ },
+ {
+ "comment": "This function `get_logger` sets up a logger with the given name. If the logger has already been initialized, it simply returns the existing logger. Otherwise, it calls `setup_logger` to initialize the logger with the given name and optional output. The logger is configured to handle debug level messages using plain formatter and this configuration is appended to the list of initialized loggers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/logger.py\":103-116",
+ "content": " fh.setLevel(logging.DEBUG)\n fh.setFormatter(plain_formatter)\n logger.addHandler(fh)\n logger_initialized.append(name)\n return logger\ndef get_logger(name, output=None):\n \"\"\"get logger\"\"\"\n logger = logging.getLogger(name)\n if name in logger_initialized:\n return logger\n return setup_logger(name=name, output=name)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/25008972-aec6-44bf-b83e-21babe2c3e0c.json b/docs/doc/25008972-aec6-44bf-b83e-21babe2c3e0c.json
new file mode 100644
index 000000000..6fc3c84ad
--- /dev/null
+++ b/docs/doc/25008972-aec6-44bf-b83e-21babe2c3e0c.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The code defines a ResNeXt-101 model in PaddlePaddle, including downsample and residual blocks, BottleneckBlock, performs convolutions, activation, max pooling on input image.",
+ "details": [
+ {
+ "comment": "This code defines a ConvBNLayer class in PaddlePaddle, which is a convolution-batch normalization layer. It takes inputs like num_channels, num_filters, filter_size, stride, padding, dilation, groups, padding_mode, weight_attr, bias_attr, and name.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":0-30",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom functools import partial\nimport paddle\nclass ConvBNLayer(paddle.nn.Layer):\n def __init__(self,\n num_channels,\n num_filters,\n filter_size,\n stride=1,\n padding=0,\n dilation=1,\n groups=1,\n padding_mode='zeros',\n weight_attr=None,\n bias_attr=None,\n name=None,"
+ },
+ {
+ "comment": "This code defines a ConvBNLayer class with specified parameters for convolutional and batch normalization layers. The convolutional layer uses Kaiming Normal initialization, while the batch normalization layer has fixed scales and offsets initialized to 1 and 0 respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":31-54",
+ "content": " data_format=\"NCDHW\"):\n super(ConvBNLayer, self).__init__()\n self._conv = paddle.nn.Conv3D(\n in_channels=num_channels,\n out_channels=num_filters,\n kernel_size=filter_size,\n stride=stride,\n padding=padding,\n dilation=dilation,\n groups=groups,\n padding_mode=padding_mode,\n weight_attr=paddle.ParamAttr(initializer=paddle.nn.initializer.KaimingNormal(\n fan_in=num_filters * filter_size * filter_size), name=name+'_weights'),\n bias_attr=bias_attr,\n data_format=data_format)\n bn_name = \"bn_\" + name\n self._batch_norm = paddle.nn.BatchNorm3D(\n num_filters,\n momentum=0.9,\n epsilon=1e-05,\n weight_attr=paddle.ParamAttr(initializer=paddle.nn.initializer.Constant(\n 1.), name=bn_name + '_scale'),\n bias_attr=paddle.ParamAttr(initializer=paddle.nn.initializer.Constant(\n 0.), name=bn_name + '_offset'),"
+ },
+ {
+ "comment": "This code defines a BottleneckBlock class and a downsample function for the ResNeXt101 model in PaddlePaddle. The BottleneckBlock has an expansion factor of 2 and uses ConvBNLayer for convolution and batch normalization. The downsample function performs average pooling and concatenation to perform downsampling.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":55-81",
+ "content": " data_format=data_format)\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n return y\ndef _downsample_basic_block(self, x, planes, stride):\n out = paddle.nn.functional.avg_pool3d(x, kernel_size=1, stride=stride)\n shape = out.shape\n zero_pads = paddle.zeros(shape=[shape[0], planes - shape[1], shape[2], shape[3], shape[4]],\n dtype='float32')\n out = paddle.concat(x=[out, zero_pads], axis=1)\nclass BottleneckBlock(paddle.nn.Layer):\n expansion = 2\n def __init__(self, inplanes, planes, cardinality, stride=1, downsample=None, name=None):\n super(BottleneckBlock, self).__init__()\n mid_planes = cardinality * int(planes / 32)\n self.conv0 = ConvBNLayer(\n inplanes, mid_planes, filter_size=1, bias_attr=False, name=name+'_branch2a')\n self.conv1 = ConvBNLayer(mid_planes, mid_planes, filter_size=3, stride=stride,\n padding=1, groups=cardinality, bias_attr=False, name=name+'_branch2b')"
+ },
+ {
+ "comment": "This code defines a ResNeXt model. The class ResNeXt has an initialization that sets inplanes to 64 and inherits from paddle.nn.Layer. It contains a convolution layer (conv) with 3 input channels, 64 output channels, filter size of 7, and stride of (1,2,2). The class also includes a ResNet-style residual block as a member variable named 'block'. It has layers, shortcut_type (defaults to B), and cardinality parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":82-121",
+ "content": " self.conv2 = ConvBNLayer(mid_planes, planes * self.expansion,\n filter_size=1, bias_attr=False, name=name+'_branch2c')\n self.downsample = downsample\n self.stride = stride\n self.relu = paddle.nn.ReLU()\n def forward(self, x):\n residual = x\n out = self.conv0(x)\n out = self.relu(out)\n out = self.conv1(out)\n out = self.relu(out)\n out = self.conv2(out)\n if self.downsample is not None:\n residual = self.downsample(x)\n out += residual\n out = self.relu(out)\n return out\nclass ResNeXt(paddle.nn.Layer):\n def __init__(self,\n block,\n layers,\n shortcut_type='B',\n cardinality=32):\n self.inplanes = 64\n super(ResNeXt, self).__init__()\n self.conv = ConvBNLayer(\n 3,\n 64,\n filter_size=7,\n stride=(1, 2, 2),\n padding=(3, 3, 3),\n bias_attr=False,\n name=\"res_conv1\""
+ },
+ {
+ "comment": "The code defines a ResNext101 backbone for a deep learning model. It includes a ReLU activation function and max pooling operation, followed by four residual layers (layer1 to layer4) with varying numbers of planes (256, 512, 1024 respectively). The _make_layer method is used to create the layers, with options for downsampling and varying expansion rates.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":122-147",
+ "content": " )\n self.relu = paddle.nn.ReLU()\n self.maxpool = paddle.nn.MaxPool3D(kernel_size=(3, 3, 3), stride=2, padding=1)\n self.layer1 = self._make_layer(block, 128, layers[0], shortcut_type,\n cardinality, stride=1, name='layer1')\n self.layer2 = self._make_layer(\n block, 256, layers[1], shortcut_type, cardinality, stride=2, name='layer2')\n self.layer3 = self._make_layer(\n block, 512, layers[2], shortcut_type, cardinality, stride=2, name='layer3')\n self.layer4 = self._make_layer(\n block, 1024, layers[3], shortcut_type, cardinality, stride=2, name='layer4')\n self.avgpool = paddle.nn.AvgPool3D((2, 1, 1), stride=1, exclusive=False)\n def _make_layer(self,\n block,\n planes,\n blocks,\n shortcut_type,\n cardinality,\n stride=1,\n name=None):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:"
+ },
+ {
+ "comment": "This code defines a ResNeXt-101 model, implementing its downsample and residual blocks. It takes an input image, performs convolutions, applies ReLU activation, and max pooling before passing through the specified number of residual blocks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":148-175",
+ "content": " if shortcut_type == 'A':\n downsample = partial(self._downsample_basic_block,\n planes=planes * block.expansion,\n stride=stride)\n else:\n downsample = ConvBNLayer(\n self.inplanes,\n planes * block.expansion,\n 1,\n stride=stride,\n bias_attr=False,\n name=name+'downsample'\n )\n layers = []\n layers.append(\n block(self.inplanes, planes, cardinality, stride, downsample, name=name+'_downsample'))\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(block(self.inplanes, planes,\n cardinality, name=name+'_res_block'+str(i)))\n return paddle.nn.Sequential(*layers)\n def forward(self, x):\n x = self.conv(x)\n x = self.relu(x)\n x = self.maxpool(x)\n x = self.layer1(x)"
+ },
+ {
+ "comment": "The ResNext101 function constructs a ResNeXt-101 model using BottleneckBlock and the specified block configurations. It applies the layer2, layer3, and layer4 operations to x before returning the result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnext101.py\":176-186",
+ "content": " x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n return x\ndef ResNext101():\n \"\"\"Constructs a ResNext-101 model.\n \"\"\"\n model = ResNeXt(BottleneckBlock, [3, 4, 23, 3])\n return model"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/254f5728-b6b4-46fc-9bc5-c8740d72e8a4.json b/docs/doc/254f5728-b6b4-46fc-9bc5-c8740d72e8a4.json
new file mode 100644
index 000000000..2629d7934
--- /dev/null
+++ b/docs/doc/254f5728-b6b4-46fc-9bc5-c8740d72e8a4.json
@@ -0,0 +1,30 @@
+{
+ "summary": "This code creates PaddleVideo dataset loaders and sets up signal handlers for graceful termination of a process group upon receiving SIGINT or SIGTERM signals.",
+ "details": [
+ {
+ "comment": "This code snippet is a part of the PaddleVideo library and contains a function named build_pipeline. It imports various modules, defines a logger for logging purposes, and uses a function called build from utils. This function seems to be building some kind of pipeline based on the provided configuration (cfg). The purpose of this pipeline might be to process data or prepare it for model training in the context of PaddleVideo.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/loader/builder.py\":0-30",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport signal\nimport os\nimport paddle\nfrom paddle.io import BatchSampler, DataLoader, DistributedBatchSampler\nfrom .pipelines.compose import Compose\nfrom .registry import DATASETS, PIPELINES, DATALOADERS, BATCH_SAMPLERS, SAMPLERS\nfrom ..utils import get_logger\nfrom ..utils.build_utils import build\nimport numpy as np\nlogger = get_logger(\"paddlevideo\")\ndef build_pipeline(cfg):\n \"\"\"Build pipeline.\n Args:\n cfg (dict): root config dict."
+ },
+ {
+ "comment": "This code defines several functions to build different components for a dataset loader. The main function is `build_dataset` which takes a configuration dictionary and returns a dataset object after building the pipeline, dataset, sampler, and dataloader as per the given configuration. It uses other helper functions like `build_pipeline`, `build_sampler`, `build_batch_pipeline`, and `build_custom_dataloader` to build these components.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/loader/builder.py\":31-79",
+ "content": " \"\"\"\n if cfg == None:\n return\n return Compose(cfg)\ndef build_dataset(cfg):\n \"\"\"Build dataset.\n Args:\n cfg (dict): root config dict.\n Returns:\n dataset: dataset.\n \"\"\"\n # XXX: ugly code here!\n cfg_dataset, cfg_pipeline = cfg\n cfg_dataset.pipeline = build_pipeline(cfg_pipeline)\n dataset = build(cfg_dataset, DATASETS, key=\"format\")\n return dataset\ndef build_sampler(cfg):\n \"\"\"Build batch_sampler.\n Args:\n cfg (dict): root config dict.\n Returns:\n batch_sampler: batch_sampler.\n \"\"\"\n sampler = build(cfg, SAMPLERS)\n return sampler\ndef build_batch_pipeline(cfg):\n batch_pipeline = build(cfg, PIPELINES)\n return batch_pipeline\ndef build_custom_dataloader(cfg):\n custom_dataloader = build(cfg, DATALOADERS, key='dataloader')\n return custom_dataloader\ndef build_dataloader(dataset,\n batch_size,\n num_workers,\n places=None,\n shuffle=True,\n drop_last=True,"
+ },
+ {
+ "comment": "The code builds a Paddle Dataloader with optional custom sampler, shuffles data if necessary, and handles distributed batch sampling. It takes dataset, batch size, number of workers, and shuffle settings as input arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/loader/builder.py\":80-105",
+ "content": " multigrid=False,\n collate_fn_cfg=None,\n **kwargs):\n \"\"\"Build Paddle Dataloader.\n XXX explain how the batch_sampler work!\n Args:\n dataset (paddle.dataset): A PaddlePaddle dataset object.\n batch_size (int): batch size on single card.\n num_worker (int): num_worker\n shuffle(bool): whether to shuffle the data at every epoch.\n \"\"\"\n if not kwargs.get('sampler'):\n batch_sampler = DistributedBatchSampler(dataset,\n batch_size=batch_size,\n shuffle=shuffle,\n drop_last=drop_last)\n else:\n sampler = build_sampler(kwargs['sampler'])\n batch_sampler = BatchSampler(dataset,\n sampler=sampler,\n batch_size=batch_size,\n shuffle=shuffle,\n drop_last=drop_last)"
+ },
+ {
+ "comment": "This code defines a mix_collate_fn for handling batches of data in a specific way. It first builds a batch pipeline and applies it to the input batch. Then, it collates the batch so that each item is stacked horizontally (axis=0) into a new batch. This function is used as the collate_fn if the collate_fn_cfg is not None.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/loader/builder.py\":106-133",
+ "content": " kwargs.update({'batch_sampler': batch_sampler})\n # NOTE(shipping): when switch the mix operator on, such as: mixup, cutmix.\n # batch like: [[img, label, attibute, ...], [imgs, label, attribute, ...], ...] will recollate to:\n # [[img, img, ...], [label, label, ...], [attribute, attribute, ...], ...] as using numpy.transpose.\n def mix_collate_fn(batch):\n pipeline = build_batch_pipeline(collate_fn_cfg)\n batch = pipeline(batch)\n slots = []\n for items in batch:\n for i, item in enumerate(items):\n if len(slots) < len(items):\n slots.append([item])\n else:\n slots[i].append(item)\n return [np.stack(slot, axis=0) for slot in slots]\n # if collate_fn_cfg is not None:\n # ugly code here. collate_fn is mix op config\n # collate_fn = mix_collate_fn(collate_fn_cfg)\n data_loader = DataLoader(\n dataset,\n places=places,\n num_workers=num_workers,\n collate_fn=mix_collate_fn if collate_fn_cfg is not None else None,"
+ },
+ {
+ "comment": "This code is setting up signal handlers for SIGINT and SIGTERM signals. It retrieves the process ID (pid) and process group ID (pgid), logs a message, then sends a SIGKILL signal to all processes in the group upon receiving either of those signals.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/loader/builder.py\":134-150",
+ "content": " **kwargs)\n return data_loader\ndef term_mp(sig_num, frame):\n \"\"\" kill all child processes\n \"\"\"\n pid = os.getpid()\n pgid = os.getpgid(os.getpid())\n logger.info(\"main proc {} exit, kill process group \" \"{}\".format(pid, pgid))\n os.killpg(pgid, signal.SIGKILL)\n return\nsignal.signal(signal.SIGINT, term_mp)\nsignal.signal(signal.SIGTERM, term_mp)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/25b47995-09f5-4019-8d32-ecd9a15146e1.json b/docs/doc/25b47995-09f5-4019-8d32-ecd9a15146e1.json
new file mode 100644
index 000000000..8b513e350
--- /dev/null
+++ b/docs/doc/25b47995-09f5-4019-8d32-ecd9a15146e1.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code constructs a learning rate scheduler based on the 'OPTIMIZER' configuration provided, returns it with specified iterations, and handles custom cases such as converting 'learning_rate' to a custom object.",
+ "details": [
+ {
+ "comment": "This code is building a learning rate scheduler according to the \"OPTIMIZER\" configuration provided in the cfg dictionary. The scheduler is based on the 'PiecewiseDecay' name, and has boundaries and values for adjusting the learning rate at specified iterations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/solver/lr.py\":0-27",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom typing import Dict\nfrom paddle.optimizer.lr import LRScheduler\nfrom . import custom_lr\ndef build_lr(cfg: Dict, num_iters: int) -> LRScheduler:\n \"\"\"Build a learning rate scheduler accroding to ```OPTIMIZER``` configuration, and it always pass into the optimizer.\n In configuration:\n learning_rate:\n name: 'PiecewiseDecay'\n boundaries: [20, 60]\n values: [0.00025, 0.000025, 0.0000025]"
+ },
+ {
+ "comment": "This function takes a learning rate configuration and the number of iterations, and returns a learning rate scheduler. If the configuration includes a 'learning_rate' key with a dictionary value, it converts it to a custom learning rate object using the build_lr() function. It also handles cases where 'iter_step' is present in the configuration, replacing it with 'num_iters'. The returned scheduler is obtained from the 'custom_lr' module with the specified 'name'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/solver/lr.py\":29-51",
+ "content": " Args:\n cfg (Dict): learning rate configuration.\n num_iters (int): The number of iterations that may be used when calculating the learning rate\n Returns:\n LRScheduler: learning rate scheduler.\n \"\"\"\n cfg_copy = cfg.copy()\n #when learning_rate is LRScheduler\n if cfg_copy.get('learning_rate') and isinstance(cfg_copy['learning_rate'],\n dict):\n cfg_copy['learning_rate'] = build_lr(\n cfg_copy['learning_rate'],\n num_iters) #not support only inner iter_step\n lr_name = cfg_copy.pop('name')\n if cfg_copy.get('iter_step'):\n cfg_copy['num_iters'] = num_iters\n cfg_copy.pop('iter_step')\n return getattr(custom_lr, lr_name)(**cfg_copy)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2685801b-a741-4770-9e5c-857dc1bf3b84.json b/docs/doc/2685801b-a741-4770-9e5c-857dc1bf3b84.json
new file mode 100644
index 000000000..581ef7854
--- /dev/null
+++ b/docs/doc/2685801b-a741-4770-9e5c-857dc1bf3b84.json
@@ -0,0 +1,30 @@
+{
+ "summary": "This code defines 3D head projection classes for PaddleVideo library, initializes a SlowFast head model with dropout regularization and adaptive average pooling, performs convolutional inference, applies softmax activation, averages when not training, and reshapes before returning.",
+ "details": [
+ {
+ "comment": "This code is from the PaddleVideo library and defines a SlowFastHead class for ResNe(X)t 3D head. It performs a fully-connected projection during training and convolutional projection during testing, with different input sizes handled accordingly. The code includes import statements, registration using HEADS registry, and base class inheritance from BaseHead.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/slowfast_head.py\":0-29",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom ..registry import HEADS\nfrom .base import BaseHead\nimport paddle\nimport paddle.nn.functional as F\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass SlowFastHead(BaseHead):\n \"\"\"\n ResNe(X)t 3D head.\n This layer performs a fully-connected projection during training, when the\n input size is 1x1x1. It performs a convolutional projection during testing\n when the input size is larger than 1x1x1. If the inputs are from multiple"
+ },
+ {
+ "comment": "The code defines a class for SlowFast_Head, which takes different pathways as input and concatenates the inputs after pooling. It has various parameters such as width_per_group, alpha, beta, etc. The ResNetBasicHead takes p pathways as input where p can be in the range of 1 to infinity. It has arguments for dim_in (list), num_classes (int), pool_size (list), and dropout_rate (float).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/slowfast_head.py\":30-55",
+ "content": " different pathways, the inputs will be concatenated after pooling.\n \"\"\"\n def __init__(self,\n width_per_group,\n alpha,\n beta,\n num_classes,\n num_frames,\n crop_size,\n dropout_rate,\n pool_size_ratio=[[1, 1, 1], [1, 1, 1]],\n loss_cfg=dict(name='CrossEntropyLoss'),\n multigrid_short=False,\n **kwargs):\n \"\"\"\n ResNetBasicHead takes p pathways as input where p in [1, infty].\n Args:\n dim_in (list): the list of channel dimensions of the p inputs to the\n ResNetHead.\n num_classes (int): the channel dimensions of the p outputs to the\n ResNetHead.\n pool_size (list): the list of kernel sizes of p spatial temporal\n poolings, temporal pool kernel size, spatial pool kernel size,\n spatial pool kernel size in order.\n dropout_rate (float): dropout rate. If equal to 0.0, perform no"
+ },
+ {
+ "comment": "This code is initializing a SlowFast head model with specified parameters such as multigrid_short, width_per_group, alpha, beta, num_classes, num_frames, crop_size, and dropout_rate. It also sets the dimension input (dim_in) based on these parameters, and determines the pool size accordingly based on whether multigrid_short is True or False.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/slowfast_head.py\":56-82",
+ "content": " dropout.\n \"\"\"\n super().__init__(num_classes, loss_cfg, **kwargs)\n self.multigrid_short = multigrid_short\n self.width_per_group = width_per_group\n self.alpha = alpha\n self.beta = beta\n self.num_classes = num_classes\n self.num_frames = num_frames\n self.crop_size = crop_size\n self.dropout_rate = dropout_rate\n self.pool_size_ratio = pool_size_ratio\n self.dim_in = [\n self.width_per_group * 32,\n self.width_per_group * 32 // self.beta,\n ]\n self.pool_size = [None, None] if self.multigrid_short else [\n [\n self.num_frames // self.alpha // self.pool_size_ratio[0][0],\n self.crop_size // 32 // self.pool_size_ratio[0][1],\n self.crop_size // 32 // self.pool_size_ratio[0][2],\n ],\n [\n self.num_frames // self.pool_size_ratio[1][0],\n self.crop_size // 32 // self.pool_size_ratio[1][1],\n self.crop_size // 32 // self.pool_size_ratio[1][2],"
+ },
+ {
+ "comment": "This code initializes a SlowFast head model. It defines the number of pathways, applies dropout regularization, and initializes weights for linear projection. The forward method expects inputs with the same number of pathways as defined in the model. It then performs adaptive average pooling on each input pathway separately.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/slowfast_head.py\":83-112",
+ "content": " ],\n ]\n assert (len({len(self.pool_size), len(self.dim_in)\n }) == 1), \"pathway dimensions are not consistent.\"\n self.num_pathways = len(self.pool_size)\n self.dropout = paddle.nn.Dropout(p=self.dropout_rate)\n self.projection = paddle.nn.Linear(\n in_features=sum(self.dim_in),\n out_features=self.num_classes,\n )\n def init_weights(self):\n weight_init_(self.projection,\n \"Normal\",\n bias_value=0.0,\n mean=0.0,\n std=0.01)\n def forward(self, inputs):\n assert (len(inputs) == self.num_pathways\n ), \"Input tensor does not contain {} pathway\".format(\n self.num_pathways)\n pool_out = []\n for pathway in range(self.num_pathways):\n if self.pool_size[pathway] is None:\n tmp_out = F.adaptive_avg_pool3d(x=inputs[pathway],\n output_size=(1, 1, 1),"
+ },
+ {
+ "comment": "This code performs pooling and dropout operations on input tensors, followed by projection and fully convolutional inference. It also applies softmax activation and averaging when not in training mode. The resulting tensor is reshaped before returning.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/slowfast_head.py\":113-136",
+ "content": " data_format=\"NCDHW\")\n else:\n tmp_out = F.avg_pool3d(x=inputs[pathway],\n kernel_size=self.pool_size[pathway],\n stride=1,\n data_format=\"NCDHW\")\n pool_out.append(tmp_out)\n x = paddle.concat(x=pool_out, axis=1)\n x = paddle.transpose(x=x, perm=(0, 2, 3, 4, 1))\n # Perform dropout.\n if self.dropout_rate > 0.0:\n x = self.dropout(x)\n x = self.projection(x)\n # Performs fully convlutional inference.\n if not self.training: # attr of base class\n x = F.softmax(x, axis=4)\n x = paddle.mean(x, axis=[1, 2, 3])\n x = paddle.reshape(x, shape=(x.shape[0], -1))\n return x"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2789548c-6052-454a-b1c9-a17b3c27bb6c.json b/docs/doc/2789548c-6052-454a-b1c9-a17b3c27bb6c.json
new file mode 100644
index 000000000..c867abb18
--- /dev/null
+++ b/docs/doc/2789548c-6052-454a-b1c9-a17b3c27bb6c.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The \"SamplerUCF24\" class samples frames from videos using parameters like frame count and interval, utilizes PIL library, initializes pipeline, generates frame indices, returns sampled frames.",
+ "details": [
+ {
+ "comment": "This code defines a class \"SamplerUCF24\" for sampling frames in videos, taking parameters such as num_frames and frame_interval. It uses PIL instead of OpenCV to read images and returns the index of sampled frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ucf24.py\":0-32",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport random\nfrom PIL import Image\nfrom ..registry import PIPELINES\n@PIPELINES.register()\nclass SamplerUCF24(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_frames(int): The amount of frames used in a video\n frame_interval(int): Sampling rate\n valid_mode(bool): True or False.\n Returns:\n frames_idx: the index of sampled #frames."
+ },
+ {
+ "comment": "This code defines a pipeline for loading and creating clips from video files. The `__init__` method initializes the number of frames, frame interval (randomly determined if valid mode is False), and valid mode flag. The `_get` method retrieves images in order, converts them to RGB, and appends them to a list. The `_make_clip` method generates a set of frame indices that create a looped clip. The pipeline is called with the results as input, extracting the image folder and filename for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ucf24.py\":33-64",
+ "content": " \"\"\"\n def __init__(self,\n num_frames=16,\n frame_interval=1,\n valid_mode=False):\n self.num_frames = num_frames\n self.frame_interval = frame_interval if valid_mode else random.randint(1, 2)\n self.valid_mode = valid_mode\n def _get(self, frames_idxs, img_folder, results):\n imgs = []\n for idx in frames_idxs:\n img = Image.open(\n os.path.join(img_folder, '{:05d}.jpg'.format(idx))).convert('RGB')\n imgs.append(img)\n results['imgs'] = imgs\n return results\n def _make_clip(self, im_ind, max_num):\n frame_idxs = []\n for i in reversed(range(self.num_frames)):\n # make it as a loop\n i_temp = im_ind - i * self.frame_interval\n if i_temp < 1:\n i_temp = 1\n elif i_temp > max_num:\n i_temp = max_num\n frame_idxs.append(i_temp)\n return frame_idxs\n def __call__(self, results):\n img_folder, key_frame = os.path.split(results['filename'])"
+ },
+ {
+ "comment": "This code retrieves the number of frames in a folder, assigns a key frame index based on the input, generates frame indices for a video clip, and returns the requested frames from their folder.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ucf24.py\":65-68",
+ "content": " frame_len = len(os.listdir(img_folder))\n key_idx = int(key_frame[0:5])\n frame_idxs = self._make_clip(key_idx, frame_len)\n return self._get(frame_idxs, img_folder, results)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/28804844-7817-4113-9692-e83d3de8eb8e.json b/docs/doc/28804844-7817-4113-9692-e83d3de8eb8e.json
new file mode 100644
index 000000000..748e80d61
--- /dev/null
+++ b/docs/doc/28804844-7817-4113-9692-e83d3de8eb8e.json
@@ -0,0 +1,20 @@
+{
+ "summary": "Summary: A comprehensive list of notable Spatio-Temporal Action Detection papers and authors from 2015-2017, covering major conferences like ICCV, BMVC, ECCV, and arXiv.",
+ "details": [
+ {
+ "comment": "The code provides a list of useful Spatio-Temporal Action Detection papers and their corresponding authors, year of publication, and conference or journal.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/Spatio-Temporal Action Detection Papers\":0-12",
+ "content": "Usefull Spatio-Temporal Action Detection Papers.\n A Better Baseline for AVA - R. Girdhar et al., ActivityNet Workshop, CVPR2018.\n Real-Time End-to-End Action Detection with Two-Stream Networks - A. El-Nouby and G. Taylor, arXiv2018.\n Human Action Localization with Sparse Spatial Supervision - P. Weinzaepfel et al., arXiv2017.\n Unsupervised Action Discovery and Localization in Videos - K. Soomro and M. Shah, ICCV2017.\n Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions - P. Mettes and C. G. M. Snoek, ICCV2017.\n Action Tubelet Detector for Spatio-Temporal Action Localization - V. Kalogeiton et al, ICCV2017. \n Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos - R. Hou et al, ICCV2017. \n Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection - M. Zolfaghari et al, ICCV2017. \n TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal - H. Zhu et al., ICCV2017."
+ },
+ {
+ "comment": "This code provides a list of notable research papers related to action detection and localization in videos, published between 2015 and 2017. The papers are from various conferences such as ICCV, BMVC, ECCV, and arXiv, demonstrating the advancements made in this field during that period.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/Spatio-Temporal Action Detection Papers\":13-23",
+ "content": " Online Real time Multiple Spatiotemporal Action Localisation and Prediction - G. Singh et al, ICCV2017. \n AMTnet: Action-Micro-Tube regression by end-to-end trainable deep architecture - S. Saha et al, ICCV2017.\n Am I Done? Predicting Action Progress in Videos - F. Becattini et al, BMVC2017.\n Generic Tubelet Proposals for Action Localization - J. He et al, arXiv2017.\n Incremental Tube Construction for Human Action Detection - H. S. Behl et al, arXiv2017.\n Multi-region two-stream R-CNN for action detection - X. Peng and C. Schmid. ECCV2016. \n Spot On: Action Localization from Pointly-Supervised Proposals - P. Mettes et al, ECCV2016.\n Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos - S. Saha et al, BMVC2016. \n Learning to track for spatio-temporal action localization - P. Weinzaepfel et al. ICCV2015.\n Action detection by implicit intentional motion clustering - W. Chen and J. Corso, ICCV2015.\n Finding Action Tubes - G. Gkioxari and J. Malik CVPR2015. "
+ },
+ {
+ "comment": "List of papers on action localization and detection in videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/Spatio-Temporal Action Detection Papers\":24-29",
+ "content": " APT: Action localization proposals from dense trajectories - J. Gemert et al, BMVC2015. \n Spatio-Temporal Object Detection Proposals - D. Oneata et al, ECCV2014.\n Action localization with tubelets from motion - M. Jain et al, CVPR2014.\n Spatiotemporal deformable part models for action detection - Y. Tian et al, CVPR2013. \n Action localization in videos through context walk - K. Soomro et al, ICCV2015.\n Fast Action Proposals for Human Action Detection and Search - G. Yu and J. Yuan, CVPR2015. "
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/288152c2-ba5b-46b3-96a6-dc101461e129.json b/docs/doc/288152c2-ba5b-46b3-96a6-dc101461e129.json
new file mode 100644
index 000000000..9315c8727
--- /dev/null
+++ b/docs/doc/288152c2-ba5b-46b3-96a6-dc101461e129.json
@@ -0,0 +1,40 @@
+{
+ "summary": "This code records metrics, calculates means, logs batch info and epoch progress in training processes with colored formatting for visibility. It uses PaddleVideo framework, AverageMeter and OrderedDict for efficient logging.",
+ "details": [
+ {
+ "comment": "Code snippet imports necessary libraries and defines functions for building a record, logging batches and epochs. It also sets up logger for the PaddleVideo framework.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":0-31",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport datetime\nfrom collections import OrderedDict\nimport paddle\nfrom .logger import coloring, get_logger\nlogger = get_logger(\"paddlevideo\")\n__all__ = ['AverageMeter', 'build_record', 'log_batch', 'log_epoch']\ndef build_record(cfg):\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"lr\", AverageMeter('lr', 'f', need_avg=False)),\n ]\n if 'Recognizer1D' in cfg.framework: #TODO: required specify str in framework"
+ },
+ {
+ "comment": "Code appends specific metrics to the record list based on the framework specified in cfg. Frameworks include 'PaddleVideo', 'Recognizer', and 'FastRCNN'. Metrics are averaged using AverageMeter and include 'hit_at_one', 'perr', 'gap', 'top1', 'top5', recall@thr=0.5, prec@thr=0.5, recall@top3, prec@top3, recall@top5, prec@top5, and mAP@0.5IOU.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":32-47",
+ "content": " record_list.append((\"hit_at_one\", AverageMeter(\"hit_at_one\", '.5f')))\n record_list.append((\"perr\", AverageMeter(\"perr\", '.5f')))\n record_list.append((\"gap\", AverageMeter(\"gap\", '.5f')))\n elif 'Recognizer' in cfg.framework:\n record_list.append((\"top1\", AverageMeter(\"top1\", '.5f')))\n record_list.append((\"top5\", AverageMeter(\"top5\", '.5f')))\n elif 'FastRCNN' in cfg.framework:\n record_list.append(\n (\"recall@thr=0.5\", AverageMeter(\"recall@thr=0.5\", '.5f')))\n record_list.append((\"prec@thr=0.5\", AverageMeter(\"prec@thr=0.5\",\n '.5f')))\n record_list.append((\"recall@top3\", AverageMeter(\"recall@top3\", '.5f')))\n record_list.append((\"prec@top3\", AverageMeter(\"prec@top3\", '.5f')))\n record_list.append((\"recall@top5\", AverageMeter(\"recall@top5\", '.5f')))\n record_list.append((\"prec@top5\", AverageMeter(\"prec@top5\", '.5f')))\n record_list.append((\"mAP@0.5IOU\", AverageMeter(\"mAP@0.5IOU\", '.5f')))"
+ },
+ {
+ "comment": "The code is conditionally adding metrics to the record list based on the value of 'cfg.framework'. It handles three different cases: 'DepthEstimator', 'MSTCN' or 'ASRF', and 'YOWOLocalizer'. For 'DepthEstimator', it adds 9 metrics, for 'MSTCN' or 'ASRF', it adds one metric, and for 'YOWOLocalizer', it adds two metrics. Each metric is associated with an AverageMeter object that keeps track of its mean value over time.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":48-64",
+ "content": " elif 'DepthEstimator' in cfg.framework:\n record_list.append((\"abs_rel\", AverageMeter(\"abs_rel\", '.5f')))\n record_list.append((\"sq_rel\", AverageMeter(\"sq_rel\", '.5f')))\n record_list.append((\"rmse\", AverageMeter(\"rmse\", '.5f')))\n record_list.append((\"rmse_log\", AverageMeter(\"rmse_log\", '.5f')))\n record_list.append((\"a1\", AverageMeter(\"a1\", '.5f')))\n record_list.append((\"a2\", AverageMeter(\"a2\", '.5f')))\n record_list.append((\"a3\", AverageMeter(\"a3\", '.5f')))\n record_list.append((\"losses_day\", AverageMeter(\"losses_day\", '.5f')))\n record_list.append((\"losses_night\", AverageMeter(\"losses_night\",\n '.5f')))\n elif 'MSTCN' in cfg.framework or 'ASRF' in cfg.framework:\n record_list.append((\"F1@0.50\", AverageMeter(\"F1@0.50\", '.5f')))\n elif 'YOWOLocalizer' in cfg.framework:\n record_list.append((\"nCorrect\", AverageMeter('nCorrect', '.1f')))\n record_list.append((\"fscore\", AverageMeter(\"fscore\", '.5f')))"
+ },
+ {
+ "comment": "This function creates a record dictionary containing two AverageMeter objects, one for batch time and another for reader time. It then converts the list to an OrderedDict and returns it. The AverageMeter class calculates and stores the average and current values of a given metric, allowing easy tracking of performance metrics during program execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":66-104",
+ "content": " record_list.append((\"batch_time\", AverageMeter('batch_cost', '.5f')))\n record_list.append((\"reader_time\", AverageMeter('reader_cost', '.5f')))\n record_list = OrderedDict(record_list)\n return record_list\nclass AverageMeter(object):\n \"\"\"\n Computes and stores the average and current value\n \"\"\"\n def __init__(self, name='', fmt='f', need_avg=True):\n self.name = name\n self.fmt = fmt\n self.need_avg = need_avg\n self.reset()\n def reset(self):\n \"\"\" reset \"\"\"\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def update(self, val, n=1):\n \"\"\" update \"\"\"\n if isinstance(val, paddle.Tensor):\n val = float(val)\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n @property\n def total(self):\n return '{self.name}_sum: {self.sum:{self.fmt}}'.format(self=self)\n @property\n def total_minute(self):\n return '{self.name}_sum: {s:{self.fmt}} min'.format(s=self.sum / 60,"
+ },
+ {
+ "comment": "This code defines a class and functions for recording metrics, calculating means, and logging batch information. The `log_batch` function records the time taken for each batch, adds other metric values, and logs the total epoch, current epoch, mode, and step. It also calculates the remaining time for the current operation if provided.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":105-135",
+ "content": " self=self)\n @property\n def mean(self):\n return '{self.name}_avg: {self.avg:{self.fmt}}'.format(\n self=self) if self.need_avg else ''\n @property\n def value(self):\n return '{self.name}: {self.val:{self.fmt}}'.format(self=self)\ndef log_batch(metric_list,\n batch_id,\n epoch_id,\n total_epoch,\n mode,\n ips,\n eta_sec: int = None):\n batch_cost = str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = str(metric_list['reader_time'].value) + ' sec,'\n metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].value)\n metric_str = ' '.join([str(v) for v in metric_values])\n epoch_str = \"epoch:[{:>3d}/{:<3d}]\".format(epoch_id, total_epoch)\n step_str = \"{:s} step:{:<4d}\".format(mode, batch_id)\n if eta_sec is not None:\n eta_str = \"eta: {:s}\".format("
+ },
+ {
+ "comment": "This code logs the progress of an epoch in a training process. It formats the logged information with colors for better visibility. The logger displays the current step, metrics, time taken, batch cost and reader cost, estimated time remaining (ETA), and maximum memory reserved and allocated on CUDA devices if available.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":136-154",
+ "content": " str(datetime.timedelta(seconds=int(eta_sec))))\n else:\n eta_str = ''\n max_mem_reserved_str = \"\"\n max_mem_allocated_str = \"\"\n if paddle.device.is_compiled_with_cuda():\n max_mem_reserved_str = f\"max_mem_reserved: {format(paddle.device.cuda.max_memory_reserved() / (1024 ** 2), '.2f')} MB\"\n max_mem_allocated_str = f\"max_mem_allocated: {format(paddle.device.cuda.max_memory_allocated() / (1024 ** 2), '.2f')} MB\"\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {} {:s}, {} {}\".format(\n coloring(epoch_str, \"HEADER\") if batch_id == 0 else epoch_str,\n coloring(step_str, \"PURPLE\"), coloring(metric_str, 'OKGREEN'),\n coloring(batch_cost, \"OKGREEN\"), coloring(reader_cost, 'OKGREEN'), ips,\n eta_str, max_mem_reserved_str, max_mem_allocated_str))\ndef log_epoch(metric_list, epoch, mode, ips):\n batch_cost = 'avg_' + str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = 'avg_' + str(metric_list['reader_time'].value) + ' sec,'\n batch_sum = str(metric_list['batch_time'].total) + ' sec,'"
+ },
+ {
+ "comment": "This code calculates the mean of metrics except 'batch_time' and 'reader_time', then joins them into a string. It formats an info message with RED for \"END epoch\", PURPLE for mode, GREEN for metric values, batch cost, reader cost, and batch sum, as well as ips (inferences per second). The logger outputs this formatted message.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/record.py\":156-167",
+ "content": " metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].mean)\n metric_str = ' '.join([str(v) for v in metric_values])\n end_epoch_str = \"END epoch:{:<3d}\".format(epoch)\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {:s} {}\".format(\n coloring(end_epoch_str, \"RED\"), coloring(mode, \"PURPLE\"),\n coloring(metric_str, \"OKGREEN\"), coloring(batch_cost, \"OKGREEN\"),\n coloring(reader_cost, \"OKGREEN\"), coloring(batch_sum, \"OKGREEN\"), ips))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/28eff272-aa18-4b97-8e97-b84c16992ace.json b/docs/doc/28eff272-aa18-4b97-8e97-b84c16992ace.json
new file mode 100644
index 000000000..2586a0279
--- /dev/null
+++ b/docs/doc/28eff272-aa18-4b97-8e97-b84c16992ace.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code defines a Compose class for image processing steps and functions to preprocess video frames, returning input/output variables. The get_preprocess_func function selects the correct preprocessing function based on the model name. Invalid names raise ValueError.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and defines the VALID_MODELS variable. It then creates a Compose class that takes in a list of transforms, allowing for composition of multiple image processing steps to be applied sequentially.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/preprocess_ops.py\":0-33",
+ "content": "import os\nimport sys\n__dir__ = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(os.path.abspath(os.path.join(__dir__, \"../../\")))\nfrom paddlevideo.loader.pipelines import (CenterCrop, Image2Array,\n Normalization, Sampler, Scale,\n VideoDecoder, TenCrop)\nimport numpy as np\nfrom typing import Dict, Tuple, List, Callable\nVALID_MODELS = [\"PPTSM\", \"PPTSN\"]\nimport os\nimport sys\n__dir__ = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(os.path.abspath(os.path.join(__dir__, \"../../\")))\nfrom paddlevideo.loader.pipelines import (CenterCrop, Image2Array,\n Normalization, Sampler, Scale,\n VideoDecoder, TenCrop)\nimport numpy as np\nfrom typing import Dict, Tuple, List, Callable\nVALID_MODELS = [\"PPTSM\", \"PPTSN\"]\nclass Compose:\n def __init__(self, transforms):\n self.transforms = transforms\n def __call__(self, img):\n for t in self.transforms:"
+ },
+ {
+ "comment": "The code is defining a function `preprocess_PPTSM` that takes a video path as input, and applies several image preprocessing steps before returning the feed and fetch data. These steps include decoding the video frames, sampling, scaling, cropping, converting to array format, and normalization using specific mean and standard deviation values. The resulting processed data is stored in the `results` dictionary, which contains the images and metadata.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/preprocess_ops.py\":34-75",
+ "content": " img = t(img)\n return img\ndef np_softmax(x: np.ndarray, axis: int = 0) -> np.ndarray:\n \"\"\"softmax function\n Args:\n x (np.ndarray): logits\n axis (int): axis\n Returns:\n np.ndarray: probs\n \"\"\"\n x -= np.max(x, axis=axis, keepdims=True)\n x = np.exp(x) / np.sum(np.exp(x), axis=axis, keepdims=True)\n return x\ndef preprocess_PPTSM(video_path: str) -> Tuple[Dict[str, np.ndarray], List]:\n \"\"\"preprocess\n Args:\n video_path (str): input video path\n Returns:\n Tuple[Dict[str, np.ndarray], List]: feed and fetch\n \"\"\"\n img_mean = [0.485, 0.456, 0.406]\n img_std = [0.229, 0.224, 0.225]\n seq = Compose([\n VideoDecoder(),\n Sampler(8, 1, valid_mode=True),\n Scale(256),\n CenterCrop(224),\n Image2Array(),\n Normalization(img_mean, img_std)\n ])\n results = {\"filename\": video_path}\n results = seq(results)\n tmp_inp = np.expand_dims(results[\"imgs\"], axis=0) # [b,t,c,h,w]\n tmp_inp = np.expand_dims(tmp_inp, axis=0) # [1,b,t,c,h,w]"
+ },
+ {
+ "comment": "The function preprocess_PPTSN takes in a video path, applies a series of image processing steps to the video frames, and returns feed and fetch variables for input and output respectively. The get_preprocess_func function returns a preprocessing function based on the given model name.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/preprocess_ops.py\":76-110",
+ "content": " feed = {\"data_batch_0\": tmp_inp}\n fetch = [\"outputs\"]\n return feed, fetch\ndef preprocess_PPTSN(video_path: str) -> Tuple[Dict[str, np.ndarray], List]:\n \"\"\"preprocess\n Args:\n video_path (str): input video path\n Returns:\n Tuple[Dict[str, np.ndarray], List]: feed and fetch\n \"\"\"\n img_mean = [0.485, 0.456, 0.406]\n img_std = [0.229, 0.224, 0.225]\n seq = Compose([\n VideoDecoder(),\n Sampler(25, 1, valid_mode=True, select_left=True),\n Scale(256, fixed_ratio=True, do_round=True, backend='cv2'),\n TenCrop(224),\n Image2Array(),\n Normalization(img_mean, img_std)\n ])\n results = {\"filename\": video_path}\n results = seq(results)\n tmp_inp = np.expand_dims(results[\"imgs\"], axis=0) # [b,t,c,h,w]\n tmp_inp = np.expand_dims(tmp_inp, axis=0) # [1,b,t,c,h,w]\n feed = {\"data_batch_0\": tmp_inp}\n fetch = [\"outputs\"]\n return feed, fetch\ndef get_preprocess_func(model_name: str) -> Callable:\n \"\"\"get preprocess function by model_name"
+ },
+ {
+ "comment": "This function takes a model name as input and returns the corresponding preprocess function based on the conditionals provided. If the model name is \"PPTSM\", it will return the preprocess_PPTSM function, if the model name is \"PPTSN\" it will return preprocess_PPTSN, otherwise it raises a ValueError with an error message stating that the model name must be in VALID_MODELS.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/preprocess_ops.py\":112-125",
+ "content": " Args:\n model_name (str): model's name, must in `VALID_MODELS`\n Returns:\n Callable: preprocess function corresponding to model name\n \"\"\"\n if model_name == \"PPTSM\":\n return preprocess_PPTSM\n elif model_name == \"PPTSN\":\n return preprocess_PPTSN\n else:\n raise ValueError(\n f\"model_name must in {VALID_MODELS}, but got model_name\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/294e5c0a-0354-42fc-a66a-9bd9614a892d.json b/docs/doc/294e5c0a-0354-42fc-a66a-9bd9614a892d.json
new file mode 100644
index 000000000..6fbd4df64
--- /dev/null
+++ b/docs/doc/294e5c0a-0354-42fc-a66a-9bd9614a892d.json
@@ -0,0 +1,130 @@
+{
+ "summary": "This code initializes a DAVIS 2017 dataset class for loading and preprocessing, creates a custom dataloader, prepares input for Ma-Net model, and outputs JSON files with sequence data.",
+ "details": [
+ {
+ "comment": "This code snippet defines a DAVIS2017_Test_Manager class for loading and managing test data from the DAVIS 2017 dataset. It accepts parameters such as split, root directory, transformations to apply, and sequence name. The get_image() method retrieves an image from the specified directory based on the index, reads it into a numpy array, and converts it into float32 data type. This class can be used for loading test images in the DAVIS 2017 dataset for further processing or analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":0-39",
+ "content": "from __future__ import division\nimport json\nimport os\nimport shutil\nimport numpy as np\nimport paddle, cv2\nfrom random import choice\nfrom paddle.io import Dataset\nimport json\nfrom PIL import Image\nfrom davisinteractive.utils.scribbles import scribbles2mask, annotated_frames\nimport sys\nsys.path.append(\"..\")\nfrom config import cfg\nimport time\nclass DAVIS2017_Test_Manager():\n def __init__(self,\n split='val',\n root=cfg.DATA_ROOT,\n transform=None,\n rgb=False,\n seq_name=None):\n self.split = split\n self.db_root_dir = root\n self.rgb = rgb\n self.transform = transform\n self.seq_name = seq_name\n def get_image(self, idx):\n frame_name = str(idx)\n while len(frame_name) != 5:\n frame_name = '0' + frame_name\n imgpath = os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n str(self.seq_name), frame_name + '.jpg')\n img = cv2.imread(imgpath)\n img = np.array(img, dtype=np.float32)"
+ },
+ {
+ "comment": "This code is initializing a dataset for DAVIS2017, which contains images and their features. It loads the image list from a specified directory, and applies optional transformations to the samples before returning them. The dataset supports different splits (e.g., training or validation) and allows for specifying an optional sequence name.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":40-72",
+ "content": " sample = {'img': img}\n if self.transform is not None:\n sample = self.transform(sample)\n return sample\nclass DAVIS2017_Feature_Extract(Dataset):\n def __init__(self,\n split='val',\n root=cfg.DATA_ROOT,\n transform=None,\n rgb=False,\n seq_name=None):\n self.split = split\n self.db_root_dir = root\n self.rgb = rgb\n self.transform = transform\n self.seq_name = seq_name\n self.img_list = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n str(seq_name))))\n def __len__(self):\n return len(self.img_list)\n def __getitem__(self, idx):\n img = self.img_list[idx]\n imgpath = os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n str(self.seq_name), img)\n current_img = cv2.imread(imgpath)\n current_img = np.array(current_img, dtype=np.float32)"
+ },
+ {
+ "comment": "The code defines a DAVIS2017_VOS_Test dataset class which loads data from the DAVIS 2017 dataset for semantic segmentation tasks. It takes various parameters such as split, root directory, transformation function, if RGB images are required, result root directory, and sequence name. It reads a list of sequences from a file and returns an image sample along with its metadata (sequence name, height, width, and image path).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":73-108",
+ "content": " h, w, _ = current_img.shape\n sample = {'img1': current_img}\n sample['meta'] = {\n 'seq_name': self.seq_name,\n 'h_w': (h, w),\n 'img_path': imgpath\n }\n if self.transform is not None:\n sample = self.transform(sample)\n return sample\nclass DAVIS2017_VOS_Test(Dataset):\n \"\"\"\n \"\"\"\n def __init__(self,\n split='val',\n root=cfg.DATA_ROOT,\n transform=None,\n rgb=False,\n result_root=None,\n seq_name=None):\n self.split = split\n self.db_root_dir = root\n self.result_root = result_root\n self.rgb = rgb\n self.transform = transform\n self.seq_name = seq_name\n self.seq_list_file = os.path.join(\n self.db_root_dir, 'ImageSets', '2017',\n '_'.join(self.split) + '_instances.txt')\n self.seqs = []\n for splt in self.split:\n with open(\n os.path.join(self.db_root_dir, 'ImageSets', '2017',"
+ },
+ {
+ "comment": "The code reads sequences from a file and extends the existing sequence list. It then checks if preprocessing is required and performs it if necessary. The code asserts that the sequence name exists in the dictionary of sequences. Next, it retrieves image names and label names, creating lists of image paths and label paths respectively. Finally, it ensures that a specific file exists.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":109-134",
+ "content": " self.split + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))\n self.seqs.extend(seqs_tmp)\n if not self._check_preprocess():\n self._preprocess()\n assert self.seq_name in self.seq_dict.keys(\n ), '{} not in {} set.'.format(self.seq_name, '_'.join(self.split))\n names_img = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n str(seq_name))))\n img_list = list(\n map(lambda x: os.path.join('JPEGImages/480p/', str(seq_name), x),\n names_img))\n name_label = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'Annotations/480p/',\n str(seq_name))))\n labels = list(\n map(lambda x: os.path.join('Annotations/480p/', str(seq_name), x),\n name_label))\n if not os.path.isfile("
+ },
+ {
+ "comment": "This code creates a data loader for the DAVIS 2017 dataset. It checks if the result directory exists, and if not, it creates it and copies the label file to the new directory. If the directory already exists, it simply copies the label file. The function then sets the first image and its label, as well as the remaining images in the list. Lastly, it defines the length of the dataset and a method for getting items from the dataset at specific indices.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":135-160",
+ "content": " os.path.join(self.result_root, seq_name, name_label[0])):\n if not os.path.exists(os.path.join(self.result_root, seq_name)):\n os.makedirs(os.path.join(self.result_root, seq_name))\n shutil.copy(\n os.path.join(self.db_root_dir, labels[0]),\n os.path.join(self.result_root, seq_name, name_label[0]))\n else:\n shutil.copy(\n os.path.join(self.db_root_dir, labels[0]),\n os.path.join(self.result_root, seq_name, name_label[0]))\n self.first_img = names_img[0]\n self.first_label = name_label[0]\n self.img_list = names_img[1:]\n def __len__(self):\n return len(self.img_list)\n def __getitem__(self, idx):\n img = self.img_list[idx]\n imgpath = os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n str(self.seq_name), img)\n num_frame = int(img.split('.')[0])\n ref_img = os.path.join(self.db_root_dir, 'JPEGImages/480p/',"
+ },
+ {
+ "comment": "The code snippet is responsible for loading images and labels from a specific path. It handles image path formatting, ensures all frames have 5 digits, reads images using cv2, converts them to numpy arrays with float32 dtype, and retrieves reference labels by opening and converting the label file to uint8 dtype.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":161-185",
+ "content": " str(self.seq_name), self.first_img)\n prev_frame = num_frame - 1\n prev_frame = str(prev_frame)\n while len(prev_frame) != 5:\n prev_frame = '0' + prev_frame\n prev_img = os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n str(self.seq_name),\n prev_frame + '.' + img.split('.')[-1])\n current_img = cv2.imread(imgpath)\n current_img = np.array(current_img, dtype=np.float32)\n ref_img = cv2.imread(ref_img)\n ref_img = np.array(ref_img, dtype=np.float32)\n prev_img = cv2.imread(prev_img)\n prev_img = np.array(prev_img, dtype=np.float32)\n ref_label = os.path.join(self.db_root_dir, 'Annotations/480p/',\n str(self.seq_name), self.first_label)\n ref_label = Image.open(ref_label)\n ref_label = np.array(ref_label, dtype=np.uint8)\n prev_label = os.path.join(\n self.result_root, str(self.seq_name),"
+ },
+ {
+ "comment": "This code appears to be part of a data loader for a video object detection task. It loads frames and labels from a specific dataset (DAVIS 2017) and creates samples for each frame. The _check_preprocess function checks if the sequence list file exists, and if so, it loads the dictionary of sequences. The _preprocess function initializes an empty dictionary for the sequence dictionary and iterates over the specified sequences to process them.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":186-218",
+ "content": " prev_frame + '.' + self.first_label.split('.')[-1])\n prev_label = Image.open(prev_label)\n prev_label = np.array(prev_label, dtype=np.uint8)\n obj_num = self.seq_dict[self.seq_name][-1]\n sample = {\n 'ref_img': ref_img,\n 'prev_img': prev_img,\n 'current_img': current_img,\n 'ref_label': ref_label,\n 'prev_label': prev_label\n }\n sample['meta'] = {\n 'seq_name': self.seq_name,\n 'frame_num': num_frame,\n 'obj_num': obj_num,\n 'current_name': img\n }\n if self.transform is not None:\n sample = self.transform(sample)\n return sample\n def _check_preprocess(self):\n _seq_list_file = self.seq_list_file\n if not os.path.isfile(_seq_list_file):\n return False\n else:\n self.seq_dict = json.load(open(self.seq_list_file, 'r'))\n return True\n def _preprocess(self):\n self.seq_dict = {}\n for seq in self.seqs:"
+ },
+ {
+ "comment": "This code reads object masks from DAVIS 2017 dataset, obtains the number of objects, and creates a dictionary containing sequence names as keys and their corresponding unique object IDs as values. The dictionary is then saved to a file in JSON format for further use in the DAVIS2017_VOS_Train class, which serves as the training dataset for the DAVIS 2017 dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":219-243",
+ "content": " # Read object masks and get number of objects\n name_label = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'Annotations/480p/', seq)))\n label_path = os.path.join(self.db_root_dir, 'Annotations/480p/',\n seq, name_label[0])\n _mask = np.array(Image.open(label_path))\n _mask_ids = np.unique(_mask)\n n_obj = _mask_ids[-1]\n self.seq_dict[seq] = list(range(1, n_obj + 1))\n with open(self.seq_list_file, 'w') as outfile:\n outfile.write('{{\\n\\t\"{:s}\": {:s}'.format(\n self.seqs[0], json.dumps(self.seq_dict[self.seqs[0]])))\n for ii in range(1, len(self.seqs)):\n outfile.write(',\\n\\t\"{:s}\": {:s}'.format(\n self.seqs[ii], json.dumps(self.seq_dict[self.seqs[ii]])))\n outfile.write('\\n}\\n')\n print('Preprocessing finished')\nclass DAVIS2017_VOS_Train(Dataset):\n \"\"\"DAVIS2017 dataset for training"
+ },
+ {
+ "comment": "This code initializes a class for loading and preprocessing data from the DAVIS 2017 dataset. It takes parameters such as split, root directory, transformation functions, and RGB mode. The code reads sequence lists and checks if pre-processing is necessary before creating a list of samples to be loaded.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":245-272",
+ "content": " Return: imgs: N*2*3*H*W,label: N*2*1*H*W, seq-name: N, frame_num:N\n \"\"\"\n def __init__(self,\n split='train',\n root=cfg.DATA_ROOT,\n transform=None,\n rgb=False):\n self.split = split\n self.db_root_dir = root\n self.rgb = rgb\n self.transform = transform\n self.seq_list_file = os.path.join(\n self.db_root_dir, 'ImageSets', '2017',\n '_'.join(self.split) + '_instances.txt')\n self.seqs = []\n for splt in self.split:\n with open(\n os.path.join(self.db_root_dir, 'ImageSets', '2017',\n self.split + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))\n self.seqs.extend(seqs_tmp)\n self.imglistdic = {}\n if not self._check_preprocess():\n self._preprocess()\n self.sample_list = []\n for seq_name in self.seqs:\n images = np.sort("
+ },
+ {
+ "comment": "The code defines a class for loading data from the DAVIS 2017 dataset, extracting image and annotation files based on the given sequence name. It also provides methods to get the length of the dataset and retrieve specific items by index. The dataset is organized into 'JPEGImages/480p' and 'Annotations/480p' directories with corresponding sequence names. It selects a random previous image from the list, increments its frame number by 1 to get the next image, and returns both the image and annotation files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":273-298",
+ "content": " os.listdir(\n os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n seq_name.strip())))\n images_path = list(\n map(\n lambda x: os.path.join('JPEGImages/480p/', seq_name.strip(),\n x), images))\n lab = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'Annotations/480p/',\n seq_name.strip())))\n lab_path = list(\n map(\n lambda x: os.path.join('Annotations/480p/', seq_name.strip(\n ), x), lab))\n self.imglistdic[seq_name] = (images, lab)\n def __len__(self):\n return len(self.seqs)\n def __getitem__(self, idx):\n seqname = self.seqs[idx]\n imagelist, lablist = self.imglistdic[seqname]\n prev_img = np.random.choice(imagelist[:-1], 1)\n prev_img = prev_img[0]\n frame_num = int(prev_img.split('.')[0]) + 1"
+ },
+ {
+ "comment": "Processing two adjacent frames and labels: Reads next image, prepares previous image and their corresponding labels from file paths.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":299-319",
+ "content": " next_frame = str(frame_num)\n while len(next_frame) != 5:\n next_frame = '0' + next_frame\n ###############################Processing two adjacent frames and labels\n img2path = os.path.join('JPEGImages/480p/', seqname,\n next_frame + '.' + prev_img.split('.')[-1])\n img2 = cv2.imread(os.path.join(self.db_root_dir, img2path))\n img2 = np.array(img2, dtype=np.float32)\n imgpath = os.path.join('JPEGImages/480p/', seqname, prev_img)\n img1 = cv2.imread(os.path.join(self.db_root_dir, imgpath))\n img1 = np.array(img1, dtype=np.float32)\n ###############\n labelpath = os.path.join(\n 'Annotations/480p/', seqname,\n prev_img.split('.')[0] + '.' + lablist[0].split('.')[-1])\n label1 = Image.open(os.path.join(self.db_root_dir, labelpath))\n label2path = os.path.join('Annotations/480p/', seqname,\n next_frame + '.' + lablist[0].split('.')[-1])\n label2 = Image.open(os.path.join(self.db_root_dir, label2path))"
+ },
+ {
+ "comment": "This code randomly selects a reference image and associated scribble label for each video frame, ensuring the labels are unique and not from the same or consecutive frames. It also ensures that the selected images have corresponding annotations in the 480p folder.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":321-343",
+ "content": " label1 = np.array(label1, dtype=np.uint8)\n label2 = np.array(label2, dtype=np.uint8)\n ###################\n ref_img = np.random.choice(imagelist, 1)\n ref_img = ref_img[0]\n ref_img_name = ref_img\n ref_scribble_label = Image.open(\n os.path.join(\n self.db_root_dir, 'Annotations/480p/', seqname,\n ref_img_name.split('.')[0] + '.' + lablist[0].split('.')[-1]))\n ref_scribble_label = np.array(ref_scribble_label, dtype=np.uint8)\n while len(np.unique(ref_scribble_label)) < self.seq_dict[seqname][\n -1] + 1 or ref_img == prev_img or ref_img == (\n next_frame + '.' + prev_img.split('.')[-1]):\n ref_img = np.random.choice(imagelist, 1)\n ref_img = ref_img[0]\n ref_img_name = ref_img\n ref_scribble_label = Image.open(\n os.path.join(\n self.db_root_dir, 'Annotations/480p/', seqname,\n ref_img_name.split('.')[0] + '.' +"
+ },
+ {
+ "comment": "This code reads an image, splits it into RGB channels if required, and stores it in a dictionary along with other images and labels. It also assigns metadata to the sample. The transform is applied if not None.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":344-373",
+ "content": " lablist[0].split('.')[-1]))\n ref_scribble_label = np.array(ref_scribble_label, dtype=np.int64)\n ref_img = os.path.join('JPEGImages/480p/', seqname, ref_img)\n ref_img = cv2.imread(os.path.join(self.db_root_dir, ref_img))\n ref_img = np.array(ref_img, dtype=np.float32)\n ####\n ###################\n if self.rgb:\n img1 = img1[:, :, [2, 1, 0]]\n img2 = img2[:, :, [2, 1, 0]]\n ref_img = ref_img[:, :, [2, 1, 0]]\n obj_num = self.seq_dict[seqname][-1]\n sample = {\n 'ref_img': ref_img,\n 'img1': img1,\n 'img2': img2,\n 'ref_scribble_label': ref_scribble_label,\n 'label1': label1,\n 'label2': label2\n }\n sample['meta'] = {\n 'seq_name': seqname,\n 'frame_num': frame_num,\n 'obj_num': obj_num\n }\n if self.transform is not None:\n sample = self.transform(sample)\n sample['ref_scribble_label'] = paddle.to_tensor("
+ },
+ {
+ "comment": "The code defines a function that loads and preprocesses data from a specific source. It checks if the sequence list file exists and then proceeds to read object masks, getting the number of objects in each sequence.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":374-399",
+ "content": " sample['ref_scribble_label'], dtype='int64')\n sample['label1'] = paddle.to_tensor(sample['label1'], dtype='int64')\n sample['label2'] = paddle.to_tensor(sample['label2'], dtype='int64')\n return sample\n ########################\n def _check_preprocess(self):\n _seq_list_file = self.seq_list_file\n if not os.path.isfile(_seq_list_file):\n return False\n else:\n self.seq_dict = json.load(open(self.seq_list_file, 'r'))\n return True\n def _preprocess(self):\n self.seq_dict = {}\n for seq in self.seqs:\n # Read object masks and get number of objects\n name_label = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'Annotations/480p/', seq)))\n label_path = os.path.join(self.db_root_dir, 'Annotations/480p/',\n seq, name_label[0])\n _mask = np.array(Image.open(label_path))\n _mask_ids = np.unique(_mask)"
+ },
+ {
+ "comment": "This code defines a class for the DAVIS2017 dataset used in training. It initializes the dataset object based on specified parameters, writes a sequence list file containing frame numbers for each sequence, and provides the functionality to load images, masks, and other data required for training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":400-430",
+ "content": " n_obj = _mask_ids[-1]\n self.seq_dict[seq] = list(range(1, n_obj + 1))\n with open(self.seq_list_file, 'w') as outfile:\n outfile.write('{{\\n\\t\"{:s}\": {:s}'.format(\n self.seqs[0], json.dumps(self.seq_dict[self.seqs[0]])))\n for ii in range(1, len(self.seqs)):\n outfile.write(',\\n\\t\"{:s}\": {:s}'.format(\n self.seqs[ii], json.dumps(self.seq_dict[self.seqs[ii]])))\n outfile.write('\\n}\\n')\n print('Preprocessing finished')\nclass DAVIS2017_Train(Dataset):\n \"\"\"DAVIS2017 dataset for training\n Return: imgs: N*2*3*H*W,label: N*2*1*H*W, seq-name: N, frame_num:N\n \"\"\"\n def __init__(self,\n split='train',\n root=cfg.DATA_ROOT,\n transform=None,\n rgb=False):\n self.split = split\n self.db_root_dir = root\n self.rgb = rgb\n self.transform = transform\n self.seq_list_file = os.path.join(\n self.db_root_dir, 'ImageSets', '2017',"
+ },
+ {
+ "comment": "This code is creating a custom dataloader for the DAVIS dataset. It loads the image and annotation files, sorts them by name, checks if preprocessing needs to be done, and then forms a sample list containing the image paths and labels. The result will be used for training or testing purposes in the Ma-Net application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":431-455",
+ "content": " '_'.join(self.split) + '_instances.txt')\n self.seqs = []\n for splt in self.split:\n with open(\n os.path.join(self.db_root_dir, 'ImageSets', '2017',\n self.split + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))\n self.seqs.extend(seqs_tmp)\n if not self._check_preprocess():\n self._preprocess()\n self.sample_list = []\n for seq_name in self.seqs:\n images = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'JPEGImages/480p/',\n seq_name.strip())))\n images_path = list(\n map(\n lambda x: os.path.join('JPEGImages/480p/', seq_name.strip(),\n x), images))\n lab = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'Annotations/480p/',"
+ },
+ {
+ "comment": "The code creates a custom dataloader for a dataset with two adjacent frames and their corresponding labels. It takes the images and label paths, appends them to a list of dictionaries, and handles any necessary padding to ensure frame numbers have 5 digits. The length of the dataloader is determined by the number of samples in the sample_list, and the __getitem__ method retrieves specific samples based on their index.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":456-484",
+ "content": " seq_name.strip())))\n lab_path = list(\n map(\n lambda x: os.path.join('Annotations/480p/', seq_name.strip(\n ), x), lab))\n for img_path, label_path in zip(images_path[:-1], lab_path[:-1]):\n tmp_dic = {\n 'img': img_path,\n 'label': label_path,\n 'seq_name': seq_name,\n 'frame_num': img_path.split('/')[-1].split('.')[0]\n }\n self.sample_list.append(tmp_dic)\n def __len__(self):\n return len(self.sample_list)\n def __getitem__(self, idx):\n tmp_sample = self.sample_list[idx]\n imgpath = tmp_sample['img']\n labelpath = tmp_sample['label']\n seqname = tmp_sample['seq_name']\n frame_num = int(tmp_sample['frame_num']) + 1\n next_frame = str(frame_num)\n while len(next_frame) != 5:\n next_frame = '0' + next_frame\n ###############################Processing two adjacent frames and labels"
+ },
+ {
+ "comment": "The code reads image and label files for a video sequence from their respective directories, converts them to numpy arrays of dtype float32 and int32 respectively for compatibility with the model's stack function. It also retrieves reference frame information from ref_frame_dic for the given sequence.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":485-505",
+ "content": " img2path = os.path.join('JPEGImages/480p/', seqname,\n next_frame + '.' + imgpath.split('.')[-1])\n img2 = cv2.imread(os.path.join(self.db_root_dir, img2path))\n img2 = np.array(img2, dtype=np.float32)\n img1 = cv2.imread(os.path.join(self.db_root_dir, imgpath))\n img1 = np.array(img1, dtype=np.float32)\n ###############\n label1 = Image.open(os.path.join(self.db_root_dir, labelpath))\n label2path = os.path.join('Annotations/480p/', seqname,\n next_frame + '.' + labelpath.split('.')[-1])\n label2 = Image.open(os.path.join(self.db_root_dir, label2path))\n label1 = np.array(\n label1, dtype=np.int32\n ) # fixed, uint8->int32, because layers.stack does not support uint8\n label2 = np.array(\n label2, dtype=np.int32\n ) # fixed, uint8->int32, because layers.stack does not support uint8\n ###################\n ref_tmp_dic = self.ref_frame_dic[seqname]"
+ },
+ {
+ "comment": "This code reads the reference image, scribble label, and ground truth frame from a dictionary. It then converts them to appropriate data types for processing. If rgb=True, it changes the color order. It also gets the total number of objects in the sequence. Finally, it creates a sample dictionary with all these elements.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":506-530",
+ "content": " ref_img = ref_tmp_dic['ref_frame']\n ref_scribble_label = ref_tmp_dic['scribble_label']\n ref_img = cv2.imread(os.path.join(self.db_root_dir, ref_img))\n ref_img = np.array(ref_img, dtype=np.float32)\n ref_frame_gt = ref_tmp_dic['ref_frame_gt']\n ref_frame_gt = Image.open(os.path.join(self.db_root_dir, ref_frame_gt))\n ref_frame_gt = np.array(\n ref_frame_gt, dtype=np.int32\n ) # fixed, uint8->int32, because layers.stack does not support uint8\n ref_frame_num = ref_tmp_dic['ref_frame_num']\n ###################\n if self.rgb:\n img1 = img1[:, :, [2, 1, 0]]\n img2 = img2[:, :, [2, 1, 0]]\n ref_img = ref_img[:, :, [2, 1, 0]]\n obj_num = self.seq_dict[seqname][-1]\n sample = {\n 'ref_img': ref_img,\n 'img1': img1,\n 'img2': img2,\n 'ref_scribble_label': ref_scribble_label,\n 'label1': label1,\n 'label2': label2,\n 'ref_frame_gt': ref_frame_gt"
+ },
+ {
+ "comment": "This code defines a function that creates a sample containing images, labels, and metadata for the Ma-Net model. It also includes a separate function to update the reference frame and label based on user input. The sample is then transformed using a specified transform if one is provided.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":531-561",
+ "content": " }\n if 'prev_round_label' in ref_tmp_dic:\n prev_round_label = ref_tmp_dic['prev_round_label']\n prev_round_label = prev_round_label.squeeze()\n prev_round_label = prev_round_label.numpy()\n sample = {\n 'ref_img': ref_img,\n 'img1': img1,\n 'img2': img2,\n 'ref_scribble_label': ref_scribble_label,\n 'label1': label1,\n 'label2': label2,\n 'ref_frame_gt': ref_frame_gt,\n 'prev_round_label': prev_round_label\n }\n sample['meta'] = {\n 'seq_name': seqname,\n 'frame_num': frame_num,\n 'obj_num': obj_num,\n 'ref_frame_num': ref_frame_num\n }\n if self.transform is not None:\n sample = self.transform(sample)\n return sample\n def update_ref_frame_and_label(self,\n round_scribble=None,\n frame_num=None,\n prev_round_label_dic=None):"
+ },
+ {
+ "comment": "Updating the reference frame and scribbles for each sequence in the dataset. If no frame number is given, uses the first frame from annotated_frames list. Ensures frame number is 5 digits long. Retrieves the corresponding reference image path and ground truth mask path. Reads the reference image. Resizes the image based on its height and width. Generates scribble masks for each frame using the provided scribble. If no frame number given, selects the first frame's scribble mask.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":562-584",
+ "content": " ##########Update reference frame and scribbles\n for seq in self.seqs:\n scribble = round_scribble[seq]\n if frame_num is None:\n scr_frame = annotated_frames(scribble)[0]\n else:\n scr_frame = frame_num[seq]\n scr_frame = int(scr_frame)\n scr_f = str(scr_frame)\n while len(scr_f) != 5:\n scr_f = '0' + scr_f\n ref_frame_path = os.path.join('JPEGImages/480p', seq,\n scr_f + '.jpg')\n #######################\n ref_frame_gt = os.path.join('Annotations/480p/', seq,\n scr_f + '.png')\n #########################\n ref_tmp = cv2.imread(os.path.join(self.db_root_dir, ref_frame_path))\n h_, w_ = ref_tmp.shape[:2]\n scribble_masks = scribbles2mask(scribble, (h_, w_))\n if frame_num is None:\n scribble_label = scribble_masks[scr_frame]"
+ },
+ {
+ "comment": "The code initializes a dictionary for reference frames, storing information such as reference frame path, scribble label, and ground truth data. If there is a previous round's label dictionary, it also includes the previous round's label in the current dictionary entry. It uses the database root directory to find the Scribbles folder and selects a random JSON file for each sequence.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":585-609",
+ "content": " else:\n scribble_label = scribble_masks[0]\n self.ref_frame_dic[seq] = {\n 'ref_frame': ref_frame_path,\n 'scribble_label': scribble_label,\n 'ref_frame_gt': ref_frame_gt,\n 'ref_frame_num': scr_frame\n }\n if prev_round_label_dic is not None:\n self.ref_frame_dic[seq] = {\n 'ref_frame': ref_frame_path,\n 'scribble_label': scribble_label,\n 'ref_frame_gt': ref_frame_gt,\n 'ref_frame_num': scr_frame,\n 'prev_round_label': prev_round_label_dic[seq]\n }\n def init_ref_frame_dic(self):\n self.ref_frame_dic = {}\n scribbles_path = os.path.join(self.db_root_dir, 'Scribbles')\n for seq in self.seqs:\n selected_json = np.random.choice(\n ['001.json', '002.json', '003.json'], 1)\n selected_json = selected_json[0]\n scribble = os.path.join(self.db_root_dir, 'Scribbles', seq,"
+ },
+ {
+ "comment": "Reading JSON file for annotated frame, extracting frame path and loading reference image using OpenCV, determining the shape of the reference image, extracting the mask from the scribble, storing reference frame path in ref_frame_dic.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":610-632",
+ "content": " selected_json)\n with open(scribble) as f:\n scribble = json.load(f)\n # print(scribble)\n scr_frame = annotated_frames(scribble)[0]\n scr_f = str(scr_frame)\n while len(scr_f) != 5:\n scr_f = '0' + scr_f\n ref_frame_path = os.path.join('JPEGImages/480p', seq,\n scr_f + '.jpg')\n ref_tmp = cv2.imread(\n os.path.join(self.db_root_dir, ref_frame_path))\n h_, w_ = ref_tmp.shape[:2]\n scribble_masks = scribbles2mask(scribble, (h_, w_))\n ########################\n ref_frame_gt = os.path.join('Annotations/480p/', seq,\n scr_f + '.png')\n ########################\n scribble_label = scribble_masks[scr_frame]\n self.ref_frame_dic[seq] = {\n 'ref_frame': ref_frame_path,"
+ },
+ {
+ "comment": "The code reads a list of sequences from the sequence_list file and checks if it exists. If the file does not exist, it returns False; otherwise, it loads the sequence dictionary using json.load() and then proceeds to preprocess each sequence by reading object masks and finding the number of objects in the masks. The code stores this information in a dictionary format for later use.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":633-661",
+ "content": " 'scribble_label': scribble_label,\n 'ref_frame_gt': ref_frame_gt,\n 'ref_frame_num': scr_frame\n }\n ########################\n def _check_preprocess(self):\n _seq_list_file = self.seq_list_file\n if not os.path.isfile(_seq_list_file):\n return False\n else:\n self.seq_dict = json.load(open(self.seq_list_file, 'r'))\n return True\n def _preprocess(self):\n self.seq_dict = {}\n for seq in self.seqs:\n # Read object masks and get number of objects\n name_label = np.sort(\n os.listdir(\n os.path.join(self.db_root_dir, 'Annotations/480p/', seq)))\n label_path = os.path.join(self.db_root_dir, 'Annotations/480p/',\n seq, name_label[0])\n _mask = np.array(Image.open(label_path))\n _mask_ids = np.unique(_mask)\n n_obj = _mask_ids[-1]\n self.seq_dict[seq] = list(range(1, n_obj + 1))"
+ },
+ {
+ "comment": "The code writes a JSON file containing video sequences and their corresponding dictionaries, which will be used for the dataset. It iterates over each sequence, formats the output as JSON strings in the file, and finishes by printing \"Preprocessing finished\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/davis_2017_f.py\":663-671",
+ "content": " with open(self.seq_list_file, 'w') as outfile:\n outfile.write('{{\\n\\t\"{:s}\": {:s}'.format(\n self.seqs[0], json.dumps(self.seq_dict[self.seqs[0]])))\n for ii in range(1, len(self.seqs)):\n outfile.write(',\\n\\t\"{:s}\": {:s}'.format(\n self.seqs[ii], json.dumps(self.seq_dict[self.seqs[ii]])))\n outfile.write('\\n}\\n')\n print('Preprocessing finished')"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/29e71c71-93fa-4b6d-8c00-8886b0e95962.json b/docs/doc/29e71c71-93fa-4b6d-8c00-8886b0e95962.json
new file mode 100644
index 000000000..410ce0133
--- /dev/null
+++ b/docs/doc/29e71c71-93fa-4b6d-8c00-8886b0e95962.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code uses PaddleVideo library to classify videos, perform feature extraction, and predict bounding box results. It logs information, saves outputs if needed, and writes inference results into a JSON file.",
+ "details": [
+ {
+ "comment": "This code is for Baidu Cloud action and loads the model using the BMN (Behaved Motion Network) model from a given configuration file. It also defines a function to classify videos by predicting their actions and prints the information about the video being processed. The code uses logger for logging the information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/extractor/extract_bmn_for_tabletennis.py\":0-49",
+ "content": "#!./python27-gcc482/bin/python\n# coding: utf-8\n\"\"\"\nBAIDU CLOUD action\n\"\"\"\nimport os\nimport sys\nimport pickle\nimport json\nimport time\nimport shutil\nimport numpy as np\nsys.path.append(\n \"/workspace/bianjiang03/App_TableTennis/PaddleVideo/FootballAction/predict/action_detect\"\n)\nimport models.bmn_infer as prop_model\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config, print_configs\nimport utils.config_utils as config_utils\nimport logger\nlogger = logger.Logger()\ndef load_model(cfg_file=\"configs/configs.yaml\"):\n \"\"\"\n load_model\n \"\"\"\n logger.info(\"load model ... \")\n global infer_configs\n infer_configs = parse_config(cfg_file)\n print_configs(infer_configs, \"Infer\")\n t0 = time.time()\n global prop_model\n prop_model = prop_model.InferModel(infer_configs)\n t1 = time.time()\n logger.info(\"step0: load model time: {} min\\n\".format((t1 - t0) * 1.0 / 60))\ndef video_classify(video_name, dataset_dir):\n \"\"\"\n extract_feature\n \"\"\"\n logger.info('predict ... ')\n logger.info(video_name)"
+ },
+ {
+ "comment": "This code performs video feature extraction and proposal generation using the PaddleVideo library. It first loads the video features from a pickle file, then predicts bounding box minimum notation (Bmn) results using a pre-trained model. Finally, it returns the Bmn results and saves them in an output directory if it doesn't already exist.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/extractor/extract_bmn_for_tabletennis.py\":51-83",
+ "content": " # step 1: extract feature\n feature_path = dataset_dir + video_name\n video_features = pickle.load(open(feature_path, 'rb'))\n print('===video_features===', video_name)\n # step2: get proposal\n t0 = time.time()\n bmn_results = prop_model.predict(infer_configs, material=video_features)\n t1 = time.time()\n logger.info(np.array(bmn_results).shape)\n logger.info(\"step2: proposal time: {} min\".format((t1 - t0) * 1.0 / 60))\n return bmn_results\nif __name__ == '__main__':\n dataset_dir = '/workspace/bianjiang03/DATA/Features_competition_test_A/'\n output_dir = '/workspace/bianjiang03/DATA'\n if not os.path.exists(output_dir + '/Output_for_bmn'):\n os.mkdir(output_dir + '/Output_for_bmn')\n results = []\n load_model()\n directory = os.fsencode(dataset_dir)\n for file in os.listdir(directory):\n filename = os.fsdecode(file)\n bmn_results = video_classify(filename, dataset_dir)\n results.append({\n 'video_name': filename.split('.pkl')[0],\n 'num_proposal': len(bmn_results),"
+ },
+ {
+ "comment": "This code segment writes the inference results into a JSON file. It first stores the 'bmn_results' dictionary and then writes it to a file named \"Output_for_bmn/prop.json\". The JSON data is formatted for readability with indentation and using UTF-8 encoding. Once writing is complete, it prints \"Done with the inference!\" indicating successful execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/extractor/extract_bmn_for_tabletennis.py\":84-92",
+ "content": " 'bmn_results': bmn_results\n })\n with open(output_dir + '/Output_for_bmn/prop.json', 'w',\n encoding='utf-8') as f:\n data = json.dumps(results, indent=4, ensure_ascii=False)\n f.write(data)\n print('Done with the inference!')"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2a3ee5e8-0ed0-470e-8093-d3d61b489df6.json b/docs/doc/2a3ee5e8-0ed0-470e-8093-d3d61b489df6.json
new file mode 100644
index 000000000..cb12621a6
--- /dev/null
+++ b/docs/doc/2a3ee5e8-0ed0-470e-8093-d3d61b489df6.json
@@ -0,0 +1,140 @@
+{
+ "summary": "OctConv3D is a configurable 3D convolutional layer in TransNetV2's backbone, utilizing features such as max pooling and SDDCNNV2 blocks for shot transition detection. ConvNextV2 applies feature extraction and pooling, while the code defines models using Linear and ConvexCombinationRegularization layers for classification tasks.",
+ "details": [
+ {
+ "comment": "This code defines a 3D convolutional neural network layer called OctConv3D. It takes input and output channels, kernel size, dilation rate, alpha (for octave pooling), use_bias flag, and initializer as parameters for creating the layer. This layer can be used in other models by utilizing the BACKBONES registry.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":0-27",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as functional\nimport random\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\nclass OctConv3D(nn.Layer):\n def __init__(self, in_filters, filters, kernel_size=3, dilation_rate=(1, 1, 1), alpha=0.25,\n use_bias=True, kernel_initializer=nn.initializer.KaimingNormal()):\n super(OctConv3D, self).__init__()"
+ },
+ {
+ "comment": "Defines a 3D Convolutional network with interleaved low and high-resolution paths. Low-to-high and high-to-low convolutions are performed to maintain spatial resolution while reducing dimensionality for the TransNetV2 backbone model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":29-42",
+ "content": " self.low_channels = int(filters * alpha)\n self.high_channels = filters - self.low_channels\n self.high_to_high = nn.Conv3D(in_filters, self.high_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),\n weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.high_to_low = nn.Conv3D(self.high_channels, self.low_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),\n weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=False)\n self.low_to_high = nn.Conv3D(in_filters, self.high_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),"
+ },
+ {
+ "comment": "This code defines a TransNetV2 backbone model for video analysis. It includes convolutional layers, an upsampler, and downsampler to process input data. The `pad_to` function pads the tensor with zeros to match a target shape, useful for maintaining consistent dimensions throughout the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":43-57",
+ "content": " weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=False)\n self.low_to_low = nn.Conv3D(self.high_channels, self.low_channels, kernel_size=kernel_size,\n dilation=dilation_rate, padding=(dilation_rate[0], 1, 1),\n weight_attr=ParamAttr(initializer=kernel_initializer),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.upsampler = nn.Upsample(size=(1, 2, 2), data_format='NCDHW')\n self.downsampler = nn.AvgPool3D(kernel_size=(1, 2, 2), stride=(1, 2, 2), padding=(0, 1, 1))\n @staticmethod\n def pad_to(tensor, target_shape):\n shape = tensor.shape\n padding = [[0, tar - curr] for curr, tar in zip(shape, target_shape)]\n return functional.pad(tensor, padding, \"CONSTANT\", data_format='NCDHW')"
+ },
+ {
+ "comment": "The code defines a forward function that takes inputs and performs high-to-high, high-to-low, low-to-high, and low-to-low transformations. It also includes a Conv3DConfigurable class with parameters for in_filters, filters, dilation_rate, separable, octave, and use_bias. The code asserts that separable and octave cannot both be True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":59-89",
+ "content": " @staticmethod\n def crop_to(tensor, target_width, target_height):\n return tensor[:, :, :target_height, :target_width]\n def forward(self, inputs):\n low_inputs, high_inputs = inputs\n high_to_high = self.high_to_high(high_inputs)\n high_to_low = self.high_to_low(self.downsampler(high_inputs))\n low_to_high = self.upsampler(self.low_to_high(low_inputs))\n low_to_low = self.low_to_low(low_inputs)\n high_output = high_to_high[:, :, :, :low_to_high.shape[3], :low_to_high.shape[4]] + low_to_high\n low_output = low_to_low + high_to_low[:, :, :, :low_to_low.shape[3], :low_to_low.shape[4]]\n return low_output, high_output\nclass Conv3DConfigurable(nn.Layer):\n def __init__(self,\n in_filters,\n filters,\n dilation_rate,\n separable=True,\n octave=False,\n use_bias=True):\n super(Conv3DConfigurable, self).__init__()\n assert not (separable and octave)\n if separable:"
+ },
+ {
+ "comment": "The code initializes a Conv3D layer and an optional octave convolution layer for the TransNetV2 backbone. The Conv3D layers apply 3x3 kernel with varying dilation rates, while the optional OctConv3D layer has a 3x1x1 kernel and dilation rate (dilation_rate, 1, 1). The layers are added to a LayerList for further processing in the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":90-103",
+ "content": " conv1 = nn.Conv3D(in_filters, 2 * filters, kernel_size=(1, 3, 3),\n dilation=(1, 1, 1), padding=(0, 1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.KaimingNormal()),\n bias_attr=False)\n conv2 = nn.Conv3D(2 * filters, filters, kernel_size=(3, 1, 1),\n dilation=(dilation_rate, 1, 1), padding=(dilation_rate, 0, 0),\n weight_attr=ParamAttr(initializer=nn.initializer.KaimingNormal()),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.layers = nn.LayerList([conv1, conv2])\n elif octave:\n conv = OctConv3D(in_filters, filters, kernel_size=3, dilation_rate=(dilation_rate, 1, 1),\n use_bias=use_bias,\n kernel_initializer=nn.initializer.KaimingNormal())"
+ },
+ {
+ "comment": "This code defines a neural network backbone called TransnetV2, which consists of convolutional layers. The Conv3DConfigurable class is used to configure the layers with specified input and output filters, kernel size, dilation rate, padding, and whether to use bias or batch normalization. The DilatedDCNNV2 class extends this concept by allowing the choice between octave convolution and batch normalization. Both classes inherit from nn.Layer and have a forward method for processing inputs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":104-130",
+ "content": " self.layers = [conv]\n else:\n conv = nn.Conv3D(in_filters, filters, kernel_size=3,\n dilation=(dilation_rate, 1, 1), padding=(dilation_rate, 1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.KaimingNormal()),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.)) if use_bias else use_bias)\n self.layers = nn.LayerList([conv])\n def forward(self, inputs):\n x = inputs\n for layer in self.layers:\n x = layer(x)\n return x\nclass DilatedDCNNV2(nn.Layer):\n def __init__(self,\n in_filters,\n filters,\n batch_norm=True,\n activation=None,\n octave_conv=False):\n super(DilatedDCNNV2, self).__init__()\n assert not (octave_conv and batch_norm)\n self.Conv3D_1 = Conv3DConfigurable(in_filters, filters, 1, use_bias=not batch_norm, octave=octave_conv)"
+ },
+ {
+ "comment": "This code defines a TransNetV2 model, which uses multiple Conv3D layers to process input data. The model includes configurable convolution layers with different filter sizes (2, 4, and 8), batch normalization, and activation functions. The forward method applies these layers to the inputs and concatenates their outputs along the channel dimension.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":131-149",
+ "content": " self.Conv3D_2 = Conv3DConfigurable(in_filters, filters, 2, use_bias=not batch_norm, octave=octave_conv)\n self.Conv3D_4 = Conv3DConfigurable(in_filters, filters, 4, use_bias=not batch_norm, octave=octave_conv)\n self.Conv3D_8 = Conv3DConfigurable(in_filters, filters, 8, use_bias=not batch_norm, octave=octave_conv)\n self.octave = octave_conv\n self.bn = nn.BatchNorm3D(filters * 4, momentum=0.99, epsilon=1e-03,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n ) if batch_norm else None\n self.activation = activation\n def forward(self, inputs):\n conv1 = self.Conv3D_1(inputs)\n conv2 = self.Conv3D_2(inputs)\n conv3 = self.Conv3D_4(inputs)\n conv4 = self.Conv3D_8(inputs)\n # shape of convi[j]/convi is [B, 3, T, H, W], concat in channel dimension\n if self.octave:"
+ },
+ {
+ "comment": "The code defines a StackedDDCNNV2 class that is a type of neural network layer. It takes in parameters such as number of input filters, number of blocks, and output filters. The class uses convolutions with optional batch normalization and activation functions. The convolutions can be either octave or non-octave depending on the parameter setting. The pooling type is either max or average pooling, and there is a stochastic depth drop probability parameter.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":150-178",
+ "content": " x = [paddle.concat([conv1[0], conv2[0], conv3[0], conv4[0]], axis=1),\n paddle.concat([conv1[1], conv2[1], conv3[1], conv4[1]], axis=1)]\n else:\n x = paddle.concat([conv1, conv2, conv3, conv4], axis=1)\n if self.bn is not None:\n x = self.bn(x)\n if self.activation is not None:\n if self.octave:\n x = [self.activation(x[0]), self.activation(x[1])]\n else:\n x = self.activation(x)\n return x\nclass StackedDDCNNV2(nn.Layer):\n def __init__(self,\n in_filters,\n n_blocks,\n filters,\n shortcut=True,\n use_octave_conv=False,\n pool_type=\"avg\",\n stochastic_depth_drop_prob=0.0):\n super(StackedDDCNNV2, self).__init__()\n assert pool_type == \"max\" or pool_type == \"avg\"\n if use_octave_conv and pool_type == \"max\":\n print(\"WARN: Octave convolution was designed with average pooling, not max pooling.\")"
+ },
+ {
+ "comment": "Initializes backbone layers and sets parameters. Applies octave convolution if use_octave_conv is True, and performs max or avg pooling depending on pool_type. Stochastic depth is applied with probability stochastic_depth_drop_prob. Forward pass applies blocks of DDCNNV2, concatenates and applies ReLU activation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":180-206",
+ "content": " self.shortcut = shortcut\n self.DDCNN = nn.LayerList([\n DilatedDCNNV2(in_filters if i == 1 else filters * 4, filters, octave_conv=use_octave_conv,\n activation=functional.relu if i != n_blocks else None) for i in range(1, n_blocks + 1)\n ])\n self.pool = nn.MaxPool3D(kernel_size=(1, 2, 2)) if pool_type == \"max\" else nn.AvgPool3D(kernel_size=(1, 2, 2))\n self.octave = use_octave_conv\n self.stochastic_depth_drop_prob = stochastic_depth_drop_prob\n def forward(self, inputs):\n x = inputs\n shortcut = None\n if self.octave:\n x = [self.pool(x), x]\n for block in self.DDCNN:\n x = block(x)\n if shortcut is None:\n shortcut = x\n # shape of x[i] is [B, 3, T, H, W], concat in channel dimension\n if self.octave:\n x = paddle.concat([x[0], self.pool(x[1])], axis=1)\n x = functional.relu(x)\n if self.shortcut is not None:\n if self.stochastic_depth_drop_prob != 0.:"
+ },
+ {
+ "comment": "This code defines a ResNetBlock class that consists of Conv2D layer and BatchNorm2D layer. The stochastic depth is applied during training by randomly dropping connections with a specified probability, while in non-octave cases, it applies pooling to the output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":207-231",
+ "content": " if self.training:\n if random.random() < self.stochastic_depth_drop_prob:\n x = shortcut\n else:\n x = x + shortcut\n else:\n x = (1 - self.stochastic_depth_drop_prob) * x + shortcut\n else:\n x += shortcut\n if not self.octave:\n x = self.pool(x)\n return x\nclass ResNetBlock(nn.Layer):\n def __init__(self, in_filters, filters, strides=(1, 1)):\n super(ResNetBlock, self).__init__()\n self.conv1 = nn.Conv2D(in_filters, filters, kernel_size=(3, 3), stride=strides, padding=(1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.bn1 = nn.BatchNorm2D(filters,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))"
+ },
+ {
+ "comment": "The code defines a Conv2D layer and BatchNorm2D layer in the `TransNetV2` class, followed by a forward function that applies these layers in sequence. The ResNetFeatures class initializes a Conv2D layer for extracting features from input images. Both classes are part of an object-oriented model architecture.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":233-259",
+ "content": " self.conv2 = nn.Conv2D(filters, filters, kernel_size=(3, 3), padding=(1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.bn2 = nn.BatchNorm2D(filters,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))\n def forward(self, inputs):\n x = self.conv1(inputs)\n x = self.bn1(x)\n x = functional.relu(x)\n x = self.conv2(x)\n x = self.bn2(x)\n shortcut = inputs\n x += shortcut\n return functional.relu(x)\nclass ResNetFeatures(nn.Layer):\n def __init__(self, in_filters=3,\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]):\n super(ResNetFeatures, self).__init__()\n self.conv1 = nn.Conv2D(in_channels=in_filters, out_channels=64, kernel_size=(7, 7),"
+ },
+ {
+ "comment": "This code is for TransNetV2 backbone model initialization. It includes a convolution layer with padding, batch normalization, max pooling, and ResNetBlocks (layer2a, layer2b). The forward function performs normalization, reshaping, convolution, and batch normalization on the input.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":260-281",
+ "content": " stride=(2, 2), padding=(3, 3),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.bn1 = nn.BatchNorm2D(num_features=64, momentum=0.99, epsilon=1e-03,\n weight_attr=ParamAttr(initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n )\n self.max_pool = nn.MaxPool2D(kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n self.layer2a = ResNetBlock(64, 64)\n self.layer2b = ResNetBlock(64, 64)\n self.mean = paddle.to_tensor(mean)\n self.std = paddle.to_tensor(std)\n def forward(self, inputs):\n shape = inputs.shape\n x = paddle.reshape(inputs, [shape[0] * shape[2], shape[1], shape[3], shape[4]])\n x = (x - self.mean) / self.std\n x = self.conv1(x)\n x = self.bn1(x)"
+ },
+ {
+ "comment": "This code defines a class \"FrameSimilarity\" that takes in filters, similarity dimension, lookup window, output dimension, stop_gradient flag, and use_bias as parameters. It initializes the layer with a projection linear layer and an fc linear layer. The projection layer maps input features to a specified similarity dimension using XavierUniform initialization. The fc layer maps the lookup window to the output dimension, using XavierUniform initialization for weights and Constant initialization for biases.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":282-306",
+ "content": " x = functional.relu(x)\n x = self.max_pool(x)\n x = self.layer2a(x)\n x = self.layer2b(x)\n new_shape = x.shape\n x = paddle.reshape(x, [shape[0], new_shape[1], shape[2], new_shape[2], new_shape[3]])\n return x\nclass FrameSimilarity(nn.Layer):\n def __init__(self,\n in_filters,\n similarity_dim=128,\n lookup_window=101,\n output_dim=128,\n stop_gradient=False,\n use_bias=False):\n super(FrameSimilarity, self).__init__()\n self.projection = nn.Linear(in_filters, similarity_dim,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=use_bias)\n self.fc = nn.Linear(lookup_window, output_dim,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))"
+ },
+ {
+ "comment": "The code initializes a TransNetV2 model with lookup window and stop_gradient options. It then calculates similarities between time windows using batch mean, transpose, projection, and normalization. Finally, it pads the similarities for further calculations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":308-329",
+ "content": " self.lookup_window = lookup_window\n self.stop_gradient = stop_gradient\n assert lookup_window % 2 == 1, \"`lookup_window` must be odd integer\"\n def forward(self, inputs):\n x = paddle.concat([paddle.mean(x, axis=[3, 4]) for x in inputs], axis=1)\n x = paddle.transpose(x, (0, 2, 1))\n if self.stop_gradient:\n x = x.stop_gradient\n x = self.projection(x)\n x = functional.normalize(x, p=2, axis=2)\n batch_size = paddle.slice(x.shape, starts=[0], ends=[1], axes=[0]) if x.shape[0] == -1 else x.shape[0]\n time_window = x.shape[1]\n similarities = paddle.bmm(x, x.transpose([0, 2, 1])) # [batch_size, time_window, time_window]\n similarities_padded = functional.pad(similarities,\n [(self.lookup_window - 1) // 2, (self.lookup_window - 1) // 2],\n data_format='NCL')\n batch_indices = paddle.arange(0, batch_size).reshape([batch_size, 1, 1])"
+ },
+ {
+ "comment": "This code is calculating the indices for gathering similarities from a padded tensor. It tiles and stacks batch, time, and lookup indices to create an array of valid indices. Then it uses these indices to gather similarities from the padded tensor and applies ReLU activation on top of an FC layer to return the output. The ConvexCombinationRegularization class initializes a projection layer with specified parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":330-345",
+ "content": " batch_indices = paddle.tile(batch_indices, [1, time_window, self.lookup_window])\n time_indices = paddle.arange(0, time_window).reshape([1, time_window, 1])\n time_indices = paddle.tile(time_indices, [batch_size, 1, self.lookup_window])\n lookup_indices = paddle.arange(0, self.lookup_window).reshape([1, 1, self.lookup_window])\n lookup_indices = paddle.tile(lookup_indices, [batch_size, time_window, 1]) + time_indices\n indices = paddle.stack([batch_indices, time_indices, lookup_indices], -1)\n similarities = paddle.gather_nd(similarities_padded, indices)\n return functional.relu(self.fc(similarities))\nclass ConvexCombinationRegularization(nn.Layer):\n def __init__(self, in_filters, filters=32, delta_scale=10., loss_weight=0.01):\n super(ConvexCombinationRegularization, self).__init__()\n self.projection = nn.Conv3D(in_filters, filters, kernel_size=1, dilation=1, padding=(0, 0, 0),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),"
+ },
+ {
+ "comment": "This code defines a Conv3D model for the TransNetV2 backbone. It has a projection layer, relu activation, and takes in image_inputs and feature_inputs. The forward function processes these inputs, extracting the first and last frame windows.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":346-363",
+ "content": " bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))\n self.features = nn.Conv3D((filters * 3), filters * 2,\n kernel_size=(3, 3, 3), dilation=1, padding=(1, 1, 1),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))\n self.dense = nn.Linear(64, 1, weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()), bias_attr=True)\n self.loss = nn.SmoothL1Loss(reduction='none')\n self.delta_scale = delta_scale\n self.loss_weight = loss_weight\n def forward(self, image_inputs, feature_inputs):\n x = feature_inputs\n x = self.projection(x)\n x = functional.relu(x)\n batch_size = x.shape[0]\n window_size = x.shape[2]\n first_frame = paddle.tile(x[:, :, :1], [1, 1, window_size, 1, 1])\n last_frame = paddle.tile(x[:, :, -1:], [1, 1, window_size, 1, 1])"
+ },
+ {
+ "comment": "This code is part of the TransnetV2 model in PaddleVideo. It concatenates frames, processes them through layers, and calculates alpha values for first and last images. It then combines these images based on the calculated alphas and performs loss calculation using a loss function. The ColorHistograms layer is initialized with a linear transformation layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":364-389",
+ "content": " x = paddle.concat([x, first_frame, last_frame], 1)\n x = self.features(x)\n x = functional.relu(x)\n x = paddle.mean(x, axis=[3, 4])\n x = paddle.transpose(x, (0, 2, 1))\n alpha = self.dense(x)\n alpha = paddle.transpose(alpha, (0, 2, 1))\n first_img = paddle.tile(image_inputs[:, :, :1], [1, 1, window_size, 1, 1])\n last_img = paddle.tile(image_inputs[:, :, -1:], [1, 1, window_size, 1, 1])\n alpha_ = functional.sigmoid(alpha)\n alpha_ = paddle.reshape(alpha_, [batch_size, 1, window_size, 1, 1])\n predictions_ = (alpha_ * first_img + (1 - alpha_) * last_img)\n loss_ = self.loss(label=image_inputs / self.delta_scale, input=predictions_ / self.delta_scale)\n loss_ = self.loss_weight * paddle.mean(loss_)\n return alpha, loss_\nclass ColorHistograms(nn.Layer):\n def __init__(self,\n lookup_window=101,\n output_dim=None):\n super(ColorHistograms, self).__init__()\n self.fc = nn.Linear(lookup_window, output_dim,"
+ },
+ {
+ "comment": "This code defines a function to compute color histograms of frames. It first converts the frame values to int32, then defines a function get_bin which extracts and scales RGB values. The batch size is extracted from the frames shape, and the frames are flattened into a 3-dimensional array if the number of channels is 3 or 6.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":390-410",
+ "content": " weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=0.))) if output_dim is not None else None\n self.lookup_window = lookup_window\n assert lookup_window % 2 == 1, \"`lookup_window` must be odd integer\"\n def compute_color_histograms(self, frames):\n frames = frames.astype('int32')\n def get_bin(frames):\n # returns 0 .. 511\n R, G, B = frames[:, :, 0], frames[:, :, 1], frames[:, :, 2]\n R, G, B = R // 32, G // 32, B // 32\n return (R * 64) + (G * 8) + B\n batch_size = paddle.slice(frames.shape, starts=[0], ends=[1], axes=[0]) if frames.shape[0] == -1 else frames.shape[0]\n time_window, height, width, no_channels = frames.shape[1:]\n assert no_channels == 3 or no_channels == 6\n if no_channels == 3:\n frames_flatten = frames.reshape([-1, height * width, 3])"
+ },
+ {
+ "comment": "This code computes color histograms for each frame in a video and then calculates similarities between frames using batch matrix multiplication. It first checks the input shape to determine whether it should extract only the batch size and time window or use the total shape. It then reshapes and bins the frame values, normalizes the histograms, and finally computes the similarity matrix for each frame pair. The purpose is likely for video sequence analysis or comparison.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":411-428",
+ "content": " else:\n frames_flatten = frames.reshape([-1, height * width * 2, 3])\n binned_values = get_bin(frames_flatten)\n frame_bin_prefix = (paddle.arange(0, batch_size * time_window) * 512).reshape([-1, 1])\n binned_values = (binned_values + frame_bin_prefix).reshape([-1, 1])\n histograms = paddle.zeros_like(frame_bin_prefix, dtype='int32').tile([512]).reshape([-1])\n histograms = histograms.scatter_nd_add(binned_values, paddle.ones_like(binned_values, dtype='int32').reshape([-1]))\n histograms = histograms.reshape([batch_size, time_window, 512]).astype('float32')\n histograms_normalized = functional.normalize(histograms, p=2, axis=2)\n return histograms_normalized\n def forward(self, inputs):\n x = self.compute_color_histograms(inputs)\n batch_size = paddle.slice(x.shape, starts=[0], ends=[1], axes=[0]) if x.shape[0] == -1 else x.shape[0]\n time_window = x.shape[1]\n similarities = paddle.bmm(x, x.transpose([0, 2, 1])) # [batch_size, time_window, time_window]"
+ },
+ {
+ "comment": "This code performs lookup on a padded tensor using gathered indices from batch, time window, and lookup window. It then applies an optional fully connected layer with ReLU activation function if present.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":429-445",
+ "content": " similarities_padded = functional.pad(similarities,\n [(self.lookup_window - 1) // 2, (self.lookup_window - 1) // 2],\n data_format='NCL')\n batch_indices = paddle.arange(0, batch_size).reshape([batch_size, 1, 1])\n batch_indices = paddle.tile(batch_indices, [1, time_window, self.lookup_window])\n time_indices = paddle.arange(0, time_window).reshape([1, time_window, 1])\n time_indices = paddle.tile(time_indices, [batch_size, 1, self.lookup_window])\n lookup_indices = paddle.arange(0, self.lookup_window).reshape([1, 1, self.lookup_window])\n lookup_indices = paddle.tile(lookup_indices, [batch_size, time_window, 1]) + time_indices\n indices = paddle.stack([batch_indices, time_indices, lookup_indices], -1)\n similarities = paddle.gather_nd(similarities_padded, indices)\n if self.fc is not None:\n return functional.relu(self.fc(similarities))\n return similarities"
+ },
+ {
+ "comment": "The code defines the TransNetV2 model, a deep network architecture for shot transition detection. It has multiple input sources and various options to use or not use different features and operations. The mean and std are provided as initialization parameters to standardize the input data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":448-472",
+ "content": "@BACKBONES.register()\nclass TransNetV2(nn.Layer):\n \"\"\"TransNetV2 model from\n `\"TransNet V2: An effective deep network architecture for fast shot transition detection\" `_\n \"\"\"\n def __init__(self,\n F=16, L=3, S=2, D=1024,\n use_many_hot_targets=True,\n use_frame_similarity=True,\n use_color_histograms=True,\n use_mean_pooling=False,\n dropout_rate=0.5,\n use_convex_comb_reg=False,\n use_resnet_features=False,\n use_resnet_like_top=False,\n frame_similarity_on_last_layer=False,\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]):\n super(TransNetV2, self).__init__()\n self.mean = np.array(mean, np.float32).reshape([1, 3, 1, 1]) * 255\n self.std = np.array(std, np.float32).reshape([1, 3, 1, 1]) * 255\n self.use_resnet_features = use_resnet_features\n s"
+ },
+ {
+ "comment": "Code snippet is from PaddleVideo's TransNetV2 model. It checks if use_resnet_features is True and if so, initializes resnet_layers with ResNetFeatures. If resnet_like_top is also True, it then initializes resnet_like_top_conv and resnet_like_top_bn for ResNet-like top layers with specified parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":472-483",
+ "content": "elf.resnet_layers = ResNetFeatures(in_filters=3, mean=self.mean, std=self.std) if self.use_resnet_features else None\n self.resnet_like_top = use_resnet_like_top\n if self.resnet_like_top:\n self.resnet_like_top_conv = nn.Conv3D(64 if self.use_resnet_features else 3, 32, kernel_size=(3, 7, 7),\n stride=(1, 2, 2),\n padding=(1, 3, 3),\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=False)\n self.resnet_like_top_bn = nn.BatchNorm3D(32, momentum=0.99, epsilon=1e-03,\n weight_attr=ParamAttr(\n initializer=nn.initializer.Constant(value=1.)),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.)))"
+ },
+ {
+ "comment": "This code initializes the model components of a TransNetv2 backbone. It sets up max pooling, creates a LayerList for SDDCNNV2 blocks, initializes frame similarity and color histogram layers based on flags, and includes dropout layer if needed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":484-507",
+ "content": " self.resnet_like_top_max_pool = nn.MaxPool3D(kernel_size=(1, 3, 3), stride=(1, 2, 2),\n padding=(0, 1, 1))\n if self.resnet_like_top:\n in_filters = 32\n elif self.use_resnet_features:\n in_filters = 64\n else:\n in_filters = 3\n self.SDDCNN = nn.LayerList(\n [StackedDDCNNV2(in_filters=in_filters, n_blocks=S, filters=F,\n stochastic_depth_drop_prob=0.)] +\n [StackedDDCNNV2(in_filters=(F * 2 ** (i - 1)) * 4, n_blocks=S, filters=F * 2 ** i) for i in range(1, L)]\n )\n self.frame_sim_layer = FrameSimilarity(\n sum([(F * 2 ** i) * 4 for i in range(L)]), lookup_window=101, output_dim=128, similarity_dim=128,\n use_bias=True\n ) if use_frame_similarity else None\n self.color_hist_layer = ColorHistograms(\n lookup_window=101, output_dim=128\n ) if use_color_histograms else None\n self.dropout = nn.Dropout(dropout_rate) if dropout_rate is not None else None"
+ },
+ {
+ "comment": "This code initializes a neural network model with a linear layer (`self.fc1`) that takes an input dimension of 512 if certain conditions are met, otherwise it takes the output_dim calculated earlier. The layer has D output dimensions and uses Xavier uniform initialization for weights and constant initialization for biases. Additionally, there's another linear layer (`self.cls_layer1`) with 1 output dimension that is initialized with Xavier uniform initialization for weights and a constant value of 0 for biases. It takes an input dimension of either 1152 or D based on whether frame similarity is added to the last layer or not.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":509-525",
+ "content": " output_dim = ((F * 2 ** (L - 1)) * 4) * 3 * 6 # 3x6 for spatial dimensions\n if use_frame_similarity: output_dim += 128\n if use_color_histograms: output_dim += 128\n self.use_mean_pooling = use_mean_pooling\n self.has_downsample = False\n if self.use_resnet_features or self.resnet_like_top or self.use_mean_pooling:\n self.has_downsample = True\n self.fc1 = nn.Linear(512 if self.has_downsample else output_dim, D,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n )\n self.frame_similarity_on_last_layer = frame_similarity_on_last_layer\n self.cls_layer1 = nn.Linear(1152 if self.frame_similarity_on_last_layer else D, 1,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))"
+ },
+ {
+ "comment": "The code defines a model with two layers, a Linear layer and ConvexCombinationRegularization, depending on the use_many_hot_targets and use_convex_comb_reg parameters. The Linear layer has 1 output for each frame, unless frame_similarity_on_last_layer is set, in which case it has D outputs. If use_many_hot_targets is False, the layer is None. The forward function receives inputs of shape [B, T, H, W, 3] and performs transpose, resnet_features processing (if use_resnet_features=True), and normalization to apply the model layers. It also clips the input values between 0 and 255 before applying the regularization if use_convex_comb_reg is True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":526-547",
+ "content": " )\n self.cls_layer2 = nn.Linear(1152 if self.frame_similarity_on_last_layer else D, 1,\n weight_attr=ParamAttr(initializer=nn.initializer.XavierUniform()),\n bias_attr=ParamAttr(initializer=nn.initializer.Constant(value=0.))\n ) if use_many_hot_targets else None\n self.convex_comb_reg = ConvexCombinationRegularization(\n in_filters=(F * 2 ** (L - 1) * 4)) if use_convex_comb_reg else None\n def forward(self, inputs):\n assert list(inputs.shape[2:]) == [27, 48, 3] and inputs.dtype == paddle.float32, \\\n \"incorrect input type and/or shape\"\n out_dict = {}\n # shape [B, T, H, W, 3] to shape [B, 3, T, H, W]\n x = inputs.transpose([0, 4, 1, 2, 3])\n if self.use_resnet_features:\n x = self.resnet_layers(x)\n else:\n x = x / 255.\n inputs = inputs.clip(min=0).astype('uint8')\n if self.resnet_like_top:"
+ },
+ {
+ "comment": "This code performs feature extraction and pooling operations for a ConvNextV2 backbone model. It applies residual blocks, top convolutions, batch normalization, and max pooling to the input. Then it calculates convex combination regression if required. The code either applies mean pooling or 3D reshaping based on the use_mean_pooling flag. Finally, it concatenates frame similarity layer outputs and color histogram layer outputs before performing fully connected layer calculations and applying relu activation and dropout if necessary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":548-570",
+ "content": " x = self.resnet_like_top_conv(x)\n x = self.resnet_like_top_bn(x)\n x = self.resnet_like_top_max_pool(x)\n block_features = []\n for block in self.SDDCNN:\n x = block(x)\n block_features.append(x)\n if self.convex_comb_reg is not None:\n out_dict[\"alphas\"], out_dict[\"comb_reg_loss\"] = self.convex_comb_reg(inputs.transpose([0, 4, 1, 2, 3]), x)\n if self.use_mean_pooling:\n x = paddle.mean(x, axis=[3, 4])\n x = x.transpose([0, 2, 1])\n else:\n x = x.transpose([0, 2, 3, 4, 1])\n x = x.reshape([x.shape[0], x.shape[1], x.shape[2]*x.shape[3]*x.shape[4]])\n if self.frame_sim_layer is not None:\n x = paddle.concat([self.frame_sim_layer(block_features), x], 2)\n if self.color_hist_layer is not None:\n x = paddle.concat([self.color_hist_layer(inputs), x], 2)\n x = self.fc1(x)\n x = functional.relu(x)\n if self.dropout is not None:\n x = self.dropout(x)"
+ },
+ {
+ "comment": "This code checks if the frame similarity layer and classifier layers are not None, then performs a concatenation operation on block features and x. It applies the classifier layer to the resulting output and optionally applies another classifier layer. The function returns one_hot and an optional out_dict if they exist.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/transnetv2.py\":571-580",
+ "content": " if self.frame_sim_layer is not None and self.frame_similarity_on_last_layer:\n x = paddle.concat([self.frame_sim_layer(block_features), x], 2)\n one_hot = self.cls_layer1(x)\n if self.cls_layer2 is not None:\n out_dict[\"many_hot\"] = self.cls_layer2(x)\n if len(out_dict) > 0:\n return one_hot, out_dict\n return one_hot"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2a982ad5-ef2a-4e9a-baf5-63578a23e618.json b/docs/doc/2a982ad5-ef2a-4e9a-baf5-63578a23e618.json
new file mode 100644
index 000000000..afd7152ea
--- /dev/null
+++ b/docs/doc/2a982ad5-ef2a-4e9a-baf5-63578a23e618.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code calculates recall for object detection by iterating through images, selecting proposals based on scores, and calculating IoU between these proposals and ground truth boxes. It defines `recalls()` and `eval_recalls()`, with the latter computing recalls for each image.",
+ "details": [
+ {
+ "comment": "This code calculates recall metric for object detection. It takes in all IoUs (intersection over union), proposal numbers, and thresholds as input. It iterates through images and gt (ground truth) boxes, and then computes the recall scores for each image and stores them in an array. The recall is computed by finding the maximum IOU between ground truth and proposals for each image and storing it in a temporary array, then concatenating these values into the final result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/recall.py\":0-26",
+ "content": "import numpy as np\nimport paddle \ndef _recalls(all_ious, proposal_nums, thrs):\n img_num = all_ious.shape[0]\n total_gt_num = sum([ious.shape[0] for ious in all_ious])\n ious_ = np.zeros((proposal_nums.size, total_gt_num), dtype=np.float32)\n for k, proposal_num in enumerate(proposal_nums):\n tmp_ious = np.zeros(0)\n for i in range(img_num):\n ious = all_ious[i][:, :proposal_num].copy()\n gt_ious = np.zeros(ious.shape[0])\n if ious.size == 0:\n tmp_ious = np.hstack((tmp_ious, gt_ious))\n continue\n for j in range(ious.shape[0]):\n gt_max_overlaps = ious.argmax(axis=1)\n max_ious = ious[np.arange(0, ious.shape[0]), gt_max_overlaps]\n gt_idx = max_ious.argmax()\n gt_ious[j] = max_ious[gt_idx]\n box_idx = gt_max_overlaps[gt_idx]\n ious[gt_idx, :] = -1\n ious[:, box_idx] = -1\n tmp_ious = np.hstack((tmp_ious, gt_ious))\n ious_[k, :] = tmp_ious"
+ },
+ {
+ "comment": "The code defines two functions: `recalls()` and `eval_recalls()`. \n\n`recalls()` calculates the average precision-recall curve by comparing predicted bounding boxes with ground truth ones. It does this by sorting intersection over union (IOU) values, creating a recall matrix based on IOU thresholds, and averaging recall across images.\n\n`eval_recalls()` is a wrapper function that calls `recalls()`. It calculates recalls for each image given ground truths and proposals. It also checks input types and sets default parameters if necessary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/recall.py\":28-61",
+ "content": " ious_ = np.fliplr(np.sort(ious_, axis=1))\n recalls = np.zeros((proposal_nums.size, thrs.size))\n for i, thr in enumerate(thrs):\n recalls[:, i] = (ious_ >= thr).sum(axis=1) / float(total_gt_num)\n return recalls\ndef set_recall_param(proposal_nums, iou_thrs):\n if isinstance(proposal_nums, list):\n proposal_nums_ = np.array(proposal_nums)\n elif isinstance(proposal_nums, int):\n proposal_nums_ = np.array([proposal_nums])\n else:\n proposal_nums_ = proposal_nums\n if iou_thrs is None:\n _iou_thrs = np.array([0.5])\n elif isinstance(iou_thrs, list):\n _iou_thrs = np.array(iou_thrs)\n elif isinstance(iou_thrs, float):\n _iou_thrs = np.array([iou_thrs])\n else:\n _iou_thrs = iou_thrs\n return proposal_nums_, _iou_thrs\ndef eval_recalls(gts, proposals, proposal_nums=None, iou_thrs=None):\n \"\"\"Calculate recalls. \"\"\"\n img_num = len(gts)\n assert img_num == len(proposals)\n proposal_nums, iou_thrs = set_recall_param(proposal_nums, iou_thrs)"
+ },
+ {
+ "comment": "This code calculates recall for object detection. It iterates through images, sorts and selects proposals based on scores, and then calculates IoU (intersection over union) between these proposals and ground truth boxes. If no ground truth is found or it has zero boxes, all_ious is filled with zeros. The function returns recalls for each image number.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/recall.py\":63-83",
+ "content": " all_ious = []\n for i in range(img_num):\n if proposals[i].ndim == 2 and proposals[i].shape[1] == 5:\n scores = proposals[i][:, 4]\n sort_idx = np.argsort(scores)[::-1]\n img_proposal = proposals[i][sort_idx, :]\n else:\n img_proposal = proposals[i]\n prop_num = min(img_proposal.shape[0], proposal_nums[-1])\n if gts[i] is None or gts[i].shape[0] == 0:\n ious = np.zeros((0, img_proposal.shape[0]), dtype=np.float32)\n else:\n ious = bbox_overlaps(\n torch.tensor(gts[i]),\n torch.tensor(img_proposal[:prop_num, :4]))\n ious = ious.data.numpy()\n all_ious.append(ious)\n all_ious = np.array(all_ious)\n recalls = _recalls(all_ious, proposal_nums, iou_thrs)\n return recalls"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2cc8b836-a7ef-4eab-aabe-28f3ee4f2da3.json b/docs/doc/2cc8b836-a7ef-4eab-aabe-28f3ee4f2da3.json
new file mode 100644
index 000000000..6cef317ea
--- /dev/null
+++ b/docs/doc/2cc8b836-a7ef-4eab-aabe-28f3ee4f2da3.json
@@ -0,0 +1,75 @@
+{
+ "summary": "The code utilizes PaddleVideo's image processing pipeline for efficient frame sampling, defines a sampler class for video decoding and data conversion, and calculates sampling positions, offsets, and generates frame indices for video sequences.",
+ "details": [
+ {
+ "comment": "This code is a Python module that imports various libraries and defines an image processing pipeline for PaddleVideo. It checks if SimpleITK is installed, handles pickling, and registers the pipeline using PaddleVideo's registry.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":0-37",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport random\nimport numpy as np\nfrom PIL import Image\ntry:\n import SimpleITK as sitk\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [SimpleITK] package and it's dependencies is required for PP-Care.\"\n )\nimport cv2\nfrom ..registry import PIPELINES\ntry:\n import cPickle as pickle\n from cStringIO import StringIO\nexcept ImportError:\n import pickle\n from io import BytesIO\n@PIPELINES.register()"
+ },
+ {
+ "comment": "The `Sampler` class is used to sample frames based on various parameters such as number of segments, length of each segment, frame interval, valid mode, select left flag and whether to use PIL for reading images. It returns the index of sampled frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":38-65",
+ "content": "class Sampler(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_seg(int): number of segments.\n seg_len(int): number of sampled frames in each segment.\n valid_mode(bool): True or False.\n select_left: Whether to select the frame to the left in the middle when the sampling interval is even in the test mode.\n Returns:\n frames_idx: the index of sampled #frames.\n \"\"\"\n def __init__(self,\n num_seg,\n seg_len,\n frame_interval=None,\n valid_mode=False,\n select_left=False,\n dense_sample=False,\n linspace_sample=False,\n use_pil=True):\n self.num_seg = num_seg\n self.seg_len = seg_len\n self.frame_interval = frame_interval\n self.valid_mode = valid_mode\n self.select_left = select_left\n self.dense_sample = dense_sample\n self.linspace_sample = linspace_sample"
+ },
+ {
+ "comment": "The code defines a class with an attribute 'use_pil' that determines the image format. The '_get' method retrieves frames based on data format (frame, MRI, or video), applies necessary conversions and resizing, and stores them in 'imgs'. It uses different libraries such as Image, sitk, and cv2 for different formats.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":66-95",
+ "content": " self.use_pil = use_pil\n def _get(self, frames_idx, results):\n data_format = results['format']\n if data_format == \"frame\":\n frame_dir = results['frame_dir']\n imgs = []\n for idx in frames_idx:\n img = Image.open(\n os.path.join(frame_dir,\n results['suffix'].format(idx))).convert('RGB')\n imgs.append(img)\n elif data_format == \"MRI\":\n frame_dir = results['frame_dir']\n imgs = []\n MRI = sitk.GetArrayFromImage(sitk.ReadImage(frame_dir))\n for idx in frames_idx:\n item = MRI[idx]\n item = cv2.resize(item, (224, 224))\n imgs.append(item)\n elif data_format == \"video\":\n if results['backend'] == 'cv2':\n frames = np.array(results['frames'])\n imgs = []\n for idx in frames_idx:\n imgbuf = frames[idx]\n img = Image.fromarray(imgbuf, mode='RGB')"
+ },
+ {
+ "comment": "Code is handling video decoding using different backends such as 'opencv', 'decord', and 'pyav'. It appends the frames to imgs list, converts numpy array to image using Image.fromarray method for 'decord' backend, and handles frame indexing and data structures based on backend used.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":96-118",
+ "content": " imgs.append(img)\n elif results['backend'] == 'decord':\n container = results['frames']\n if self.use_pil:\n frames_select = container.get_batch(frames_idx)\n # dearray_to_img\n np_frames = frames_select.asnumpy()\n imgs = []\n for i in range(np_frames.shape[0]):\n imgbuf = np_frames[i]\n imgs.append(Image.fromarray(imgbuf, mode='RGB'))\n else:\n if frames_idx.ndim != 1:\n frames_idx = np.squeeze(frames_idx)\n frame_dict = {\n idx: container[idx].asnumpy()\n for idx in np.unique(frames_idx)\n }\n imgs = [frame_dict[idx] for idx in frames_idx]\n elif results['backend'] == 'pyav':\n imgs = []\n frames = np.array(results['frames'])\n for idx in frames_idx:"
+ },
+ {
+ "comment": "This code snippet is responsible for sampling frames from a video sequence, and it handles different scenarios based on the input parameters. If `dense_sample` is True, it adjusts the index before accessing the frame. The frames are then appended to a list called `imgs`. If neither of the else conditions are met, it raises a `NotImplementedError`. The function also includes another method, `_get_train_clips`, which calculates clip offsets for training purposes based on the number of frames and other parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":119-143",
+ "content": " if self.dense_sample:\n idx = idx - 1\n imgbuf = frames[idx]\n imgs.append(imgbuf)\n imgs = np.stack(imgs) # thwc\n else:\n raise NotImplementedError\n else:\n raise NotImplementedError\n results['imgs'] = imgs\n return results\n def _get_train_clips(self, num_frames):\n ori_seg_len = self.seg_len * self.frame_interval\n avg_interval = (num_frames - ori_seg_len + 1) // self.num_seg\n if avg_interval > 0:\n base_offsets = np.arange(self.num_seg) * avg_interval\n clip_offsets = base_offsets + np.random.randint(avg_interval,\n size=self.num_seg)\n elif num_frames > max(self.num_seg, ori_seg_len):\n clip_offsets = np.sort(\n np.random.randint(num_frames - ori_seg_len + 1,\n size=self.num_seg))\n elif avg_interval == 0:"
+ },
+ {
+ "comment": "The code defines a class with methods to determine clip offsets based on the number of frames and segment length. If the number of frames exceeds the original segment length, it calculates clip offsets for each segment. Otherwise, it sets all clip offsets to zero. The class also has a __call__ method that takes frames length as input and returns sampling indices.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":144-170",
+ "content": " ratio = (num_frames - ori_seg_len + 1.0) / self.num_seg\n clip_offsets = np.around(np.arange(self.num_seg) * ratio)\n else:\n clip_offsets = np.zeros((self.num_seg, ), dtype=np.int)\n return clip_offsets\n def _get_test_clips(self, num_frames):\n ori_seg_len = self.seg_len * self.frame_interval\n avg_interval = (num_frames - ori_seg_len + 1) / float(self.num_seg)\n if num_frames > ori_seg_len - 1:\n base_offsets = np.arange(self.num_seg) * avg_interval\n clip_offsets = (base_offsets + avg_interval / 2.0).astype(np.int)\n else:\n clip_offsets = np.zeros((self.num_seg, ), dtype=np.int)\n return clip_offsets\n def __call__(self, results):\n \"\"\"\n Args:\n frames_len: length of frames.\n return:\n sampling id.\n \"\"\"\n frames_len = int(results['frames_len'])\n frames_idx = []\n if self.frame_interval is not None:\n assert isinstance(self.frame_interval, int)"
+ },
+ {
+ "comment": "This code determines the sampling method for frames based on the mode (valid or train) and format ('video' or 'frame'). It calculates offsets, handles different formats, and if linspace_sample is True, it generates offsets using linear spacing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":171-198",
+ "content": " if not self.valid_mode:\n offsets = self._get_train_clips(frames_len)\n else:\n offsets = self._get_test_clips(frames_len)\n offsets = offsets[:, None] + np.arange(\n self.seg_len)[None, :] * self.frame_interval\n offsets = np.concatenate(offsets)\n offsets = offsets.reshape((-1, self.seg_len))\n offsets = np.mod(offsets, frames_len)\n offsets = np.concatenate(offsets)\n if results['format'] == 'video':\n frames_idx = offsets\n elif results['format'] == 'frame':\n frames_idx = list(offsets + 1)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)\n if self.linspace_sample:\n if 'start_idx' in results and 'end_idx' in results:\n offsets = np.linspace(results['start_idx'], results['end_idx'],\n self.num_seg)\n else:\n offsets = np.linspace(0, frames_len - 1, self.num_seg)"
+ },
+ {
+ "comment": "This code segment calculates the frames to sample from a video, based on its format (video/frame/MRI). It also handles dense sampling for ppTSM. In non-dense mode, it selects random positions for each segment within the range of 1 to frames_len. For dense sampling in train mode, it generates a set of evenly spaced frame indices between start_idx and sample_pos, which is calculated based on frames_len and 64 (to ensure at least one frame within the window). The offsets are then used to fetch corresponding data using the _get method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":199-222",
+ "content": " offsets = np.clip(offsets, 0, frames_len - 1).astype(np.int64)\n if results['format'] == 'video':\n frames_idx = list(offsets)\n frames_idx = [x % frames_len for x in frames_idx]\n elif results['format'] == 'frame':\n frames_idx = list(offsets + 1)\n elif results['format'] == 'MRI':\n frames_idx = list(offsets)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)\n average_dur = int(frames_len / self.num_seg)\n if not self.select_left:\n if self.dense_sample: # For ppTSM\n if not self.valid_mode: # train\n sample_pos = max(1, 1 + frames_len - 64)\n t_stride = 64 // self.num_seg\n start_idx = 0 if sample_pos == 1 else np.random.randint(\n 0, sample_pos - 1)\n offsets = [(idx * t_stride + start_idx) % frames_len + 1\n for idx in range(self.num_seg)]"
+ },
+ {
+ "comment": "The code determines the sampling position based on frames length, number of segments, and valid mode. If no offsets are provided, it calculates the starting positions for each segment using a linear space. Then, it generates the offsets by multiplying the stride with the current segment index and adding the start index. Finally, if in valid mode, it randomly selects indices within the average duration per segment and adds them to the offsets list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":223-244",
+ "content": " frames_idx = offsets\n else:\n sample_pos = max(1, 1 + frames_len - 64)\n t_stride = 64 // self.num_seg\n start_list = np.linspace(0,\n sample_pos - 1,\n num=10,\n dtype=int)\n offsets = []\n for start_idx in start_list.tolist():\n offsets += [\n (idx * t_stride + start_idx) % frames_len + 1\n for idx in range(self.num_seg)\n ]\n frames_idx = offsets\n else:\n for i in range(self.num_seg):\n idx = 0\n if not self.valid_mode:\n if average_dur >= self.seg_len:\n idx = random.randint(0, average_dur - self.seg_len)\n idx += i * average_dur"
+ },
+ {
+ "comment": "Code calculates index based on average duration, then appends corresponding frame indices to frames_idx list based on the format specified in results. If the format is not recognized, it raises NotImplementedError. Finally, it returns the frames_idx and results to an unknown method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":245-267",
+ "content": " elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n else:\n if average_dur >= self.seg_len:\n idx = (average_dur - 1) // 2\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n for jj in range(idx, idx + self.seg_len):\n if results['format'] == 'video':\n frames_idx.append(int(jj % frames_len))\n elif results['format'] == 'frame':\n frames_idx.append(jj + 1)\n elif results['format'] == 'MRI':\n frames_idx.append(jj)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)"
+ },
+ {
+ "comment": "This code generates random offsets for selecting frames from a video. If the valid mode is not enabled, it randomly selects frame offsets within the available duration or number of frames. If the valid mode is enabled, it evenly distributes the frames across the video duration. The 'format' variable determines if the selected frames are in video format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":269-291",
+ "content": " else: # for TSM\n if not self.valid_mode:\n if average_dur > 0:\n offsets = np.multiply(list(range(self.num_seg)),\n average_dur) + np.random.randint(\n average_dur, size=self.num_seg)\n elif frames_len > self.num_seg:\n offsets = np.sort(\n np.random.randint(frames_len, size=self.num_seg))\n else:\n offsets = np.zeros(shape=(self.num_seg, ))\n else:\n if frames_len > self.num_seg:\n average_dur_float = frames_len / self.num_seg\n offsets = np.array([\n int(average_dur_float / 2.0 + average_dur_float * x)\n for x in range(self.num_seg)\n ])\n else:\n offsets = np.zeros(shape=(self.num_seg, ))\n if results['format'] == 'video':\n frames_idx = list(offsets)"
+ },
+ {
+ "comment": "This code snippet defines a SamplerPkl class that samples frames' indices for video loading. It takes arguments num_seg, seg_len, and backend and returns the index of sampled frames. Depending on the results format ('frame', 'MRI', or others), it sets the frames_idx accordingly before returning it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":292-326",
+ "content": " frames_idx = [x % frames_len for x in frames_idx]\n elif results['format'] == 'frame':\n frames_idx = list(offsets + 1)\n elif results['format'] == 'MRI':\n frames_idx = list(offsets)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)\n@PIPELINES.register()\nclass SamplerPkl(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_seg(int): number of segments.\n seg_len(int): number of sampled frames in each segment.\n mode(str): 'train', 'valid'\n Returns:\n frames_idx: the index of sampled #frames.\n \"\"\"\n def __init__(self, num_seg, seg_len, backend='pillow', valid_mode=False):\n self.num_seg = num_seg\n self.seg_len = seg_len\n self.valid_mode = valid_mode\n self.backend = backend\n def _get(self, buf):\n if isinstance(buf, str):\n img = Image.open(StringIO(buf))\n else:"
+ },
+ {
+ "comment": "This code is part of a pipeline for image sampling in video processing. It loads data from disk, converts images to RGB format, and handles labels. The `__call__` method takes results as input, retrieves the video name, label, and frames from the loaded data. If the label is a dictionary or has multiple elements, it assigns the label to '\u52a8\u4f5c\u7c7b\u578b' or randomly chooses between the first two elements. It sets the 'frames_len' based on the length of frames and calculates the average duration per segment. Then, it initializes an empty list for the images and loops through the segments to create image samples. If valid mode is not enabled, it also resets the index variable.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":327-357",
+ "content": " img = Image.open(BytesIO(buf))\n img = img.convert('RGB')\n if self.backend != 'pillow':\n img = np.array(img)\n return img\n def __call__(self, results):\n \"\"\"\n Args:\n frames_len: length of frames.\n return:\n sampling id.\n \"\"\"\n filename = results['frame_dir']\n data_loaded = pickle.load(open(filename, 'rb'), encoding='bytes')\n video_name, label, frames = data_loaded\n if isinstance(label, dict):\n label = label['\u52a8\u4f5c\u7c7b\u578b']\n results['labels'] = label\n elif len(label) == 1:\n results['labels'] = int(label[0])\n else:\n results['labels'] = int(label[0]) if random.random() < 0.5 else int(\n label[1])\n results['frames_len'] = len(frames)\n frames_len = results['frames_len']\n average_dur = int(int(frames_len) / self.num_seg)\n imgs = []\n for i in range(self.num_seg):\n idx = 0\n if not self.valid_mode:"
+ },
+ {
+ "comment": "The code calculates the index for a segment of frames based on average duration and frame length. It then retrieves images from the frames list, appends them to imgs, sets backend type, and returns the results including the imgs and backend information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample.py\":358-381",
+ "content": " if average_dur >= self.seg_len:\n idx = random.randint(0, average_dur - self.seg_len)\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n else:\n if average_dur >= self.seg_len:\n idx = (average_dur - 1) // 2\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i\n for jj in range(idx, idx + self.seg_len):\n imgbuf = frames[int(jj % results['frames_len'])]\n img = self._get(imgbuf)\n imgs.append(img)\n results['backend'] = self.backend\n results['imgs'] = imgs\n return results"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2cedbf37-69dd-428a-baa3-e50f088fa25f.json b/docs/doc/2cedbf37-69dd-428a-baa3-e50f088fa25f.json
new file mode 100644
index 000000000..0176dbbeb
--- /dev/null
+++ b/docs/doc/2cedbf37-69dd-428a-baa3-e50f088fa25f.json
@@ -0,0 +1,25 @@
+{
+ "summary": "This code defines a Sampler class for sampling frame IDs in video data, using PIL to read images instead of OpenCV, and returns the index of sampled frames. It can calculate indices randomly or by formula.",
+ "details": [
+ {
+ "comment": "This code defines a Sampler class that samples frames IDs for video data. It takes arguments: num_seg (number of segments), seg_len (number of sampled frames in each segment), and mode ('train' or 'valid'). The class uses PIL to read images instead of OpenCV (cv2) for better compatibility. The sampler returns the index of sampled frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/sample.py\":0-31",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport random\nfrom PIL import Image\nfrom ..registry import PIPELINES\nimport os\nimport numpy as np\n@PIPELINES.register()\nclass Sampler(object):\n \"\"\"\n Sample frames id.\n NOTE: Use PIL to read image here, has diff with CV2\n Args:\n num_seg(int): number of segments.\n seg_len(int): number of sampled frames in each segment.\n mode(str): 'train', 'valid'\n Returns:\n frames_idx: the index of sampled #frames."
+ },
+ {
+ "comment": "The code defines a class with an initialization function and two methods, \"_get\" and \"__call__\". The \"_get\" method takes frames_idx and results as arguments, and based on the data format (frame or video), it retrieves and appends images to imgs. If the format is not frame or video, it raises a NotImplementedError. The \"__call__\" method takes frames_len as an argument and returns a sampling id.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/sample.py\":32-69",
+ "content": " \"\"\"\n def __init__(self, num_seg, seg_len, valid_mode=False):\n self.num_seg = num_seg\n self.seg_len = seg_len\n self.valid_mode = valid_mode\n def _get(self, frames_idx, results):\n data_format =results['format']\n if data_format == \"frame\":\n frame_dir = results['frame_dir']\n imgs = []\n for idx in frames_idx:\n img = Image.open(os.path.join(frame_dir, results['suffix'].format(idx))).convert('RGB')\n imgs.append(img)\n elif data_format == \"video\":\n frames = np.array(results['frames'])\n imgs = []\n for idx in frames_idx:\n imgbuf = frames[idx]\n img = Image.fromarray(imgbuf, mode='RGB')\n imgs.append(img)\n else:\n raise NotImplementedError\n results['imgs'] = imgs\n return results\n def __call__(self, results):\n \"\"\"\n Args:\n frames_len: length of frames.\n return:\n sampling id."
+ },
+ {
+ "comment": "This code calculates frame indices for video or frame data. It takes in 'frames_len' and 'num_seg' as inputs, and if 'valid_mode' is False, it generates random frame indices within the valid frame range. If 'valid_mode' is True, it calculates frame indices based on specific formulas. The output is stored in 'frames_idx'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/sample.py\":70-95",
+ "content": " \"\"\"\n frames_len = int(results['frames_len'])\n average_dur = int(int(frames_len) / self.num_seg)\n frames_idx = []\n for i in range(self.num_seg):\n idx = 0\n if not self.valid_mode:\n if average_dur >= self.seg_len:\n idx = random.randint(0, average_dur - self.seg_len)\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else: # average_dur = 0\n idx = i % frames_len\n else:\n if average_dur >= self.seg_len:\n idx = (average_dur - 1) // 2\n idx += i * average_dur\n elif average_dur >= 1:\n idx += i * average_dur\n else:\n idx = i % frames_len\n for jj in range(idx, idx+self.seg_len):\n if results['format'] == 'video':\n frames_idx.append(int(jj%frames_len))\n elif results['format'] == 'frame':"
+ },
+ {
+ "comment": "This code snippet is part of a class method that retrieves frames from a video file based on their index. If the frame index (jj+1) is not equal to 0, it appends the index to the frames_idx list; otherwise, it raises a NotImplementedError. The method then returns the results using the _get method with the frames_idx and results as arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/sample.py\":96-101",
+ "content": " #frame from 000001\n frames_idx.append(jj+1)\n else:\n raise NotImplementedError\n return self._get(frames_idx, results)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2cfd042c-4704-4187-8777-a90abf09ff85.json b/docs/doc/2cfd042c-4704-4187-8777-a90abf09ff85.json
new file mode 100644
index 000000000..f983feeae
--- /dev/null
+++ b/docs/doc/2cfd042c-4704-4187-8777-a90abf09ff85.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code defines two functions, get_class2id_map and get_arguments. It reads ground truth text files, splits them by class labels, saves as .npy files, defines boundary frames for new actions, and saves these as separate .npy files. Assumes input files preprocessed and split by lines.",
+ "details": [
+ {
+ "comment": "This code defines two functions: get_class2id_map and get_arguments. The get_class2id_map function takes a dataset name (50salads, gtea, or breakfast) and the path to the dataset directory, and returns a dictionary mapping class names to their respective IDs by reading the \"mapping.txt\" file in the specified dataset directory. The get_arguments function parses all arguments from the command line interface for converting ground truth txt files to numpy arrays.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/prepare_asrf_data.py\":0-41",
+ "content": "import argparse\nimport glob\nimport os\nimport sys\nfrom typing import Dict\nimport numpy as np\nsys.path.append(os.path.join(os.path.dirname(__file__), \"..\"))\ndataset_names = [\"50salads\", \"breakfast\", \"gtea\"]\ndef get_class2id_map(dataset: str,\n dataset_dir: str = \"./dataset\") -> Dict[str, int]:\n \"\"\"\n Args:\n dataset: 50salads, gtea, breakfast\n dataset_dir: the path to the datset directory\n \"\"\"\n assert (dataset in dataset_names\n ), \"You have to choose 50salads, gtea or breakfast as dataset.\"\n with open(os.path.join(dataset_dir, \"{}/mapping.txt\".format(dataset)),\n \"r\") as f:\n actions = f.read().split(\"\\n\")[:-1]\n class2id_map = dict()\n for a in actions:\n class2id_map[a.split()[1]] = int(a.split()[0])\n return class2id_map\ndef get_arguments() -> argparse.Namespace:\n \"\"\"\n parse all the arguments from command line inteface\n return a list of parsed arguments\n \"\"\"\n parser = argparse.ArgumentParser(\n description=\"convert ground truth txt files to numpy array\")"
+ },
+ {
+ "comment": "This code sets up the dataset directory path and creates directories for saving ground truth numpy arrays. It also creates a class to index mapping using get_class2id_map function, and retrieves all groundTruth text files' paths in the specified dataset directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/prepare_asrf_data.py\":42-73",
+ "content": " parser.add_argument(\n \"--dataset_dir\",\n type=str,\n default=\"./dataset\",\n help=\"path to a dataset directory (default: ./dataset)\",\n )\n return parser.parse_args()\ndef main() -> None:\n args = get_arguments()\n datasets = [\"50salads\", \"gtea\", \"breakfast\", \"baseball\"]\n for dataset in datasets:\n # make directory for saving ground truth numpy arrays\n cls_save_dir = os.path.join(args.dataset_dir, dataset, \"gt_arr\")\n if not os.path.exists(cls_save_dir):\n os.mkdir(cls_save_dir)\n # make directory for saving ground truth numpy arrays\n boundary_save_dir = os.path.join(args.dataset_dir, dataset,\n \"gt_boundary_arr\")\n if not os.path.exists(boundary_save_dir):\n os.mkdir(boundary_save_dir)\n # class to index mapping\n class2id_map = get_class2id_map(dataset, dataset_dir=args.dataset_dir)\n gt_dir = os.path.join(args.dataset_dir, dataset, \"groundTruth\")\n gt_paths = glob.glob(os.path.join(gt_dir, \"*.txt\"))"
+ },
+ {
+ "comment": "This code is reading ground truth text files, splitting them into arrays based on class labels, and saving these arrays as .npy files. It also defines boundary frames for new actions and saves these as separate .npy files. The code assumes that the input files are already processed and split by lines.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/prepare_asrf_data.py\":75-105",
+ "content": " for gt_path in gt_paths:\n # the name of ground truth text file\n gt_name = os.path.relpath(gt_path, gt_dir)\n with open(gt_path, \"r\") as f:\n gt = f.read().split(\"\\n\")[:-1]\n gt_array = np.zeros(len(gt))\n for i in range(len(gt)):\n gt_array[i] = class2id_map[gt[i]]\n # save array\n np.save(os.path.join(cls_save_dir, gt_name[:-4] + \".npy\"), gt_array)\n # the name of ground truth text file\n gt_name = os.path.relpath(gt_path, gt_dir)\n with open(gt_path, \"r\") as f:\n gt = f.read().split(\"\\n\")[:-1]\n # define the frame where new action starts as boundary frame\n boundary = np.zeros(len(gt))\n last = gt[0]\n boundary[0] = 1\n for i in range(1, len(gt)):\n if last != gt[i]:\n boundary[i] = 1\n last = gt[i]\n # save array\n np.save(os.path.join(boundary_save_dir, gt_name[:-4] + \".npy\"),"
+ },
+ {
+ "comment": "This code snippet defines a function named \"main\" and checks if the script is being run directly. If it is, the \"main\" function is called to execute the desired task. The code prints \"Done\" after completing the specified operation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/50salads/prepare_asrf_data.py\":106-112",
+ "content": " boundary)\n print(\"Done\")\nif __name__ == \"__main__\":\n main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2d143039-84f2-4840-a4b5-3828135abccf.json b/docs/doc/2d143039-84f2-4840-a4b5-3828135abccf.json
new file mode 100644
index 000000000..d3b0d2afe
--- /dev/null
+++ b/docs/doc/2d143039-84f2-4840-a4b5-3828135abccf.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The Registry class supports module customization, allowing users to register objects and retrieve them via unique names in a name-to-object mapping system.",
+ "details": [
+ {
+ "comment": "The Registry class provides a name-to-object mapping, enabling third-party users to customize modules. To register an object, use @BACKBONES.register() or BACKBONES.register(ResNet).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/registry.py\":0-34",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nclass Registry(object):\n \"\"\"\n The registry that provides name -> object mapping, to support third-party users' custom modules.\n To register an object:\n .. code-block:: python\n BACKBONES = Registry('backbone')\n @BACKBONES.register()\n class ResNet:\n pass\n Or:\n .. code-block:: python\n BACKBONES = Registry('backbone')\n class ResNet:\n pass\n BACKBONES.register(ResNet)"
+ },
+ {
+ "comment": "The code provides a registry class for building modules based on their names. It allows registration of objects with unique names, and can be used as a decorator or without. The usage example demonstrates how to get a backbone module using its name from the registered objects map.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/registry.py\":36-71",
+ "content": " Usage: To build a module.\n .. code-block:: python\n backbone_name = \"ResNet\"\n b = BACKBONES.get(backbone_name)()\n \"\"\"\n def __init__(self, name):\n \"\"\"\n Args:\n name (str): the name of this registry\n \"\"\"\n self._name = name\n self._obj_map = {}\n def __contains__(self, key):\n return self._obj_map.get(key) is not None\n def _do_register(self, name, obj):\n \"\"\"do register\"\"\"\n assert (\n name not in self._obj_map\n ), \"An object named '{}' was already registered in '{}' registry!\".format(\n name, self._name)\n self._obj_map[name] = obj\n def register(self, obj=None, name=None):\n \"\"\"\n Register the given object under the the name `obj.__name__`.\n Can be used as either a decorator or not. See docstring of this class for usage.\n \"\"\"\n if obj is None:\n # used as a decorator\n def deco(func_or_class, name=name):\n if name is None:\n name = func_or_class.__name__"
+ },
+ {
+ "comment": "This code registers and retrieves objects in a registry. It allows registering functions or classes with optional names, and can retrieve the registered object by its name. The `get` function returns the class if found in the registry, otherwise raises KeyError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/registry.py\":72-97",
+ "content": " self._do_register(name, func_or_class)\n return func_or_class\n return deco\n # used as a function call\n if name is None:\n name = obj.__name__\n self._do_register(name, obj)\n def get(self, name):\n \"\"\"Get the registry record.\n Args:\n name (str): The class name.\n Returns:\n ret: The class.\n \"\"\"\n ret = self._obj_map.get(name)\n if ret is None:\n raise KeyError(\n \"No object named '{}' found in '{}' registry!\".format(\n name, self._name))\n return ret"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2dbf4402-58a7-4db6-8e75-1ec98bde3314.json b/docs/doc/2dbf4402-58a7-4db6-8e75-1ec98bde3314.json
new file mode 100644
index 000000000..4340c5043
--- /dev/null
+++ b/docs/doc/2dbf4402-58a7-4db6-8e75-1ec98bde3314.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The Python script uses Baidu Cloud for action detection and includes audio, image processing functions. It has classes like ActionDetection and ModelPredict to initialize models, extract features from video input, retrieve proposals using BMN, classify actions based on extracted features and proposals, and log debugging information. Results are stored in a JSON file.",
+ "details": [
+ {
+ "comment": "This code is a Python script for action detection using Baidu Cloud, which includes functions for processing audio and image data to predict actions. It utilizes various models such as mfcc_extractor, image_model, audio_model, prop_model, and classify_model. The ActionDetection class is defined, which likely contains the main logic of the action detection algorithm. The record_time_info function is a decorator used to log the time taken for executing specific functions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":0-47",
+ "content": "#!./python27-gcc482/bin/python\n# coding: utf-8\n\"\"\"\nBAIDU CLOUD action\n\"\"\"\nimport os\nimport sys\nimport pickle\nimport json\nimport time\nimport functools\nimport numpy as np\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config, print_configs\nimport mfcc.feature_extractor as mfcc_extractor\nimport models.pptsm_infer as image_model\nimport models.audio_infer as audio_model\nimport models.bmn_infer as prop_model\nimport models.lstm_infer as classify_model\nimport logger\nlogger = logger.Logger()\ndef record_time_info(func):\n \"\"\"decorator func to log cost time for func\n \"\"\"\n @functools.wraps(func)\n def timer(*args):\n \"\"\"log cost time for func\n \"\"\"\n logger.info(\"function [{}] processing ...\".format(func.__name__))\n start_time = time.time()\n retval = func(*args)\n cost_time = round(time.time() - start_time, 5)\n logger.info(\"function [{}] run time: {:.2f} min\".format(\n func.__name__, cost_time / 60))\n return retval\n return timer\nclass ActionDetection(object):"
+ },
+ {
+ "comment": "This code defines a ModelPredict class with an initializer that reads configs from a specified file and prints them. It also checks certain conditions related to LSTM_ONLY, sets properties based on those conditions, and loads a model if not in DEBUG mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":48-75",
+ "content": " \"\"\"ModelPredict\"\"\"\n def __init__(self, cfg_file=\"configs/configs.yaml\"):\n cfg = parse_config(cfg_file)\n self.configs = cfg\n print_configs(self.configs, \"Infer\")\n name = 'COMMON'\n self.DEBUG = cfg[name]['DEBUG']\n self.BMN_ONLY = cfg[name]['BMN_ONLY']\n self.LSTM_ONLY = cfg[name]['LSTM_ONLY']\n self.PCM_ONLY = cfg[name]['PCM_ONLY']\n if self.LSTM_ONLY:\n self.prop_dict = {}\n for dataset in ['EuroCup2016']:\n prop_json = '/home/work/datasets/{}/feature_bmn/prop.json'.format(\n dataset)\n json_data = json.load(open(prop_json, 'r'))\n for item in json_data:\n basename = prop_json.replace('feature_bmn/prop.json', 'mp4')\n basename = basename + '/' + item['video_name'] + '.mp4'\n self.prop_dict[basename] = item['bmn_results']\n @record_time_info\n def load_model(self):\n \"\"\"\n load_model\n \"\"\"\n if not self.DEBUG:"
+ },
+ {
+ "comment": "The code initializes different models for image, audio, and proposal extraction, and a classifier. It then extracts features from the input video, retrieves proposals using BMN (Bidirectional Motion Model), and finally classifies the action based on these extracted features and proposals.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":76-107",
+ "content": " self.image_model = image_model.InferModel(self.configs)\n if not self.PCM_ONLY:\n self.audio_model = audio_model.InferModel(self.configs)\n if not self.LSTM_ONLY:\n self.prop_model = prop_model.InferModel(self.configs)\n if not self.BMN_ONLY:\n self.classify_model = classify_model.InferModel(self.configs)\n logger.info(\"==> Action Detection prepared.\")\n @record_time_info\n def infer(self, imgs_path, pcm_path, fps=5):\n \"\"\"\n extract_feature\n \"\"\"\n self.imgs_path = imgs_path\n self.pcm_path = pcm_path\n self.configs['COMMON']['fps'] = fps\n logger.info(\"==> input video {}\".format(os.path.basename(\n self.imgs_path)))\n # step 1: extract feature\n video_features = self.extract_feature()\n # step2: get proposal\n bmn_results = self.extract_proposal(video_features)\n # step3: classify\n material = {'feature': video_features, 'proposal': bmn_results}"
+ },
+ {
+ "comment": "This code defines several methods for video classification and feature extraction. It uses a model called \"classify_model\" to predict actions based on input material, and another model called \"prop_model\" to extract proposals. The BMN_ONLY and LSTM_ONLY flags determine if certain models are used or not. The code also includes logging for debugging purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":108-135",
+ "content": " action_results = self.video_classify(material)\n return bmn_results, action_results\n @record_time_info\n def video_classify(self, material):\n \"\"\"video classify\"\"\"\n if self.BMN_ONLY:\n return []\n action_results = self.classify_model.predict(self.configs,\n material=material)\n logger.info('action shape {}'.format(np.array(action_results).shape))\n return action_results\n @record_time_info\n def extract_proposal(self, video_features):\n \"\"\"extract proposal\"\"\"\n if self.LSTM_ONLY:\n basename = self.imgs_path.replace('frames', 'mp4') + '.mp4'\n bmn_results = self.prop_dict[basename]\n return bmn_results\n bmn_results = self.prop_model.predict(self.configs,\n material=video_features)\n logger.info('proposal shape {}'.format(np.array(bmn_results).shape))\n return bmn_results\n @record_time_info\n def extract_feature(self):"
+ },
+ {
+ "comment": "Extracts features from images and audio in a video file for further processing. If PCM_ONLY is True, extracts only MFCC features from audio using mfcc_extractor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":136-157",
+ "content": " \"\"\"extract feature\"\"\"\n if not self.DEBUG:\n image_path_list = get_images(self.imgs_path)\n self.configs['PPTSM']['frame_list'] = image_path_list\n self.configs['AUDIO']['pcm_file'] = self.pcm_path\n image_features = self.image_model.predict(self.configs)\n if self.PCM_ONLY:\n sample_rate = self.configs['AUDIO']['sample_rate']\n pcm_features = mfcc_extractor.extract_pcm(\n self.pcm_path, sample_rate)\n audio_features = []\n else:\n audio_features, pcm_features = self.audio_model.predict(\n self.configs)\n np_image_features = np.array(image_features, dtype=np.float32)\n np_audio_features = np.array(audio_features, dtype=np.float32)\n np_pcm_features = np.array(pcm_features, dtype=np.float32)\n video_features = {\n 'image_feature': np_image_features,\n 'audio_feature': np_audio_features,"
+ },
+ {
+ "comment": "The code loads video features from frames or pcm file and returns the features in the form of a dictionary. It then proceeds to initialize an instance of the ActionDetection class, load the model, define image and audio paths, and finally calls the infer function to generate bmn_results and action_results which are stored in the results dictionary and saved into a json file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":158-184",
+ "content": " 'pcm_feature': np_pcm_features\n }\n else:\n feature_path = self.imgs_path.replace(\"frames\", \"features\") + '.pkl'\n video_features = pickle.load(open(feature_path, 'rb'))\n logger.info(\"feature shape {} {} {}\".format(\n video_features['image_feature'].shape,\n video_features['audio_feature'].shape,\n video_features['pcm_feature'].shape))\n return video_features\nif __name__ == '__main__':\n model_predict = ActionDetection(cfg_file=\"../configs/configs.yaml\")\n model_predict.load_model()\n imgs_path = \"/home/work/datasets/EuroCup2016/frames/1be705a8f67648da8ec4b4296fa80895\"\n pcm_path = \"/home/work/datasets/EuroCup2016/pcm/1be705a8f67648da8ec4b4296fa80895.pcm\"\n bmn_results, action_results = model_predict.infer(imgs_path, pcm_path)\n results = {'bmn_results': bmn_results, 'action_results': action_results}\n with open('results.json', 'w', encoding='utf-8') as f:\n data = json.dumps(results, indent=4, ensure_ascii=False)"
+ },
+ {
+ "comment": "Writes data to file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/action.py\":185-185",
+ "content": " f.write(data)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2debd80a-ec06-4938-a0db-556a83147597.json b/docs/doc/2debd80a-ec06-4938-a0db-556a83147597.json
new file mode 100644
index 000000000..fed3e10f6
--- /dev/null
+++ b/docs/doc/2debd80a-ec06-4938-a0db-556a83147597.json
@@ -0,0 +1,105 @@
+{
+ "summary": "The code imports modules, defines a Manet class for video segmentation using PaddleVideo's Manet_Stage1 model, and implements training, inference, mask generation, parallel processing, and frame saving steps. It is for deep learning models, visualizations, and measuring time efficiency.",
+ "details": [
+ {
+ "comment": "This code is importing necessary modules and functions from different locations, including image processing utilities and machine learning libraries. It also defines some specific functions related to the MANET model in PaddlePaddle. The code is part of a larger framework for video modeling and image segmentation tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":0-25",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom EIVideo.paddlevideo.loader.builder import build_pipeline\nfrom EIVideo.paddlevideo.loader.pipelines import ToTensor_manet\nimport os\nimport timeit\nimport paddle\nfrom PIL import Image\nfrom davisinteractive.utils.scribbles import scribbles2mask, annotated_frames\nfrom paddle import nn\nfrom EIVideo.paddlevideo.utils import load\nfrom EIVideo.paddlevideo.utils.manet_utils import float_, _palette, damage_masks, long_, write_dict, rough_ROI"
+ },
+ {
+ "comment": "The code is defining a class \"Manet\" that inherits from the BaseSegment class for video segmentation. It has train_step, val_step, infer_step, and test_step methods which are defined but not implemented. The class checks if the model configuration is Manet and then builds the model using build_model function before calling the test_step method with additional parameters like weights and parallel set to False.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":26-60",
+ "content": "from EIVideo.api import load_video, get_scribbles, submit_masks\nfrom ...builder import build_model\nfrom ...registry import SEGMENT\nfrom .base import BaseSegment\n# if cfg.MODEL.framework == \"Manet\":\n# cfg_helper = {\"knns\": 1,\n# \"is_save_image\": True}\n# cfg.update(cfg_helper)\n# build_model(cfg['MODEL']).test_step(**cfg,\n# weights=weights,\n# parallel=False)\n# return\n@SEGMENT.register()\nclass Manet(BaseSegment):\n def __init__(self, backbone=None, head=None, **cfg):\n super().__init__(backbone, head, **cfg)\n def train_step(self, data_batch, step, **cfg):\n pass\n def val_step(self, data_batch, **kwargs):\n pass\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n pass\n def test_step(self, weights, parallel=True, is_save_image=True, **cfg):\n # 1. Construct model.\n cfg['MODEL'].head.pretrained = ''"
+ },
+ {
+ "comment": "This code initializes the model with test mode enabled, builds it using a function, potentially makes it parallel, loads a video for data, prints \"stage1 load_video success\" message, creates a report save directory if it doesn't exist, sets the maximum number of interactions to 8, and evaluates the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":61-86",
+ "content": " cfg['MODEL'].head.test_mode = True\n model = build_model(cfg['MODEL'])\n if parallel:\n model = paddle.DataParallel(model)\n # 2. Construct data.\n sequence = cfg[\"video_path\"].split('/')[-1].split('.')[0]\n obj_nums = 1\n images, _ = load_video(cfg[\"video_path\"], 480)\n print(\"stage1 load_video success\")\n # [195, 389, 238, 47, 244, 374, 175, 399]\n # .shape: (502, 480, 600, 3)\n report_save_dir = cfg.get(\"output_dir\",\n f\"./output/{cfg['model_name']}\")\n if not os.path.exists(report_save_dir):\n os.makedirs(report_save_dir)\n # Configuration used in the challenges\n max_nb_interactions = 8 # Maximum number of interactions\n # Interactive parameters\n model.eval()\n state_dicts_ = load(weights)['state_dict']\n state_dicts = {}\n for k, v in state_dicts_.items():\n if 'num_batches_tracked' not in k:\n state_dicts['head.' + k] = v"
+ },
+ {
+ "comment": "This code segment checks if certain keys are present in the model's state dictionary. If not, it prints a message and writes the state dictionaries to a file named 'model_for_infer.txt'. It then sets the model's state dict with the state dictionaries, opens an inter_file.txt for writing, and initializes a variable 'seen_seq' as False. Inside a no_grad context, it retrieves scribbles and iterates over them, calculating total time, image shape, and checks if there are any annotated frames. If not, it assigns the previous label storage as final masks and submits those masks to the specified save path with corresponding images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":87-108",
+ "content": " if ('head.' + k) not in model.state_dict().keys():\n print(f'pretrained -----{k} -------is not in model')\n write_dict(state_dicts, 'model_for_infer.txt', **cfg)\n model.set_state_dict(state_dicts)\n inter_file = open(\n os.path.join(\n cfg.get(\"output_dir\", f\"./output/{cfg['model_name']}\"),\n 'inter_file.txt'), 'w')\n seen_seq = False\n with paddle.no_grad():\n # Get the current iteration scribbles\n for scribbles, first_scribble in get_scribbles():\n t_total = timeit.default_timer()\n f, h, w = images.shape[:3]\n if 'prev_label_storage' not in locals().keys():\n prev_label_storage = paddle.zeros([f, h, w])\n if len(annotated_frames(scribbles)) == 0:\n final_masks = prev_label_storage\n # ToDo To AP-kai: save_path\u4f20\u8fc7\u6765\u4e86\n submit_masks(cfg[\"save_path\"], final_masks.numpy(), images)"
+ },
+ {
+ "comment": "The code handles the first round of scribbles and initializes memory for future interactions. It writes information to an inter_file, extracts pixel embeddings if it's the first round, and sets up variables for tracking interactions and embedding memories.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":109-133",
+ "content": " continue\n # if no scribbles return, keep masks in previous round\n start_annotated_frame = annotated_frames(scribbles)[0]\n pred_masks = []\n pred_masks_reverse = []\n if first_scribble: # If in the first round, initialize memories\n n_interaction = 1\n eval_global_map_tmp_dic = {}\n local_map_dics = ({}, {})\n total_frame_num = f\n else:\n n_interaction += 1\n inter_file.write(sequence + ' ' + 'interaction' +\n str(n_interaction) + ' ' + 'frame' +\n str(start_annotated_frame) + '\\n')\n if first_scribble: # if in the first round, extract pixel embbedings.\n if not seen_seq:\n seen_seq = True\n inter_turn = 1\n embedding_memory = []\n places = paddle.set_device('cpu')"
+ },
+ {
+ "comment": "This code is iterating through each image in a batch and applying a pipeline transformation if testing mode is enabled. It then creates frame embeddings either by looping over model children or directly from the model head. The frame embeddings are appended to a list, concatenated, and stored as embedding_memory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":135-156",
+ "content": " for imgs in images:\n if cfg['PIPELINE'].get('test'):\n imgs = paddle.to_tensor([\n build_pipeline(cfg['PIPELINE'].test)({\n 'img1':\n imgs\n })['img1']\n ])\n else:\n imgs = paddle.to_tensor([imgs])\n if parallel:\n for c in model.children():\n frame_embedding = c.head.extract_feature(\n imgs)\n else:\n frame_embedding = model.head.extract_feature(\n imgs)\n embedding_memory.append(frame_embedding)\n del frame_embedding\n embedding_memory = paddle.concat(embedding_memory, 0)"
+ },
+ {
+ "comment": "The code initializes the reference frame embedding and handles cases where the annotation is present or not. It extracts the reference frame embedding from the embedding memory, reshapes it, and then creates a scribble sample with the scribble label for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":157-175",
+ "content": " _, _, emb_h, emb_w = embedding_memory.shape\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n else:\n inter_turn += 1\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n else:\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n ########\n scribble_masks = scribbles2mask(scribbles, (emb_h, emb_w))\n scribble_label = scribble_masks[start_annotated_frame]\n scribble_sample = {'scribble_label': scribble_label}\n scribble_sample = ToTensor_manet()(scribble_sample)"
+ },
+ {
+ "comment": "This code snippet is responsible for saving an interactive scribble image. It first retrieves the scribble label, then constructs the file path to save the image based on configuration settings and iteration parameters. If the directory doesn't exist, it creates one. Finally, it saves the scribble image using a specific palette.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":176-194",
+ "content": " # print(ref_frame_embedding, ref_frame_embedding.shape)\n scribble_label = scribble_sample['scribble_label']\n scribble_label = scribble_label.unsqueeze(0)\n model_name = cfg['model_name']\n output_dir = cfg.get(\"output_dir\", f\"./output/{model_name}\")\n inter_file_path = os.path.join(\n output_dir, sequence, 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn))\n if is_save_image:\n ref_scribble_to_show = scribble_label.squeeze().numpy()\n im_ = Image.fromarray(\n ref_scribble_to_show.astype('uint8')).convert('P', )\n im_.putpalette(_palette)\n ref_img_name = str(start_annotated_frame)\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im_.save("
+ },
+ {
+ "comment": "This code segment is part of a video modeling framework. It deals with handling scribbles and generating masks based on them. If there are no scribbles after the first one, it prints a message and continues execution by submitting the previous label storage as final masks. This code also checks for parallel processing and seems to be part of an interaction segmentation head.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":195-215",
+ "content": " os.path.join(inter_file_path,\n 'inter_' + ref_img_name + '.png'))\n if first_scribble:\n prev_label = None\n prev_label_storage = paddle.zeros([f, h, w])\n else:\n prev_label = prev_label_storage[start_annotated_frame]\n prev_label = prev_label.unsqueeze(0).unsqueeze(0)\n # check if no scribbles.\n if not first_scribble and paddle.unique(\n scribble_label).shape[0] == 1:\n print(\n 'not first_scribble and paddle.unique(scribble_label).shape[0] == 1'\n )\n print(paddle.unique(scribble_label))\n final_masks = prev_label_storage\n submit_masks(cfg[\"save_path\"], final_masks.numpy(), images)\n continue\n ###inteaction segmentation head\n if parallel:"
+ },
+ {
+ "comment": "This code is part of the Manet_Stage1 segmentation model in PaddleVideo. It iterates through the children of the model and calls the 'int_seghead' function to generate temporary dictionaries and local map dictionaries for each child. The 'int_seghead' function takes various parameters such as reference frame embedding, previous round label, global map temporary dictionary, etc., and returns a tuple containing the temporary dictionary and local map dictionaries. If there are no children in the model, it directly calls the 'int_seghead' function on the model's head for the same set of parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":216-233",
+ "content": " for c in model.children():\n tmp_dic, local_map_dics = c.head.int_seghead(\n ref_frame_embedding=ref_frame_embedding,\n ref_scribble_label=scribble_label,\n prev_round_label=prev_label,\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n frame_num=[start_annotated_frame],\n first_inter=first_scribble)\n else:\n tmp_dic, local_map_dics = model.head.int_seghead(\n ref_frame_embedding=ref_frame_embedding,\n ref_scribble_label=scribble_label,\n prev_round_label=prev_label,\n global_map_tmp_dic=eval_global_map_tmp_dic,"
+ },
+ {
+ "comment": "Creates a temporary dictionary with local maps and other parameters. Obtains the predicted label for the sequence, interpolates it to original size, gets the argument of maximum value along axis 1, adds it to prediction masks list, stores the first predicted label for current frame in prev_label_storage if saving images, converts pred_label to numpy array and displays unique elements.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":234-253",
+ "content": " local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n frame_num=[start_annotated_frame],\n first_inter=first_scribble)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks.append(float_(pred_label))\n # np.unique(pred_label)\n # array([0], dtype=int64)\n prev_label_storage[start_annotated_frame] = float_(\n pred_label[0])\n if is_save_image: # save image\n pred_label_to_save = pred_label.squeeze(0).numpy()"
+ },
+ {
+ "comment": "The code segment is generating annotated images from predicted labels and creating scribble-based reference labels. It saves the images in a specified folder path, and initializes variables for iterating through the frames of the video.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":254-273",
+ "content": " im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)\n imgname = str(start_annotated_frame)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im.save(os.path.join(inter_file_path, imgname + '.png'))\n #######################################\n if first_scribble:\n scribble_label = rough_ROI(scribble_label)\n ##############################\n ref_prev_label = pred_label.unsqueeze(0)\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = ref_frame_embedding\n for ii in range(start_annotated_frame + 1, total_frame_num):\n current_embedding = embedding_memory[ii]\n current_embedding = current_embedding.unsqueeze(0)"
+ },
+ {
+ "comment": "The code iterates over the model's children and calls `prop_seghead` on each child, passing relevant embeddings and labels to calculate local maps and global maps for segmentation. It also takes into account nearest neighbors, interaction numbers, and annotated frame start.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":274-291",
+ "content": " prev_label = prev_label\n if parallel:\n for c in model.children():\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = c.head.prop_seghead(\n ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,"
+ },
+ {
+ "comment": "Code segment is part of a larger function in PaddleVideo library. It checks if frame number is the start_annotated_frame, if so, it extracts the current embedding, else it calls head.prop_seghead to get temporary dictionary, global map temporary dictionary and local maps based on reference frame embedding, previous embedding, current embedding, scribble label and previous label using Paddle (a deep learning framework). It also considers K nearest neighbors and interaction number while performing its operation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":292-309",
+ "content": " frame_num=[ii],\n dynamic_seghead=c.head.dynamic_seghead)\n else:\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = model.head.prop_seghead(\n ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,"
+ },
+ {
+ "comment": "This code segment is responsible for predicting the labels, creating masks and storing them in a list, and possibly saving an image. The predicted label is interpolated to match the frame size, converted to mask and added to the list of masks. This process continues for each frame. If saving images, the predicted labels are converted to an image format and saved as a grayscale PALETTE image.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":310-326",
+ "content": " frame_num=[ii],\n dynamic_seghead=model.head.dynamic_seghead)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks.append(float_(pred_label))\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = current_embedding\n prev_label_storage[ii] = float_(pred_label[0])\n if is_save_image:\n pred_label_to_save = pred_label.squeeze(0).numpy()\n im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)"
+ },
+ {
+ "comment": "Code snippet saves frames to disk, initializes variables for propagation loop, and begins the propagation process by iterating through frames from start_annotated_frame down to 0. The model's children are then processed in parallel for segmentation head propagation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":327-346",
+ "content": " imgname = str(ii)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im.save(os.path.join(inter_file_path,\n imgname + '.png'))\n #######################################\n prev_label = ref_prev_label\n prev_embedding = ref_frame_embedding\n #######\n # Propagation <-\n for ii in range(start_annotated_frame):\n current_frame_num = start_annotated_frame - 1 - ii\n current_embedding = embedding_memory[current_frame_num]\n current_embedding = current_embedding.unsqueeze(0)\n prev_label = prev_label\n if parallel:\n for c in model.children():\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = c.head.prop_seghead("
+ },
+ {
+ "comment": "This code appears to be part of a deep learning model for video segmentation. It is calling the \"prop_seghead\" function from the \"model.head\" object with specific parameters including reference frame embedding, previous and current embeddings, scribble label, and previous label. If certain conditions are met, additional parameters such as normalize nearest neighbor distances, use local map, sequence names, ground truth IDs, number of nearest neighbors, start annotated frame, and dynamic seghead are passed. The function returns a temporary dictionary, evaluation global map temporary dictionary, and local map dictionaries.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":347-364",
+ "content": " ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,\n frame_num=[current_frame_num],\n dynamic_seghead=c.head.dynamic_seghead)\n else:\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = model.head.prop_seghead("
+ },
+ {
+ "comment": "This code is calculating the predictions for a specific sequence by using various embeddings, labels, and configurations. It involves interacting with multiple dictionaries, tensor operations, and a dynamic seghead model. The predicted label is then interpolated to match the resolution of the original frame.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":365-382",
+ "content": " ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg['knns'],\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,\n frame_num=[current_frame_num],\n dynamic_seghead=model.head.dynamic_seghead)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,"
+ },
+ {
+ "comment": "This code snippet is part of an image segmentation model. It extracts predictions from the model, converts them to masks, and stores previous label information for each frame. Additionally, it saves visualizations of these predictions as palette-colored images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":383-401",
+ "content": " size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks_reverse.append(float_(pred_label))\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = current_embedding\n ####\n prev_label_storage[current_frame_num] = float_(\n pred_label[0])\n ###\n if is_save_image:\n pred_label_to_save = pred_label.squeeze(0).numpy()\n im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)\n imgname = str(current_frame_num)\n while len(imgname) < 5:"
+ },
+ {
+ "comment": "This code saves images and their corresponding masks, creates final masks, and writes the total time for a single interaction. It handles non-existent folders by creating them before saving images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/manet_stage1.py\":402-416",
+ "content": " imgname = '0' + imgname\n if not os.path.exists(inter_file_path):\n os.makedirs(inter_file_path)\n im.save(os.path.join(inter_file_path,\n imgname + '.png'))\n pred_masks_reverse.reverse()\n pred_masks_reverse.extend(pred_masks)\n final_masks = paddle.concat(pred_masks_reverse, 0)\n submit_masks(cfg[\"save_path\"], final_masks.numpy(), images)\n t_end = timeit.default_timer()\n print('Total time for single interaction: ' +\n str(t_end - t_total))\n inter_file.close()\n return None"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2e744e01-920a-4c0b-8699-31b8562984ee.json b/docs/doc/2e744e01-920a-4c0b-8699-31b8562984ee.json
new file mode 100644
index 000000000..d63e461b2
--- /dev/null
+++ b/docs/doc/2e744e01-920a-4c0b-8699-31b8562984ee.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code file contains the initialization for a Video Quality Assessment application. It includes registrar for metrics, builder function for metrics and defines the QualityMetric class. The code is licensed under Apache License, Version 2.0 and distributed as-is without warranties or conditions.",
+ "details": [
+ {
+ "comment": "This code file contains the initialization for a Video Quality Assessment application. It includes registrar for metrics, builder function for metrics and defines the QualityMetric class. The code is licensed under Apache License, Version 2.0 and distributed as-is without warranties or conditions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/__init__.py\":0-22",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .registry import METRIC\nfrom .build import build_metric\nfrom .quality_metric import QuqlityMetric\n__all__ = [\n 'METRIC', 'build_metric', 'QuqlityMetric'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2ef6c407-6499-4b33-b88c-692844b93c1e.json b/docs/doc/2ef6c407-6499-4b33-b88c-692844b93c1e.json
new file mode 100644
index 000000000..656a04c7f
--- /dev/null
+++ b/docs/doc/2ef6c407-6499-4b33-b88c-692844b93c1e.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The BaseSegmenter class serves as a foundation for PaddleVideo segmenters, handling training, validation, testing, and inference with a mode parameter. Subclasses must implement train_step, valid_step, test_step, and feature extraction modules.",
+ "details": [
+ {
+ "comment": "The code is defining a BaseSegmenter class, which serves as the base class for all segmenters. It requires subclasses to override train_step, valid_step, and test_step methods. The class also accepts backbone and head modules to extract features and process them respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/segmenters/base.py\":0-29",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseSegmenter(nn.Layer):\n \"\"\"Base class for segementers.\n All segementers should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Classification head to process feature."
+ },
+ {
+ "comment": "This code defines a segmenter base class for PaddleVideo. It initializes the backbone, head, and loss layers based on user input. The `forward` method specifies how the model processes data in either infer or train mode. Initializing weights is optional but can be called if the layer supports it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/segmenters/base.py\":31-62",
+ "content": " \"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__()\n # build backbone\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n # build head\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n # build loss\n if loss is not None:\n self.loss_name = loss.name\n self.loss = builder.build_loss(loss)\n if hasattr(self.loss, 'init_weights'):\n self.loss.init_weights()\n else:\n self.loss = None\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output."
+ },
+ {
+ "comment": "This code defines a base class for segmenters that supports training, validation, testing, and inference steps. The `mode` parameter determines which step to execute, and abstract methods must be implemented by subclasses for each step.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/segmenters/base.py\":63-98",
+ "content": " 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\""
+ },
+ {
+ "comment": "This code block raises a NotImplementedError, indicating that the current implementation of the function or method is not complete and requires further development.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/segmenters/base.py\":99-99",
+ "content": " raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/2fcf21c5-4cd0-4b12-a56c-5e2b57e23fb2.json b/docs/doc/2fcf21c5-4cd0-4b12-a56c-5e2b57e23fb2.json
new file mode 100644
index 000000000..54b5a8217
--- /dev/null
+++ b/docs/doc/2fcf21c5-4cd0-4b12-a56c-5e2b57e23fb2.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The BMNINFReader class in PaddleVideo reads and processes data from the BMN model for football action detection, filtering invalid proposals and handling image/audio data. This code creates a batch reader that pairs video features with names and scales, yielding batches until completion.",
+ "details": [
+ {
+ "comment": "This code defines a class called BMNINFReader which is a data reader for the BMN model. It reads data that has been extracted by prior networks and uses the \"get_sw_prop\" function to filter out invalid proposals. The get_sw_prop function calculates proposal regions based on a given duration, window size, and step size. Proposals with less than one second in the video are filtered out. This data reader is part of the PaddleVideo package for FootballAction application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/reader/bmninf_reader.py\":0-48",
+ "content": "\"\"\"\n# @File : bmninf_reader.py \n# @Author: macaihong\n# @Date : 2019/12/15\n# @Desc :\n\"\"\"\nimport os\nimport random\nimport pickle\nimport json\nimport numpy as np\nimport multiprocessing\nimport numpy as np\nfrom .reader_utils import DataReader\ndef get_sw_prop(duration, window=200, step=10):\n \"\"\"\n get_sw_prop\n \"\"\"\n pr = []\n local_boxes = []\n for k in np.arange(0, duration - window + step, step):\n start_id = k\n end_id = min(duration, k + window)\n if end_id - start_id < window:\n start_id = end_id - window\n local_boxes = (start_id, end_id)\n pr.append(local_boxes)\n def valid_proposal(duration, span):\n \"\"\"\n valid_proposal\n \"\"\"\n # fileter proposals\n # a valid proposal should have at least one second in the video\n real_span = min(duration, span[1]) - span[0]\n return real_span >= 1\n pr = list(filter(lambda x: valid_proposal(duration, x), pr))\n return pr\nclass BMNINFReader(DataReader):\n \"\"\"\n Data reader for BMN model, which was stored as features extracted by prior networks"
+ },
+ {
+ "comment": "This code initializes a class, likely for data reading and processing. It takes parameters such as name, mode, configuration (cfg), and material. It sets attributes like temporal length (tscale) and duration scale (dscale) from the configuration. The code reshapes pcm_feature to fit the needed shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/reader/bmninf_reader.py\":49-72",
+ "content": " dataset cfg: feat_path, feature path,\n tscale, temporal length of BM map,\n dscale, duration scale of BM map,\n anchor_xmin, anchor_xmax, the range of each point in the feature sequence,\n batch_size, batch size of input data,\n num_threads, number of threads of data processing\n \"\"\"\n def __init__(self, name, mode, cfg, material=None):\n self.name = name\n self.mode = mode\n self.tscale = cfg[self.name.upper()]['tscale'] # 200\n self.dscale = cfg[self.name.upper()]['dscale'] # 200\n # self.subset = cfg[self.name.upper()]['subset']\n self.tgap = 1. / self.tscale\n self.step = cfg[self.name.upper()]['window_step']\n self.material = material\n src_feature = self.material\n image_feature = src_feature['image_feature']\n pcm_feature = src_feature['pcm_feature']\n pcm_feature = pcm_feature.reshape((pcm_feature.shape[0] * 5, 640))\n # print(rgb_feature.shape, audio_feature.shape, pcm_feature.shape)"
+ },
+ {
+ "comment": "This code reads image and audio data for video analysis, concatenates them into a feature vector, sets the duration, window size, and batch size. It then retrieves the list of videos to process and creates a match map for analyzing video frames. The code is part of a machine learning model used in football action detection.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/reader/bmninf_reader.py\":73-104",
+ "content": " min_length = min(image_feature.shape[0], pcm_feature.shape[0])\n #if min_length == 0:\n # continue\n image_feature = image_feature[:min_length, :]\n pcm_feature = pcm_feature[:min_length, :]\n self.features = np.concatenate((image_feature, pcm_feature), axis=1)\n self.duration = len(self.features)\n self.window = self.tscale\n self.get_dataset_dict()\n self.get_match_map()\n self.batch_size = cfg[self.name.upper()]['batch_size']\n if (mode == 'test') or (mode == 'infer'):\n self.num_threads = 1 # set num_threads as 1 for test and infer\n def get_dataset_dict(self):\n \"\"\"\n get_dataset_dict\n \"\"\"\n self.video_list = get_sw_prop(self.duration, self.window, self.step)\n def get_match_map(self):\n \"\"\"\n get_match_map\n \"\"\"\n match_map = []\n for idx in range(self.tscale):\n tmp_match_window = []\n xmin = self.tgap * idx\n for jdx in range(1, self.tscale + 1):"
+ },
+ {
+ "comment": "This code is for creating a reader function to handle BMNINF file loading and defining the match_map attribute. It defines the load_file function, create_reader function, and make_infer_reader function. The load_file function loads features from a given video window range, converts them to float32 type, and transposes the data. The create_reader function creates a reader for the CTCN model. The make_infer_reader function defines a reader for inference purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/reader/bmninf_reader.py\":105-137",
+ "content": " xmax = xmin + self.tgap * jdx\n tmp_match_window.append([xmin, xmax])\n match_map.append(tmp_match_window)\n match_map = np.array(match_map)\n match_map = np.transpose(match_map, [1, 0, 2])\n match_map = np.reshape(match_map, [-1, 2])\n self.match_map = match_map\n self.anchor_xmin = [self.tgap * i for i in range(self.tscale)]\n self.anchor_xmax = [self.tgap * i for i in range(1, self.tscale + 1)]\n def load_file(self, video_wind):\n \"\"\"\n load_file\n \"\"\"\n start_feat_id = video_wind[0]\n end_feat_id = video_wind[1]\n video_feat = self.features[video_wind[0]: video_wind[1]]\n video_feat = video_feat.T\n video_feat = video_feat.astype(\"float32\")\n return video_feat\n def create_reader(self):\n \"\"\"\n reader creator for ctcn model\n \"\"\"\n return self.make_infer_reader()\n def make_infer_reader(self):\n \"\"\"\n reader for inference\n \"\"\"\n def reader():"
+ },
+ {
+ "comment": "This code creates a batch reader for video data in a football action detection application. It loads features from videos, pairs them with their corresponding names and scales, and yields batches of this data until the batch size is reached or all videos are processed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/reader/bmninf_reader.py\":138-154",
+ "content": " \"\"\"\n reader\n \"\"\"\n batch_out = []\n # for video_name in self.video_list:\n for video_wind in self.video_list:\n video_idx = self.video_list.index(video_wind)\n video_feat = self.load_file(video_wind)\n batch_out.append((video_feat, video_wind, [self.duration, self.dscale]))\n if len(batch_out) == self.batch_size:\n yield batch_out\n batch_out = []\n if len(batch_out) > 0:\n yield batch_out\n return reader"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/311bc662-f84a-48a4-b3b0-9dd6651566f7.json b/docs/doc/311bc662-f84a-48a4-b3b0-9dd6651566f7.json
new file mode 100644
index 000000000..1d0d1d338
--- /dev/null
+++ b/docs/doc/311bc662-f84a-48a4-b3b0-9dd6651566f7.json
@@ -0,0 +1,65 @@
+{
+ "summary": "The code defines the AttentionLstmHead class for LSTM-based attention mechanism in PaddleVideo, performing feature extraction and softmax normalization for video and audio classification tasks.",
+ "details": [
+ {
+ "comment": "This code defines a class called AttentionLstmHead, which is a type of head used in a neural network. It is part of the PaddleVideo library and inherits from the BaseHead class. The class uses LSTM for attention, has its own parameters (specified by ParamAttr), and utilizes weight initialization. This code also includes license information, documentation on arguments, and registration in the HEADS registry.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":0-31",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn.initializer import Normal\nfrom paddle.regularizer import L2Decay\nimport paddle.nn.functional as F\nfrom ...metrics.youtube8m import eval_util as youtube8m_metrics\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\nfrom .base import BaseHead\n@HEADS.register()\nclass AttentionLstmHead(BaseHead):\n \"\"\"AttentionLstmHead.\n Args: TODO\n \"\"\"\n def __init__(self,"
+ },
+ {
+ "comment": "This code initializes an AttentionLstmHead object with specified parameters. It creates a Linear layer for each feature dimension (rgb, audio) and adds a bi-directional LSTM layer with specified sizes. The AttentionLstmHead will be used to process video frames and audio data in parallel for classification tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":32-52",
+ "content": " num_classes=3862,\n feature_num=2,\n feature_dims=[1024, 128],\n embedding_size=512,\n lstm_size=1024,\n in_channels=2048,\n loss_cfg=dict(name='CrossEntropyLoss')):\n super(AttentionLstmHead, self).__init__(num_classes, in_channels,\n loss_cfg)\n self.num_classes = num_classes\n self.feature_dims = feature_dims\n self.embedding_size = embedding_size\n self.lstm_size = lstm_size\n self.feature_num = len(self.feature_dims)\n for i in range(self.feature_num): # 0:rgb, 1:audio\n fc_feature = paddle.nn.Linear(in_features=self.feature_dims[i],\n out_features=self.embedding_size)\n self.add_sublayer(\"fc_feature{}\".format(i), fc_feature)\n bi_lstm = paddle.nn.LSTM(input_size=self.embedding_size,\n hidden_size=self.lstm_size,"
+ },
+ {
+ "comment": "The code initializes an LSTM layer with bidirectional capability and adds dropout for regularization. It defines a linear layer (att_fc) to map the output of the LSTM layer to 1 feature, applies softmax activation, and then defines two fully connected layers (fc_out1 and fc_out2) for further processing with specific activations and parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":53-73",
+ "content": " direction=\"bidirectional\")\n self.add_sublayer(\"bi_lstm{}\".format(i), bi_lstm)\n drop_rate = 0.5\n self.dropout = paddle.nn.Dropout(drop_rate)\n att_fc = paddle.nn.Linear(in_features=self.lstm_size * 2,\n out_features=1)\n self.add_sublayer(\"att_fc{}\".format(i), att_fc)\n self.softmax = paddle.nn.Softmax()\n self.fc_out1 = paddle.nn.Linear(in_features=self.lstm_size * 4,\n out_features=8192,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.relu = paddle.nn.ReLU()\n self.fc_out2 = paddle.nn.Linear(in_features=8192,\n out_features=4096,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),"
+ },
+ {
+ "comment": "The code defines a class for an attention LSTM head in PaddleVideo. It initializes two linear layers and a sigmoid activation function. The `init_weights` method is currently empty, and the `forward` method takes inputs of different lengths and processes them before storing the results in the `att_outs` list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":74-94",
+ "content": " initializer=Normal()))\n self.fc_logit = paddle.nn.Linear(in_features=4096,\n out_features=self.num_classes,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.sigmoid = paddle.nn.Sigmoid()\n def init_weights(self):\n pass\n def forward(self, inputs):\n # inputs = [(rgb_data, rgb_len, rgb_mask), (audio_data, audio_len, audio_mask)]\n # deal with features with different length\n # 1. padding to same lenght, make a tensor\n # 2. make a mask tensor with the same shpae with 1\n # 3. compute output using mask tensor, s.t. output is nothing todo with padding\n assert (len(inputs) == self.feature_num\n ), \"Input tensor does not contain {} features\".format(\n self.feature_num)\n att_outs = []"
+ },
+ {
+ "comment": "The code performs feature extraction, bi-directional LSTM processing, attention weight calculation, and finally softmax normalization on each input in a list. It uses dropout to prevent overfitting, applies masking for attention calculations, and calculates the denominator using power function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":95-119",
+ "content": " for i in range(len(inputs)):\n # 1. fc\n m = getattr(self, \"fc_feature{}\".format(i))\n output_fc = m(inputs[i][0])\n output_fc = paddle.tanh(output_fc)\n # 2. bi_lstm\n m = getattr(self, \"bi_lstm{}\".format(i))\n lstm_out, _ = m(inputs=output_fc, sequence_length=inputs[i][1])\n lstm_dropout = self.dropout(lstm_out)\n # 3. att_fc\n m = getattr(self, \"att_fc{}\".format(i))\n lstm_weight = m(lstm_dropout)\n # 4. softmax replace start, for it's relevant to sum in time step\n lstm_exp = paddle.exp(lstm_weight)\n lstm_mask = paddle.mean(inputs[i][2], axis=2)\n lstm_mask = paddle.unsqueeze(lstm_mask, axis=2)\n lstm_exp_with_mask = paddle.multiply(x=lstm_exp, y=lstm_mask)\n lstm_sum_with_mask = paddle.sum(lstm_exp_with_mask, axis=1)\n exponent = -1\n lstm_denominator = paddle.pow(lstm_sum_with_mask, exponent)\n lstm_denominator = paddle.unsqueeze(lstm_denominator, axis=2)"
+ },
+ {
+ "comment": "This code performs LSTM-based attention mechanism for a sequence modeling task. It applies softmax, dropout, and mask operations on the LSTM outputs to compute the attention weights. The attention weights are then used to generate an attentive pooling of the sequence, which is passed through fully connected layers and sigmoid activation for the final output. The loss function uses labels with stop_gradient=True for training the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":120-143",
+ "content": " lstm_softmax = paddle.multiply(x=lstm_exp, y=lstm_denominator)\n lstm_weight = lstm_softmax\n # softmax replace end\n lstm_scale = paddle.multiply(x=lstm_dropout, y=lstm_weight)\n # 5. sequence_pool's replace start, for it's relevant to sum in time step\n lstm_scale_with_mask = paddle.multiply(x=lstm_scale, y=lstm_mask)\n fea_lens = inputs[i][1]\n fea_len = int(fea_lens[0])\n lstm_pool = paddle.sum(lstm_scale_with_mask, axis=1)\n # sequence_pool's replace end\n att_outs.append(lstm_pool)\n att_out = paddle.concat(att_outs, axis=1)\n fc_out1 = self.fc_out1(att_out)\n fc_out1_act = self.relu(fc_out1)\n fc_out2 = self.fc_out2(fc_out1_act)\n fc_out2_act = paddle.tanh(fc_out2)\n fc_logit = self.fc_logit(fc_out2_act)\n output = self.sigmoid(fc_logit)\n return fc_logit, output\n def loss(self, lstm_logit, labels, **kwargs):\n labels.stop_gradient = True"
+ },
+ {
+ "comment": "This code defines an ActionAttentionLstmHead class which is a type of BaseHead. It uses LSTM for attention and takes in various arguments like num_classes, feature_num, feature_dims, embedding_size, lstm_size, in_channels, and loss_cfg. The metric function calculates hit_at_one, perr (precision at equal recall rate), and gap values from the LSTM output and labels. The sum_cost function calculates the loss using BCEWithLogitsLoss.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":144-172",
+ "content": " losses = dict()\n bce_logit_loss = paddle.nn.BCEWithLogitsLoss(reduction='sum')\n sum_cost = bce_logit_loss(lstm_logit, labels)\n return sum_cost\n def metric(self, lstm_output, labels):\n pred = lstm_output.numpy()\n label = labels.numpy()\n hit_at_one = youtube8m_metrics.calculate_hit_at_one(pred, label)\n perr = youtube8m_metrics.calculate_precision_at_equal_recall_rate(\n pred, label)\n gap = youtube8m_metrics.calculate_gap(pred, label)\n return hit_at_one, perr, gap\n@HEADS.register()\nclass ActionAttentionLstmHead(BaseHead):\n \"\"\"AttentionLstmHead for FootballAction\n Args: TODO\n \"\"\"\n def __init__(self,\n num_classes=8,\n feature_num=2,\n feature_dims=[2048, 1024],\n embedding_size=512,\n lstm_size=1024,\n in_channels=2048,\n loss_cfg=dict(name='CrossEntropyLoss')):\n super(ActionAttentionLstmHead, self).__init__(num_classes, in_channels,"
+ },
+ {
+ "comment": "This code initializes a LSTM network for feature processing and attention mechanism. It defines bidirectional LSTM layers for each feature dimension (RGB, audio), followed by dropout and fully connected layers. The model has 8192 output features and is used for multimodal fusion in a video understanding task.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":173-194",
+ "content": " loss_cfg)\n self.num_classes = num_classes\n self.feature_dims = feature_dims\n self.embedding_size = embedding_size\n self.lstm_size = lstm_size\n self.feature_num = len(self.feature_dims)\n for i in range(self.feature_num): # 0:rgb, 1:audio\n bi_lstm = paddle.nn.LSTM(input_size=self.feature_dims[i],\n hidden_size=self.feature_dims[i],\n direction=\"bidirectional\")\n self.add_sublayer(\"bi_lstm{}\".format(i), bi_lstm)\n drop_rate = 0.5\n self.dropout = paddle.nn.Dropout(drop_rate)\n att_fc = paddle.nn.Linear(in_features=self.feature_dims[i] * 2,\n out_features=1)\n self.add_sublayer(\"att_fc{}\".format(i), att_fc)\n self.softmax = paddle.nn.Softmax()\n self.fc1 = paddle.nn.Linear(in_features=2 * sum(self.feature_dims),\n out_features=8192,"
+ },
+ {
+ "comment": "This code defines a class for an attention-based LSTM head in PaddleVideo. It includes several fully connected layers, batch normalization, dropout, and two linear layers. The `init_weights` function is not implemented, and the `forward` method takes input data as a tuple of (rgb_data, rgb_len, rgb_mask) and (audio_data, audio_len, audio_mask).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":195-220",
+ "content": " bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.bn1 = paddle.nn.BatchNorm(num_channels=8192)\n self.dropout1 = paddle.nn.Dropout(0.5)\n self.fc2 = paddle.nn.Linear(in_features=8192,\n out_features=4096,\n bias_attr=ParamAttr(\n regularizer=L2Decay(0.0),\n initializer=Normal()))\n self.bn2 = paddle.nn.BatchNorm(num_channels=4096)\n self.dropout2 = paddle.nn.Dropout(0.5)\n self.fc3 = paddle.nn.Linear(\n in_features=4096,\n out_features=self.num_classes,\n )\n self.fc4 = paddle.nn.Linear(\n in_features=4096,\n out_features=1,\n )\n def init_weights(self):\n pass\n def forward(self, inputs):\n # inputs = [(rgb_data, rgb_len, rgb_mask), (audio_data, audio_len, audio_mask)]"
+ },
+ {
+ "comment": "This code handles features with varying lengths. It pads features to the same length, creates a mask tensor, and computes the output using the mask tensor, effectively ignoring padding values. It asserts that the input tensor contains the expected number of features. It iterates over each feature, performs bi-directional LSTM, applies dropout, calculates weighted sum using attention mechanism, applies softmax to the weights, multiplies by a mask, and stores the results in att_outs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":221-243",
+ "content": " # deal with features with different length\n # 1. padding to same lenght, make a tensor\n # 2. make a mask tensor with the same shpae with 1\n # 3. compute output using mask tensor, s.t. output is nothing todo with padding\n assert (len(inputs) == self.feature_num\n ), \"Input tensor does not contain {} features\".format(\n self.feature_num)\n att_outs = []\n for i in range(len(inputs)):\n m = getattr(self, \"bi_lstm{}\".format(i))\n lstm_out, _ = m(inputs=inputs[i][0], sequence_length=inputs[i][1])\n lstm_dropout = self.dropout(lstm_out)\n # 3. att_fc\n m = getattr(self, \"att_fc{}\".format(i))\n lstm_weight = m(lstm_dropout)\n # 4. softmax replace start, for it's relevant to sum in time step\n lstm_exp = paddle.exp(lstm_weight)\n lstm_mask = paddle.mean(inputs[i][2], axis=2)\n lstm_mask = paddle.unsqueeze(lstm_mask, axis=2)\n lstm_exp_with_mask = paddle.multiply(x=lstm_exp, y=lstm_mask)"
+ },
+ {
+ "comment": "This code segment calculates the attention scores using LSTM and applies them to sequence pooling. It then passes the output through multiple layers of neural networks, including fully connected layers, batch normalization, ReLU activation, and dropout. The final result is stored in `att_out` for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":244-266",
+ "content": " lstm_sum_with_mask = paddle.sum(lstm_exp_with_mask, axis=1)\n exponent = -1\n lstm_denominator = paddle.pow(lstm_sum_with_mask, exponent)\n lstm_denominator = paddle.unsqueeze(lstm_denominator, axis=2)\n lstm_softmax = paddle.multiply(x=lstm_exp, y=lstm_denominator)\n lstm_weight = lstm_softmax\n # softmax replace end\n lstm_scale = paddle.multiply(x=lstm_dropout, y=lstm_weight)\n # 5. sequence_pool's replace start, for it's relevant to sum in time step\n lstm_scale_with_mask = paddle.multiply(x=lstm_scale, y=lstm_mask)\n # fea_lens = inputs[i][1]\n # fea_len = int(fea_lens[0])\n lstm_pool = paddle.sum(lstm_scale_with_mask, axis=1)\n # sequence_pool's replace end\n att_outs.append(lstm_pool)\n att_out = paddle.concat(att_outs, axis=1)\n y = self.fc1(att_out)\n y = self.bn1(y)\n y = F.relu(y)\n y = self.dropout1(y)\n y = self.fc2(y)"
+ },
+ {
+ "comment": "The code contains two main components: a LSTM attention head and loss/metric functions. The LSTM attention head computes attention weights for the input sequence, followed by softmax and sigmoid activation functions. The loss function calculates cross-entropy and mean squared error losses, with alpha as a weight parameter. The metric function computes top1 and top5 accuracy.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/attention_lstm_head.py\":267-287",
+ "content": " y = self.bn2(y)\n y = F.relu(y)\n y = self.dropout2(y)\n out1 = self.fc3(y)\n out1 = F.softmax(out1)\n out2 = self.fc4(y)\n out2 = F.sigmoid(out2)\n return out1, out2\n def loss(self, logits, iou, labels, labels_iou, **kwargs):\n alpha = 10\n softmax_loss = F.cross_entropy(logits, labels)\n labels_iou = labels_iou.astype('float32')\n mse_loss = paddle.sum(F.square_error_cost(iou, labels_iou), axis=-1)\n sum_loss = softmax_loss + alpha * mse_loss\n return sum_loss\n def metric(self, scores, labels):\n top1 = paddle.metric.accuracy(input=scores, label=labels, k=1)\n top5 = paddle.metric.accuracy(input=scores, label=labels, k=5)\n return top1, top5"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/31782bb7-e595-45e6-9f2a-de9c0ea82e0e.json b/docs/doc/31782bb7-e595-45e6-9f2a-de9c0ea82e0e.json
new file mode 100644
index 000000000..d33630f61
--- /dev/null
+++ b/docs/doc/31782bb7-e595-45e6-9f2a-de9c0ea82e0e.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code sets environment variables for GPU utilization and then runs the eval_and_save_model.py script in scenario_lib, evaluating a model named AttentionLstmErnie with provided configuration file and saving its parameters and inference models to specified directories. The \"--save_only\" flag is not used, so both evaluation and saving will occur.",
+ "details": [
+ {
+ "comment": "This code sets environment variables for GPU utilization and then runs the eval_and_save_model.py script in scenario_lib, evaluating a model named AttentionLstmErnie with provided configuration file and saving its parameters and inference models to specified directories. The \"--save_only\" flag is not used, so both evaluation and saving will occur.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/eval_and_save_model.sh\":0-12",
+ "content": "# eval sh \nexport CUDA_VISIBLE_DEVICES=0\nexport FLAGS_eager_delete_tensor_gb=0.0\nexport FLAGS_sync_nccl_allreduce=1\nexport FLAGS_fast_eager_deletion_mode=1\nexport FLAGS_fraction_of_gpu_memory_to_use=0.5\nexport FLAGS_reallocate_gpu_memory_in_mb=0\nexport FLAGS_memory_fraction_of_eager_deletion=1\npython scenario_lib/eval_and_save_model.py --model_name=AttentionLstmErnie \\\n--config=./conf/conf.txt \\\n--save_model_param_dir=checkpoints_save \\\n--save_inference_model=inference_models_save \\\n# --save_only"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/31dfc0aa-3043-4679-8ce5-2bf63a00a92f.json b/docs/doc/31dfc0aa-3043-4679-8ce5-2bf63a00a92f.json
new file mode 100644
index 000000000..fede83cd8
--- /dev/null
+++ b/docs/doc/31dfc0aa-3043-4679-8ce5-2bf63a00a92f.json
@@ -0,0 +1,75 @@
+{
+ "summary": "This code handles video sequence object detection, converting results to CSV format and evaluating AVA metrics using error handling, utility functions, and GPU-based processing.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and modules for evaluating AVA (Activity-driven Visual Attention) metrics. It also includes a license notice, time management functions, and error handling measures. The code uses defaultdict from collections and eval_recalls function from the same repository to perform evaluation tasks related to object detection in video sequences. Additionally, it incorporates paddlevideo's get_logger() function for logging, dist library for distributed processing, and numpy for numerical operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":0-30",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport csv\nimport heapq\nimport logging\nimport time\nfrom collections import defaultdict\nfrom .ava_evaluation import object_detection_evaluation as det_eval\nfrom .ava_evaluation import standard_fields\nfrom .recall import eval_recalls\nimport shutil\nimport pickle\nimport time\nimport os\nimport os.path as osp\nfrom paddlevideo.utils import get_logger, get_dist_info\nimport paddle.distributed as dist\nimport sys\nimport numpy as np"
+ },
+ {
+ "comment": "The code defines two functions: \"det2csv\" and \"results2csv\". \"det2csv\" takes in information, dataset length, results, and custom classes (if any), and returns a list of tuples representing the results in CSV format. It loops through each entry, extracts relevant data, converts tensors to numpy arrays if needed, and appends the information to the csv_results list. \"results2csv\" checks if the results are organized by class or not, then calls either \"det2csv\" or performs CSV conversion directly using it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":31-63",
+ "content": "from pathlib import Path\nfrom datetime import datetime\nimport paddle\ndef det2csv(info, dataset_len, results, custom_classes):\n csv_results = []\n for idx in range(dataset_len):\n video_id = info[idx]['video_id']\n timestamp = info[idx]['timestamp']\n result = results[idx]\n for label, _ in enumerate(result):\n for bbox in result[label]:\n if type(bbox) == paddle.Tensor:\n bbox = bbox.numpy()\n bbox_ = tuple(bbox.tolist())\n if custom_classes is not None:\n actual_label = custom_classes[label + 1]\n else:\n actual_label = label + 1\n csv_results.append((\n video_id,\n timestamp,\n ) + bbox_[:4] + (actual_label, ) + bbox_[4:])\n return csv_results\n# results is organized by class\ndef results2csv(info, dataset_len, results, out_file, custom_classes=None):\n if isinstance(results[0], list):\n csv_results = det2csv(info, dataset_len, results, custom_classes)"
+ },
+ {
+ "comment": "This code snippet contains several utility functions used for video analysis. The \"tostr\" function converts a float to a string representation with 3 decimal places, while the \"print_time\" function calculates and prints the time elapsed since a given start point. The \"make_image_key\" function generates a unique identifier for a video ID and timestamp, and \"read_csv\" function loads boxes and class labels from a CSV file in AVA format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":65-96",
+ "content": " # save space for float\n def tostr(item):\n if isinstance(item, float):\n return f'{item:.3f}'\n return str(item)\n with open(out_file, 'w') as f:\n for csv_result in csv_results:\n f.write(','.join(map(lambda x: tostr(x), csv_result)))\n f.write('\\n')\ndef print_time(message, start):\n print('==> %g seconds to %s' % (time.time() - start, message))\ndef make_image_key(video_id, timestamp):\n \"\"\"Returns a unique identifier for a video id & timestamp.\"\"\"\n return f'{video_id},{int(timestamp):04d}'\ndef read_csv(csv_file, class_whitelist=None, capacity=0):\n \"\"\"Loads boxes and class labels from a CSV file in the AVA format.\n CSV file format described at https://research.google.com/ava/download.html.\n Args:\n csv_file: A file object.\n class_whitelist: If provided, boxes corresponding to (integer) class\n labels not in this set are skipped.\n capacity: Maximum number of labeled boxes allowed for each example.\n Default is 0 where there is no limit."
+ },
+ {
+ "comment": "This code reads a CSV file with video frame data, and for each row, it creates dictionaries for boxes, labels, and scores. It checks the class whitelist before adding the data to the respective lists. If scores are not provided in the CSV, they default to 1.0. The time taken for this process is measured at the beginning with start = time.time().",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":98-119",
+ "content": " Returns:\n boxes: A dictionary mapping each unique image key (string) to a list of\n boxes, given as coordinates [y1, x1, y2, x2].\n labels: A dictionary mapping each unique image key (string) to a list\n of integer class lables, matching the corresponding box in `boxes`.\n scores: A dictionary mapping each unique image key (string) to a list\n of score values lables, matching the corresponding label in `labels`.\n If scores are not provided in the csv, then they will default to 1.0.\n \"\"\"\n start = time.time()\n entries = defaultdict(list)\n boxes = defaultdict(list)\n labels = defaultdict(list)\n scores = defaultdict(list)\n reader = csv.reader(csv_file)\n for row in reader:\n assert len(row) in [7, 8], 'Wrong number of columns: ' + row\n image_key = make_image_key(row[0], row[1])\n x1, y1, x2, y2 = [float(n) for n in row[2:6]]\n action_id = int(row[6])\n if class_whitelist and action_id not in class_whitelist:\n continue"
+ },
+ {
+ "comment": "This code reads a CSV file containing object detection results and stores them in three lists: boxes, labels, and scores. The code also handles exclusions by reading a separate CSV file that contains excluded timestamps. The score is determined based on the length of each row in the CSV file and added to the corresponding image key's entry in the entries dictionary if the capacity allows or if the score is higher than the current highest score for that image key. The code then sorts the entries by descending scores and appends them to the boxes, labels, and scores lists for each image key.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":121-146",
+ "content": " score = 1.0\n if len(row) == 8:\n score = float(row[7])\n if capacity < 1 or len(entries[image_key]) < capacity:\n heapq.heappush(entries[image_key],\n (score, action_id, y1, x1, y2, x2))\n elif score > entries[image_key][0][0]:\n heapq.heapreplace(entries[image_key],\n (score, action_id, y1, x1, y2, x2))\n for image_key in entries:\n # Evaluation API assumes boxes with descending scores\n entry = sorted(entries[image_key], key=lambda tup: -tup[0])\n for item in entry:\n score, action_id, y1, x1, y2, x2 = item\n boxes[image_key].append([y1, x1, y2, x2])\n labels[image_key].append(action_id)\n scores[image_key].append(score)\n print_time('read file ' + csv_file.name, start)\n return boxes, labels, scores\ndef read_exclusions(exclusions_file):\n \"\"\"Reads a CSV file of excluded timestamps.\n Args:\n exclusions_file: A file object containing a csv of video-id,timestamp."
+ },
+ {
+ "comment": "Function `read_excluded_images` reads an exclusions file and returns a set of image keys to exclude. The input file is read row by row, and for each row the function checks that there are exactly two columns and adds the image key (combination of column 1 and column 2) to the excluded set.\n\nFunction `read_labelmap` reads a labelmap file without using protocol buffers. It iterates over the file line by line. When it encounters a line starting with 'name:', it extracts the class name, and when it encounters a line starting with 'id:' it extracts the class id. The function then appends a dictionary containing the id and name to the labelmap list and adds the id to the set of valid class ids.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":148-180",
+ "content": " Returns:\n A set of strings containing excluded image keys, e.g.\n \"aaaaaaaaaaa,0904\",\n or an empty set if exclusions file is None.\n \"\"\"\n excluded = set()\n if exclusions_file:\n reader = csv.reader(exclusions_file)\n for row in reader:\n assert len(row) == 2, 'Expected only 2 columns, got: ' + row\n excluded.add(make_image_key(row[0], row[1]))\n return excluded\ndef read_labelmap(labelmap_file):\n \"\"\"Reads a labelmap without the dependency on protocol buffers.\n Args:\n labelmap_file: A file object containing a label map protocol buffer.\n Returns:\n labelmap: The label map in the form used by the\n object_detection_evaluation\n module - a list of {\"id\": integer, \"name\": classname } dicts.\n class_ids: A set containing all of the valid class id integers.\n \"\"\"\n labelmap = []\n class_ids = set()\n name = ''\n class_id = ''\n for line in labelmap_file:\n if line.startswith(' name:'):\n name = line.split('\"')[1]"
+ },
+ {
+ "comment": "This function ava_eval() takes several file paths as input and evaluates the results using mean average precision (mAP). It uses a label map to convert class labels from detections to their corresponding IDs. The code checks for 'id' or 'label_id' in each line of the label file, appends the ID and name to the label map, and adds the ID to a set of class_ids. The function also handles custom classes by excluding any category whose ID is not in the custom_classes list. The gt_boxes, gt_labels, and _ are loaded from the ann_file for evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":181-209",
+ "content": " elif line.startswith(' id:') or line.startswith(' label_id:'):\n class_id = int(line.strip().split(' ')[-1])\n labelmap.append({'id': class_id, 'name': name})\n class_ids.add(class_id)\n return labelmap, class_ids\n# Seems there is at most 100 detections for each image\ndef ava_eval(result_file,\n result_type,\n label_file,\n ann_file,\n exclude_file,\n max_dets=(100, ),\n verbose=True,\n custom_classes=None):\n assert result_type in ['mAP']\n start = time.time()\n categories, class_whitelist = read_labelmap(open(label_file))\n if custom_classes is not None:\n custom_classes = custom_classes[1:]\n assert set(custom_classes).issubset(set(class_whitelist))\n class_whitelist = custom_classes\n categories = [cat for cat in categories if cat['id'] in custom_classes]\n # loading gt, do not need gt score\n gt_boxes, gt_labels, _ = read_csv(open(ann_file), class_whitelist, 0)"
+ },
+ {
+ "comment": "The code reads detection results from a file, excludes certain keys if specified in an exclude file, and measures the time taken to read the results. It then checks if the result type is 'proposal' and creates proposals based on the gt_boxes for each image key present in boxes or adds a fake one if no corresponding proposal exists. Proposals include scores.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":210-239",
+ "content": " if verbose:\n print_time('Reading detection results', start)\n if exclude_file is not None:\n excluded_keys = read_exclusions(open(exclude_file))\n else:\n excluded_keys = list()\n start = time.time()\n boxes, labels, scores = read_csv(open(result_file), class_whitelist, 0)\n if verbose:\n print_time('Reading detection results', start)\n if result_type == 'proposal':\n gts = [\n np.array(gt_boxes[image_key], dtype=float) for image_key in gt_boxes\n ]\n proposals = []\n for image_key in gt_boxes:\n if image_key in boxes:\n proposals.append(\n np.concatenate(\n (np.array(boxes[image_key], dtype=float),\n np.array(scores[image_key], dtype=float)[:, None]),\n axis=1))\n else:\n # if no corresponding proposal, add a fake one\n proposals.append(np.array([0, 0, 1, 1, 1]))\n # Proposals used here are with scores"
+ },
+ {
+ "comment": "This code calculates the Average Recall (AR) and Recall@0.5 (R@0.5) for different detection numbers using the eval_recalls function. It then prints the results and stores them in a dictionary. If the result type is 'mAP', it initializes a PascalDetectionEvaluator, adds ground truth information for each image key, and calculates the mean average precision (mAP).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":240-264",
+ "content": " recalls = eval_recalls(gts, proposals, np.array(max_dets),\n np.arange(0.5, 0.96, 0.05))\n ar = recalls.mean(axis=1)\n ret = {}\n for i, num in enumerate(max_dets):\n print(f'Recall@0.5@{num}\\t={recalls[i, 0]:.4f}')\n print(f'AR@{num}\\t={ar[i]:.4f}')\n ret[f'Recall@0.5@{num}'] = recalls[i, 0]\n ret[f'AR@{num}'] = ar[i]\n return ret\n if result_type == 'mAP':\n pascal_evaluator = det_eval.PascalDetectionEvaluator(categories)\n start = time.time()\n for image_key in gt_boxes:\n if verbose and image_key in excluded_keys:\n logging.info(\n 'Found excluded timestamp in detections: %s.'\n 'It will be ignored.', image_key)\n continue\n pascal_evaluator.add_single_ground_truth_image_info(\n image_key, {\n standard_fields.InputDataFields.groundtruth_boxes:\n np.array(gt_boxes[image_key], dtype=float),"
+ },
+ {
+ "comment": "This code adds single detected image information to a Pascal evaluator. It converts groundtruth labels and boxes into appropriate data structures, handles excluded timestamps, and processes detection boxes and classes for evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":265-285",
+ "content": " standard_fields.InputDataFields.groundtruth_classes:\n np.array(gt_labels[image_key], dtype=int),\n standard_fields.InputDataFields.groundtruth_difficult:\n np.zeros(len(gt_boxes[image_key]), dtype=bool)\n })\n if verbose:\n print_time('Convert groundtruth', start)\n start = time.time()\n for image_key in boxes:\n if verbose and image_key in excluded_keys:\n logging.info(\n 'Found excluded timestamp in detections: %s.'\n 'It will be ignored.', image_key)\n continue\n pascal_evaluator.add_single_detected_image_info(\n image_key, {\n standard_fields.DetectionResultFields.detection_boxes:\n np.array(boxes[image_key], dtype=float),\n standard_fields.DetectionResultFields.detection_classes:\n np.array(labels[image_key], dtype=int),"
+ },
+ {
+ "comment": "Code snippet performs AVA evaluation for detection results, and prints or returns specific metrics. It also includes functions for creating directories, dumping objects to files using pickle library, and has a function for time measurement called print_time.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":286-319",
+ "content": " standard_fields.DetectionResultFields.detection_scores:\n np.array(scores[image_key], dtype=float)\n })\n if verbose:\n print_time('convert detections', start)\n start = time.time()\n metrics = pascal_evaluator.evaluate()\n if verbose:\n print_time('run_evaluator', start)\n for display_name in metrics:\n print(f'{display_name}=\\t{metrics[display_name]}')\n ret = {\n display_name: metrics[display_name]\n for display_name in metrics if 'ByCategory' not in display_name\n }\n return ret\ndef mkdir_or_exist(dir_name, mode=0o777):\n if dir_name == '':\n return\n dir_name = osp.expanduser(dir_name)\n os.makedirs(dir_name, mode=mode, exist_ok=True)\ndef dump_to_fileobj(obj, file, **kwargs):\n kwargs.setdefault('protocol', 2)\n pickle.dump(obj, file, **kwargs)\ndef dump_to_path(obj, filepath, mode='wb'):\n with open(filepath, mode) as f:\n dump_to_fileobj(obj, f)"
+ },
+ {
+ "comment": "This code defines three functions: `load_from_fileobj`, `load_from_path`, and `collect_results_cpu`. The first two are used to load data from files or file paths, respectively. The third function, `collect_results_cpu`, is a CPU-based method for collecting results across multiple GPUs by saving them in a temporary directory ('tmpdir') and having the rank 0 worker collect them. It checks if all parts exist, waits if not, then loads and returns the collected results once they are all available.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":322-356",
+ "content": "def load_from_fileobj(file, **kwargs):\n return pickle.load(file, **kwargs)\ndef load_from_path(filepath, mode='rb'):\n with open(filepath, mode) as f:\n return load_from_fileobj(f)\ndef collect_results_cpu(result_part, size):\n \"\"\"Collect results in cpu mode.\n It saves the results on different gpus to 'tmpdir' and collects\n them by the rank 0 worker.\n \"\"\"\n tmpdir = osp.join('./', 'collect_results_cpu')\n #1. load results of all parts from tmp dir\n mkdir_or_exist(tmpdir)\n rank, world_size = get_dist_info()\n dump_to_path(result_part, osp.join(tmpdir, f'part_{rank}.pkl'))\n dist.barrier()\n if rank != 0:\n return None\n #2. collect all parts\n while 1:\n all_exist = True\n for i in range(world_size):\n part_file = osp.join(tmpdir, f'part_{i}.pkl')\n if not Path(part_file).exists():\n all_exist = False\n if all_exist:\n break\n else:\n time.sleep(60)\n time.sleep(120)\n #3. load results of all parts from tmp dir"
+ },
+ {
+ "comment": "This code is used for evaluating AVA results by splitting the computation across multiple processes, then combining and ordering the partial results before deleting temporary files. It takes in information about the dataset, the evaluation results, custom class labels, and file paths for input and exclusion lists. The code creates a temporary result file, converts the results to a CSV format, performs AVA evaluation on the temporary file, and returns an evaluation result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":357-383",
+ "content": " part_list = []\n for i in range(world_size):\n part_file = osp.join(tmpdir, f'part_{i}.pkl')\n part_list.append(load_from_path(part_file))\n #4. sort the results\n ordered_results = []\n for res in zip(*part_list):\n ordered_results.extend(list(res))\n ordered_results = ordered_results[:\n size] #the dataloader may pad some samples\n #5. remove results of all parts from tmp dir, avoid dump_file fail to tmp dir when dir not exists.\n for i in range(world_size):\n part_file = osp.join(tmpdir, f'part_{i}.pkl')\n os.remove(part_file)\n return ordered_results\ndef ava_evaluate_results(info, dataset_len, results, custom_classes, label_file,\n file_path, exclude_file):\n # need to create a temp result file\n time_now = datetime.now().strftime('%Y%m%d_%H%M%S')\n temp_file = f'AVA_{time_now}_result.csv'\n results2csv(info, dataset_len, results, temp_file)\n ret = {}\n eval_result = ava_eval(\n temp_file,"
+ },
+ {
+ "comment": "This code is computing the mean average precision (mAP) for object detection metrics. It reads from a label file, file path, and excludes certain classes as specified. The results are stored in the 'ret' dictionary before removing a temporary file and returning the final results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_utils.py\":384-393",
+ "content": " 'mAP',\n label_file,\n file_path, #ann_file,\n exclude_file,\n custom_classes=custom_classes)\n ret.update(eval_result)\n os.remove(temp_file)\n return ret"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/326037c1-296b-4b1d-8c56-5d011e6df707.json b/docs/doc/326037c1-296b-4b1d-8c56-5d011e6df707.json
new file mode 100644
index 000000000..b5931ed92
--- /dev/null
+++ b/docs/doc/326037c1-296b-4b1d-8c56-5d011e6df707.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The PaddleVideo library's code contains a utility function, ReadDict, which reads a dictionary file and performs image cropping, input point adjustment, calculates image size, and converts points to standard format. Another function captures frames from a video at specific indices and releases the video object post-capture.",
+ "details": [
+ {
+ "comment": "This code is part of the PaddleVideo library, specifically in the cpp_infer/src/utility.cpp file. It includes necessary headers for utility functions, and defines the ReadDict function within the PaddleVideo namespace. The function reads a dictionary file located at the given path, and stores each line into a vector of strings named m_vec.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":0-32",
+ "content": "// Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n#include \n#include \n#include \n#include \n#include \n#include \n#include \nnamespace PaddleVideo\n{\n std::vector Utility::ReadDict(const std::string &path)\n {\n std::ifstream in(path);\n std::string line;\n std::vector m_vec;\n if (in)\n {\n while (getline(in, line))"
+ },
+ {
+ "comment": "The code reads a label file and returns its contents as a vector. It also retrieves all files in a directory, adding them to a vector if the directory is valid.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":33-66",
+ "content": " {\n m_vec.push_back(line);\n }\n }\n else\n {\n std::cout << \"no such label file: \" << path << \", exit the program...\"\n << std::endl;\n exit(1);\n }\n return m_vec; // Use fstream to read the category list and return with vector\n }\n void Utility::GetAllFiles(const char *dir_name, std::vector &all_inputs)\n {\n if (NULL == dir_name)\n {\n std::cout << \" dir_name is null ! \" << std::endl;\n return;\n }\n struct stat s;\n lstat(dir_name, &s);\n if (!S_ISDIR(s.st_mode))\n {\n std::cout << \"dir_name is not a valid directory !\" << std::endl;\n all_inputs.push_back(dir_name);\n return;\n }\n else\n {\n struct dirent *filename; // return value for readdir()\n DIR *dir; // return value for opendir()\n dir = opendir(dir_name);\n if (NULL == dir)"
+ },
+ {
+ "comment": "The code snippet opens a directory, reads all files except \".\" and \"..\", and adds the file paths to a vector. The GetRotateCropImage function takes an image and a bounding box as input, copies the source image, and stores x and y coordinates of the bounding box in separate arrays.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":67-92",
+ "content": " {\n std::cout << \"Can not open dir \" << dir_name << std::endl;\n return;\n }\n std::cout << \"Successfully opened the dir !\" << std::endl;\n while ((filename = readdir(dir)) != NULL)\n {\n if (strcmp(filename->d_name, \".\") == 0 ||\n strcmp(filename->d_name, \"..\") == 0)\n continue;\n // img_dir + std::string(\"/\") + all_inputs[0];\n all_inputs.push_back(dir_name + std::string(\"/\") +\n std::string(filename->d_name));\n }\n }\n }\n cv::Mat Utility::GetRotateCropImage(const cv::Mat &srcimage, std::vector> box)\n {\n cv::Mat image;\n srcimage.copyTo(image);\n std::vector> points = box;\n int x_collect[4] = {box[0][0], box[1][0], box[2][0], box[3][0]};\n int y_collect[4] = {box[0][1], box[1][1], box[2][1], box[3][1]};\n int left = int(*std::min_element(x_collect, x_collect + 4));"
+ },
+ {
+ "comment": "This code crops an image based on the x and y coordinates of its bounding box, then adjusts the input points accordingly. It calculates the width and height of the cropped image using the Euclidean distance formula, and converts the original input points to a standard format for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":93-117",
+ "content": " int right = int(*std::max_element(x_collect, x_collect + 4));\n int top = int(*std::min_element(y_collect, y_collect + 4));\n int bottom = int(*std::max_element(y_collect, y_collect + 4));\n cv::Mat img_crop;\n image(cv::Rect(left, top, right - left, bottom - top)).copyTo(img_crop);\n for (int i = 0; i < points.size(); i++)\n {\n points[i][0] -= left;\n points[i][1] -= top;\n }\n int img_crop_width = int(sqrt(pow(points[0][0] - points[1][0], 2) +\n pow(points[0][1] - points[1][1], 2)));\n int img_crop_height = int(sqrt(pow(points[0][0] - points[3][0], 2) +\n pow(points[0][1] - points[3][1], 2)));\n cv::Point2f pts_std[4];\n pts_std[0] = cv::Point2f(0., 0.);\n pts_std[1] = cv::Point2f(img_crop_width, 0.);\n pts_std[2] = cv::Point2f(img_crop_width, img_crop_height);\n pts_std[3] = cv::Point2f(0.f, img_crop_height);\n cv::Point2f pointsf[4];"
+ },
+ {
+ "comment": "This code initializes four points using cv::Point2f, gets a perspective transform matrix M using getPerspectiveTransform, warps the image using warpPerspective, checks if the resized image's rows exceed 1.5 times its columns, and if so, transposes and flips the image before returning it; otherwise, returns the resized image directly. This is part of a function that samples frames from a video file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":118-145",
+ "content": " pointsf[0] = cv::Point2f(points[0][0], points[0][1]);\n pointsf[1] = cv::Point2f(points[1][0], points[1][1]);\n pointsf[2] = cv::Point2f(points[2][0], points[2][1]);\n pointsf[3] = cv::Point2f(points[3][0], points[3][1]);\n cv::Mat M = cv::getPerspectiveTransform(pointsf, pts_std);\n cv::Mat dst_img;\n cv::warpPerspective(img_crop, dst_img, M,\n cv::Size(img_crop_width, img_crop_height),\n cv::BORDER_REPLICATE);\n if (float(dst_img.rows) >= float(dst_img.cols) * 1.5)\n {\n cv::Mat srcCopy = cv::Mat(dst_img.rows, dst_img.cols, dst_img.depth());\n cv::transpose(dst_img, srcCopy);\n cv::flip(srcCopy, srcCopy, 0);\n return srcCopy;\n }\n else\n {\n return dst_img;\n }\n }\n std::vector Utility::SampleFramesFromVideo(const std::string &VideoPath, const int &num_seg, const int &seg_len)\n {\n cv::VideoCapture capture(VideoPath); // Create a video object"
+ },
+ {
+ "comment": "This code snippet checks if the video can be opened and exits if it cannot. It then calculates the number of frames in the video, determines the frame indices to sample for each segment based on the length of the segment and average duration between frames, and stores the sampled frames in a vector.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":146-180",
+ "content": " if (!capture.isOpened())\n {\n printf(\"[Error] video cannot be opened, please check the video [%s]\\n\", VideoPath.c_str());\n capture.release();\n exit(1);\n }\n int frames_len = capture.get(cv::CAP_PROP_FRAME_COUNT); // Get the total number of video frames\n int average_dur = int(frames_len / num_seg);\n std::vector frames_idx;\n for (int i = 0; i < num_seg; ++i)\n {\n int idx = 0;\n if (average_dur >= seg_len)\n {\n idx = (average_dur - 1) / 2;\n idx += i * average_dur;\n }\n else if (average_dur >= 1)\n {\n idx += i * average_dur;\n }\n else\n {\n idx = i;\n }\n for (int j = idx; j < idx + seg_len; ++j)\n {\n frames_idx.emplace_back(j % frames_len);\n }\n }\n std::vector sampled_frames;\n cv::Mat frame; // Create an object for storing sampled frames"
+ },
+ {
+ "comment": "This function captures frames from a video at specific indices, stores them in sampled_frames vector, and releases the video object after capture.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/utility.cpp\":181-191",
+ "content": " for (int i = 0; i < num_seg; ++i)\n {\n const int &frame_idx = frames_idx[i];\n capture.set(cv::CAP_PROP_POS_FRAMES, frame_idx); // Set to frame_idx frame\n capture >> frame;\n sampled_frames.push_back(frame);\n }\n capture.release(); // Release the video object\n return sampled_frames;\n }\n} // namespace PaddleVideo"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/326f0f75-c6ba-4259-8a55-bc71c6c6be8b.json b/docs/doc/326f0f75-c6ba-4259-8a55-bc71c6c6be8b.json
new file mode 100644
index 000000000..fc6497b64
--- /dev/null
+++ b/docs/doc/326f0f75-c6ba-4259-8a55-bc71c6c6be8b.json
@@ -0,0 +1,75 @@
+{
+ "summary": "This code provides text to Unicode conversion and printable encoding functions, with tokenization classes for Chinese characters, punctuation splitting, and WordpieceTokenizing, preparing the text for further processing.",
+ "details": [
+ {
+ "comment": "This code block is the first 30 lines of a Python file and includes a comment with license information, a documentation string, and an import section. The function \"convert_to_unicode\" converts text to Unicode, assuming utf-8 input.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":0-31",
+ "content": "# coding=utf-8\n# Copyright 2018 The Google AI Language Team Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Tokenization classes.\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nfrom __future__ import unicode_literals\nfrom __future__ import absolute_import\nfrom io import open\nimport collections\nimport unicodedata\nimport six\ndef convert_to_unicode(text):\n \"\"\"Converts `text` to Unicode (if it's not already), assuming utf-8 input.\"\"\"\n if six.PY3:"
+ },
+ {
+ "comment": "This code is a function named \"printable_text\" that takes a text parameter and returns it encoded in a way suitable for print or tf.logging. It handles both Python 2 and Python 3 by checking the environment using six.PY2 and six.PY3, and converting strings to str format before returning them. The function checks the type of the input text (str or bytes) and decodes it accordingly (from utf-8 encoding \"ignore\"). If the input type is not supported, it raises a ValueError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":32-60",
+ "content": " if isinstance(text, str):\n return text\n elif isinstance(text, bytes):\n return text.decode(\"utf-8\", \"ignore\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n elif six.PY2:\n if isinstance(text, str):\n return text.decode(\"utf-8\", \"ignore\")\n elif isinstance(text, unicode):\n return text\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n else:\n raise ValueError(\"Not running on Python2 or Python 3?\")\ndef printable_text(text):\n \"\"\"Returns text encoded in a way suitable for print or `tf.logging`.\"\"\"\n # These functions want `str` for both Python2 and Python3, but in one case\n # it's a Unicode string and in the other it's a byte string.\n if six.PY3:\n if isinstance(text, str):\n return text\n elif isinstance(text, bytes):\n return text.decode(\"utf-8\", \"ignore\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))"
+ },
+ {
+ "comment": "This code handles loading and converting vocabulary files. It checks the Python version, loads a vocabulary file into an ordered dictionary, and defines functions to convert sequences of tokens or IDs using the vocab.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":61-95",
+ "content": " elif six.PY2:\n if isinstance(text, str):\n return text\n elif isinstance(text, unicode):\n return text.encode(\"utf-8\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n else:\n raise ValueError(\"Not running on Python2 or Python 3?\")\ndef load_vocab(vocab_file):\n \"\"\"Loads a vocabulary file into a dictionary.\"\"\"\n vocab = collections.OrderedDict()\n with open(vocab_file, encoding='utf8') as fin:\n for num, line in enumerate(fin):\n items = convert_to_unicode(line.strip()).split(\"\\t\")\n if len(items) > 2:\n break\n token = items[0]\n index = items[1] if len(items) == 2 else num\n token = token.strip()\n vocab[token] = int(index)\n return vocab\ndef convert_by_vocab(vocab, items):\n \"\"\"Converts a sequence of [tokens|ids] using the vocab.\"\"\"\n output = []\n for item in items:\n output.append(vocab[item])\n return output\ndef convert_tokens_to_ids(vocab, tokens):"
+ },
+ {
+ "comment": "This code defines a FullTokenizer class for end-to-end tokenization. It utilizes two other classes, BasicTokenizer and WordpieceTokenizer, to perform basic whitespace cleaning and splitting on text data. The FullTokenizer initializes with a vocab file, load_vocab function, and an optional flag for case sensitivity. The tokenize method processes the input text by iterating over each token produced from both BasicTokenizer and WordpieceTokenizer, resulting in split tokens for further processing or analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":96-132",
+ "content": " \"\"\"convert_tokens_to_ids\n \"\"\"\n return convert_by_vocab(vocab, tokens)\ndef convert_ids_to_tokens(inv_vocab, ids):\n \"\"\"convert_ids_to_tokens\n \"\"\"\n return convert_by_vocab(inv_vocab, ids)\ndef whitespace_tokenize(text):\n \"\"\"Runs basic whitespace cleaning and splitting on a peice of text.\"\"\"\n text = text.strip()\n if not text:\n return []\n tokens = text.split()\n return tokens\nclass FullTokenizer(object):\n \"\"\"Runs end-to-end tokenziation.\"\"\"\n def __init__(self, vocab_file, do_lower_case=True):\n \"\"\"init\n \"\"\"\n self.vocab = load_vocab(vocab_file)\n self.inv_vocab = {v: k for k, v in self.vocab.items()}\n self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)\n self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)\n def tokenize(self, text):\n \"\"\"tokenize\n \"\"\"\n split_tokens = []\n for token in self.basic_tokenizer.tokenize(text):\n for sub_token in self.wordpiece_tokenizer.tokenize(token):"
+ },
+ {
+ "comment": "The code defines a CharTokenizer class for end-to-end tokenization. It initializes with a vocab_file and do_lower_case parameter. The class has methods to tokenize text, convert tokens to ids, and convert ids to tokens using the vocab file and inverse vocab file. The tokenization process involves lowercasing the input text, splitting it into words, and then tokenizing each word using a WordpieceTokenizer with the same vocab file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":133-167",
+ "content": " split_tokens.append(sub_token)\n return split_tokens\n def convert_tokens_to_ids(self, tokens):\n \"\"\"convert_tokens_to_ids\n \"\"\"\n return convert_by_vocab(self.vocab, tokens)\n def convert_ids_to_tokens(self, ids):\n \"\"\"convert_ids_to_tokens\n \"\"\"\n return convert_by_vocab(self.inv_vocab, ids)\nclass CharTokenizer(object):\n \"\"\"Runs end-to-end tokenziation.\"\"\"\n def __init__(self, vocab_file, do_lower_case=True):\n self.vocab = load_vocab(vocab_file)\n self.inv_vocab = {v: k for k, v in self.vocab.items()}\n self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)\n def tokenize(self, text):\n \"\"\"tokenize\n \"\"\"\n split_tokens = []\n for token in text.lower().split(\" \"):\n for sub_token in self.wordpiece_tokenizer.tokenize(token):\n split_tokens.append(sub_token)\n return split_tokens\n def convert_tokens_to_ids(self, tokens):\n \"\"\"convert_tokens_to_ids\n \"\"\""
+ },
+ {
+ "comment": "This code defines a `BasicTokenizer` class that performs basic text tokenization, including punctuation splitting and lower casing. It also includes methods for converting tokens to IDs and vice versa using vocabularies. The class has an optional `do_lower_case` parameter controlling whether the input should be lowercased or not.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":168-196",
+ "content": " return convert_by_vocab(self.vocab, tokens)\n def convert_ids_to_tokens(self, ids):\n \"\"\"convert_ids_to_tokens\n \"\"\"\n return convert_by_vocab(self.inv_vocab, ids)\nclass BasicTokenizer(object):\n \"\"\"Runs basic tokenization (punctuation splitting, lower casing, etc.).\"\"\"\n def __init__(self, do_lower_case=True):\n \"\"\"Constructs a BasicTokenizer.\n Args:\n do_lower_case: Whether to lower case the input.\n \"\"\"\n self.do_lower_case = do_lower_case\n def tokenize(self, text):\n \"\"\"Tokenizes a piece of text.\"\"\"\n text = convert_to_unicode(text)\n text = self._clean_text(text)\n # This was added on November 1st, 2018 for the multilingual and Chinese\n # models. This is also applied to the English models now, but it doesn't\n # matter since the English models were not trained on any Chinese data\n # and generally don't have any Chinese data in them (there are Chinese\n # characters in the vocabulary because Wikipedia does have some Chinese"
+ },
+ {
+ "comment": "The code segment tokenizes Chinese characters, performs lower casing if needed, strips accents from text, and splits the punctuation on a given piece of text. This process is to prepare the text for further processing in the application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":197-228",
+ "content": " # words in the English Wikipedia.).\n text = self._tokenize_chinese_chars(text)\n orig_tokens = whitespace_tokenize(text)\n split_tokens = []\n for token in orig_tokens:\n if self.do_lower_case:\n token = token.lower()\n token = self._run_strip_accents(token)\n split_tokens.extend(self._run_split_on_punc(token))\n output_tokens = whitespace_tokenize(\" \".join(split_tokens))\n return output_tokens\n def _run_strip_accents(self, text):\n \"\"\"Strips accents from a piece of text.\"\"\"\n text = unicodedata.normalize(\"NFD\", text)\n output = []\n for char in text:\n cat = unicodedata.category(char)\n if cat == \"Mn\":\n continue\n output.append(char)\n return \"\".join(output)\n def _run_split_on_punc(self, text):\n \"\"\"Splits punctuation on a piece of text.\"\"\"\n chars = list(text)\n i = 0\n start_new_word = True\n output = []\n while i < len(chars):"
+ },
+ {
+ "comment": "This code defines functions for tokenizing and processing text data. The `_is_punctuation` function identifies punctuation characters, while the `tokenize_text` function separates words by detecting new word starts. The `_tokenize_chinese_chars` function adds whitespace around Chinese characters to separate them from surrounding text.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":229-258",
+ "content": " char = chars[i]\n if _is_punctuation(char):\n output.append([char])\n start_new_word = True\n else:\n if start_new_word:\n output.append([])\n start_new_word = False\n output[-1].append(char)\n i += 1\n return [\"\".join(x) for x in output]\n def _tokenize_chinese_chars(self, text):\n \"\"\"Adds whitespace around any CJK character.\"\"\"\n output = []\n for char in text:\n cp = ord(char)\n if self._is_chinese_char(cp):\n output.append(\" \")\n output.append(char)\n output.append(\" \")\n else:\n output.append(char)\n return \"\".join(output)\n def _is_chinese_char(self, cp):\n \"\"\"Checks whether CP is the codepoint of a CJK character.\"\"\"\n # This defines a \"chinese character\" as anything in the CJK Unicode block:\n # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)"
+ },
+ {
+ "comment": "The code checks if a character falls within the CJK Unicode block, which includes Japanese and Korean characters. It returns True if any of these characters are found, indicating that the text is in one of these languages. The function also performs invalid character removal and whitespace cleanup on the given text.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":259-281",
+ "content": " #\n # Note that the CJK Unicode block is NOT all Japanese and Korean characters,\n # despite its name. The modern Korean Hangul alphabet is a different block,\n # as is Japanese Hiragana and Katakana. Those alphabets are used to write\n # space-separated words, so they are not treated specially and handled\n # like the all of the other languages.\n if ((cp >= 0x4E00 and cp <= 0x9FFF) or #\n (cp >= 0x3400 and cp <= 0x4DBF) or #\n (cp >= 0x20000 and cp <= 0x2A6DF) or #\n (cp >= 0x2A700 and cp <= 0x2B73F) or #\n (cp >= 0x2B740 and cp <= 0x2B81F) or #\n (cp >= 0x2B820 and cp <= 0x2CEAF) or\n (cp >= 0xF900 and cp <= 0xFAFF) or #\n (cp >= 0x2F800 and cp <= 0x2FA1F)): #\n return True\n return False\n def _clean_text(self, text):\n \"\"\"Performs invalid character removal and whitespace cleanup on text.\"\"\"\n output = []\n for char in text:\n cp = ord(char)"
+ },
+ {
+ "comment": "This code defines a WordpieceTokenizer class that tokenizes text into word pieces using a greedy longest-match-first algorithm and a given vocabulary. The tokenize method takes in a text input, performs tokenization by matching the longest possible substrings from the vocabulary, and returns a list of wordpiece tokens.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":282-314",
+ "content": " if cp == 0 or cp == 0xfffd or _is_control(char):\n continue\n if _is_whitespace(char):\n output.append(\" \")\n else:\n output.append(char)\n return \"\".join(output)\nclass WordpieceTokenizer(object):\n \"\"\"Runs WordPiece tokenziation.\"\"\"\n def __init__(self, vocab, unk_token=\"[UNK]\", max_input_chars_per_word=100):\n self.vocab = vocab\n self.unk_token = unk_token\n self.max_input_chars_per_word = max_input_chars_per_word\n def tokenize(self, text):\n \"\"\"Tokenizes a piece of text into its word pieces.\n This uses a greedy longest-match-first algorithm to perform tokenization\n using the given vocabulary.\n For example:\n input = \"unaffable\"\n output = [\"un\", \"##aff\", \"##able\"]\n Args:\n text: A single token or whitespace separated tokens. This should have\n already been passed through `BasicTokenizer.\n Returns:\n A list of wordpiece tokens."
+ },
+ {
+ "comment": "This code tokenizes text by splitting it into words, checks if each word is in the vocabulary. If not, it adds a special unknown token. It handles long words by splitting them into smaller parts and checking each part separately.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":315-347",
+ "content": " \"\"\"\n text = convert_to_unicode(text)\n output_tokens = []\n for token in whitespace_tokenize(text):\n chars = list(token)\n if len(chars) > self.max_input_chars_per_word:\n output_tokens.append(self.unk_token)\n continue\n is_bad = False\n start = 0\n sub_tokens = []\n while start < len(chars):\n end = len(chars)\n cur_substr = None\n while start < end:\n substr = \"\".join(chars[start:end])\n if start > 0:\n substr = \"##\" + substr\n if substr in self.vocab:\n cur_substr = substr\n break\n end -= 1\n if cur_substr is None:\n is_bad = True\n break\n sub_tokens.append(cur_substr)\n start = end\n if is_bad:\n output_tokens.append(self.unk_token)"
+ },
+ {
+ "comment": "The code defines several functions for tokenizing a string: _is_whitespace checks if the character is whitespace, _is_control identifies control characters, and _is_punctuation classifies punctuation. The main function extends output tokens based on these character types.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":348-381",
+ "content": " else:\n output_tokens.extend(sub_tokens)\n return output_tokens\ndef _is_whitespace(char):\n \"\"\"Checks whether `chars` is a whitespace character.\"\"\"\n # \\t, \\n, and \\r are technically contorl characters but we treat them\n # as whitespace since they are generally considered as such.\n if char == \" \" or char == \"\\t\" or char == \"\\n\" or char == \"\\r\":\n return True\n cat = unicodedata.category(char)\n if cat == \"Zs\":\n return True\n return False\ndef _is_control(char):\n \"\"\"Checks whether `chars` is a control character.\"\"\"\n # These are technically control characters but we count them as whitespace\n # characters.\n if char == \"\\t\" or char == \"\\n\" or char == \"\\r\":\n return False\n cat = unicodedata.category(char)\n if cat.startswith(\"C\"):\n return True\n return False\ndef _is_punctuation(char):\n \"\"\"Checks whether `chars` is a punctuation character.\"\"\"\n cp = ord(char)\n # We treat all non-letter/number ASCII as punctuation.\n # Characters such as \"^\", \"$\", and \"`\" are not in the Unicode"
+ },
+ {
+ "comment": "This code checks if a given character is a punctuation or Chinese character by checking its Unicode category and code point range. It returns True if the character is a punctuation or Chinese character, and False otherwise. The function is used to tokenize Chinese characters in text by adding whitespace around them.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":382-404",
+ "content": " # Punctuation class but we treat them as punctuation anyways, for\n # consistency.\n if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or\n (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):\n return True\n cat = unicodedata.category(char)\n if cat.startswith(\"P\"):\n return True\n return False\ndef tokenize_chinese_chars(text):\n \"\"\"Adds whitespace around any CJK character.\"\"\"\n def _is_chinese_char(cp):\n \"\"\"Checks whether CP is the codepoint of a CJK character.\"\"\"\n # This defines a \"chinese character\" as anything in the CJK Unicode block:\n # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)\n #\n # Note that the CJK Unicode block is NOT all Japanese and Korean characters,\n # despite its name. The modern Korean Hangul alphabet is a different block,\n # as is Japanese Hiragana and Katakana. Those alphabets are used to write\n # space-separated words, so they are not treated specially and handled"
+ },
+ {
+ "comment": "This function tokenizes text by detecting Chinese characters and whitespace, appending non-Chinese characters to a buffer and adding the buffer to the output when a space or Chinese character is found. Finally, it appends any remaining buffer content.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/tokenization.py\":405-440",
+ "content": " # like the all of the other languages.\n if ((cp >= 0x4E00 and cp <= 0x9FFF) or #\n (cp >= 0x3400 and cp <= 0x4DBF) or #\n (cp >= 0x20000 and cp <= 0x2A6DF) or #\n (cp >= 0x2A700 and cp <= 0x2B73F) or #\n (cp >= 0x2B740 and cp <= 0x2B81F) or #\n (cp >= 0x2B820 and cp <= 0x2CEAF) or\n (cp >= 0xF900 and cp <= 0xFAFF) or #\n (cp >= 0x2F800 and cp <= 0x2FA1F)): #\n return True\n return False\n def _is_whitespace(c):\n \"\"\"_is_whitespace\n \"\"\"\n if c == \" \" or c == \"\\t\" or c == \"\\r\" or c == \"\\n\" or ord(c) == 0x202F:\n return True\n return False\n output = []\n buff = \"\"\n for char in text:\n cp = ord(char)\n if _is_chinese_char(cp) or _is_whitespace(char):\n if buff != \"\":\n output.append(buff)\n buff = \"\"\n output.append(char)\n else:\n buff += char\n if buff != \"\":\n output.append(buff)\n return output"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/328051fb-fd67-458d-8465-2d715eefefcb.json b/docs/doc/328051fb-fd67-458d-8465-2d715eefefcb.json
new file mode 100644
index 000000000..39018964a
--- /dev/null
+++ b/docs/doc/328051fb-fd67-458d-8465-2d715eefefcb.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code defines the TransNetV2Head class, a type of head used in computer vision models, inheriting from BaseHead with arguments for number of classes, input channels, and loss configuration. It also includes TransNetV2Loss class registered as HEADS registry, and two methods (loss and get_score) for calculating loss and F1 score between predictions and ground truth.",
+ "details": [
+ {
+ "comment": "This code defines the TransNetV2Head class, which is a type of head used in computer vision models. It inherits from BaseHead and takes arguments for number of classes, input channels, and loss configuration. The TransNetV2Loss class is registered with the HEADS registry to be used by this head. Additionally, the code provides comments about licensing and copyright information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/transnetv2_head.py\":0-28",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..losses import TransNetV2Loss\nfrom ...metrics.transnetv2_metric import create_scene_based_summaries\n@HEADS.register()\nclass TransNetV2Head(BaseHead):\n \"\"\"TransNetV2 Head.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name=\"TransNetV2Loss\")\n ):\n super().__init__(num_classes,"
+ },
+ {
+ "comment": "This code defines a class with two methods, `loss` and `get_score`. The `loss` method calculates the loss between predictions and ground truth, while `get_score` method calculates an F1 score based on one-hot predictions and ground truth. The calculated losses are stored in a dictionary for further use.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/transnetv2_head.py\":29-44",
+ "content": " in_channels,\n loss_cfg)\n def loss(self, one_hot_pred, one_hot_gt,\n many_hot_pred=None, many_hot_gt=None, reg_losses=None):\n losses = dict()\n loss = self.loss_func(scores, labels, **kwargs)\n f1 = self.get_score(one_hot_pred, one_hot_gt)\n losses['f1'] = f1\n losses['loss'] = loss\n return losses\n def get_score(self, one_hot_pred, one_hot_gt):\n f1 = create_scene_based_summaries(one_hot_pred, one_hot_gt)\n return f1"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/334d3227-e933-4a5b-b9a2-192dee28148a.json b/docs/doc/334d3227-e933-4a5b-b9a2-192dee28148a.json
new file mode 100644
index 000000000..526b06161
--- /dev/null
+++ b/docs/doc/334d3227-e933-4a5b-b9a2-192dee28148a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This function takes a label input, converts it into an array and uses bitwise operations to create a color map representing the label values in RGB format.",
+ "details": [
+ {
+ "comment": "This function takes a label input, converts it into an array and uses bitwise operations to create a color map representing the label values in RGB format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/utils.py\":0-11",
+ "content": "import numpy as np\ndef label2colormap(label):\n m = label.astype(np.uint8)\n r, c = m.shape\n cmap = np.zeros((r, c, 3), dtype=np.uint8)\n cmap[:, :, 0] = (m & 1) << 7 | (m & 8) << 3 | (m & 64) >> 1\n cmap[:, :, 1] = (m & 2) << 6 | (m & 16) << 2 | (m & 128) >> 2\n cmap[:, :, 2] = (m & 4) << 5 | (m & 32) << 1\n return cmap"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/33b4abff-6ca7-4d03-98eb-625dbfade5e9.json b/docs/doc/33b4abff-6ca7-4d03-98eb-625dbfade5e9.json
new file mode 100644
index 000000000..4bdec7550
--- /dev/null
+++ b/docs/doc/33b4abff-6ca7-4d03-98eb-625dbfade5e9.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code imports necessary functions and classes from other modules, defines the exported symbols (build_batch_pipeline and Compose), and sets license information.",
+ "details": [
+ {
+ "comment": "This code imports necessary functions and classes from other modules, defines the exported symbols (build_batch_pipeline and Compose), and sets license information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/loader/__init__.py\":0-19",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .builder import build_batch_pipeline\nfrom .pipelines.compose import Compose\n__all__ = [\n 'build_batch_pipeline','Compose'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/33d49b0b-7b36-4022-84a8-496d8342a35b.json b/docs/doc/33d49b0b-7b36-4022-84a8-496d8342a35b.json
new file mode 100644
index 000000000..c9378d77e
--- /dev/null
+++ b/docs/doc/33d49b0b-7b36-4022-84a8-496d8342a35b.json
@@ -0,0 +1,55 @@
+{
+ "summary": "The code configures logging, imports libraries, initializes a video tagging model using PaddlePaddle and PaddleVideo. It sets up input data with efficient execution on GPU/CPU resources, measures predictor model's execution time for performance analysis or optimization within the main script function.",
+ "details": [
+ {
+ "comment": "Code sets the logging configuration for INFO level, defines the log format, and redirects the logs to stdout. It also imports necessary libraries and modules, and configures logging handlers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":0-33",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport os\nimport sys\nimport time\nimport logging\nimport argparse\nimport ast\nimport numpy as np\nimport paddle\nimport paddle.static as static\nfrom utils.config_utils import *\nimport models\nfrom reader import get_reader\nfrom metrics import get_metrics\nfrom utils.utility import check_cuda\nfrom utils.utility import check_version\nlogging.root.handlers = []\nFORMAT = '[%(levelname)s: %(filename)s: %(lineno)4d]: %(message)s'\nlogging.basicConfig(level=logging.INFO, format=FORMAT, stream=sys.stdout)"
+ },
+ {
+ "comment": "This code snippet defines a function \"parse_args()\" which uses the argparse module to parse command-line arguments. It sets default values for extractor and predictor model configurations, names, and enables GPU usage by default.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":34-61",
+ "content": "logger = logging.getLogger(__name__)\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--extractor_config',\n type=str,\n default='configs/tsn.yaml',\n help='path to config file of model')\n parser.add_argument('--extractor_name',\n type=str,\n default='TSN',\n help='extractor model name, default TSN')\n parser.add_argument('--predictor_config',\n '--pconfig',\n type=str,\n default='configs/attention_lstm.yaml',\n help='path to config file of model')\n parser.add_argument(\n '--predictor_name',\n '--pname',\n type=str,\n default='AttentionLSTM',\n help='predictor model name, as AttentionLSTM, AttentionCluster, NEXTVLAD'\n )\n parser.add_argument('--use_gpu',\n type=ast.literal_eval,\n default=True,"
+ },
+ {
+ "comment": "This code snippet uses the argparse module to define command-line arguments for a video tagging application. These arguments include GPU usage, extractor and predictor weight paths, input file list, output directory, and Chinese label file path. The function `parser.parse_args()` is called at the end to return these arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":62-85",
+ "content": " help='default use gpu.')\n parser.add_argument('--extractor_weights',\n type=str,\n default='weights/tsn',\n help='extractor weight path')\n parser.add_argument('--predictor_weights',\n '--pweights',\n type=str,\n default='weights/attention_lstm',\n help='predictor weight path')\n parser.add_argument('--filelist',\n type=str,\n default='./data/VideoTag_test.list',\n help='path of video data, multiple video')\n parser.add_argument('--save_dir',\n type=str,\n default='data/VideoTag_results',\n help='output file path')\n parser.add_argument('--label_file',\n type=str,\n default='label_3396.txt',\n help='chinese label file path')\n args = parser.parse_args()"
+ },
+ {
+ "comment": "This code defines a video classification model with two stages: extracting features from the input video using an extractor and predicting the classification results based on those extracted features. It uses PaddlePaddle's static graph mode for performance improvement and organizes the code within name scopes \"extractor_scope\" and \"predictor_scope\". The code also checks if the save directory exists, creates it if not, and measures time taken by the extractor stage.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":86-110",
+ "content": " return args\ndef main():\n \"\"\"\n Video classification model of 3000 Chinese tags.\n videotag_extractor_prdictor (as videotag_TSN_AttentionLSTM)\n two stages in our model:\n 1. extract feature from input video(mp4 format) using extractor\n 2. predict classification results from extracted feature using predictor\n we implement this using two name scopes, ie. extractor_scope and predictor_scope.\n \"\"\"\n if not os.path.isdir(args.save_dir):\n os.makedirs(args.save_dir)\n extractor_config = parse_config(args.extractor_config)\n extractor_infer_config = merge_configs(extractor_config, 'infer',\n vars(args))\n extractor_start_time = time.time()\n extractor_scope = paddle.static.Scope()\n with static.scope_guard(extractor_scope):\n extractor_startup_prog = static.Program()\n extractor_main_prog = static.Program()\n with static.program_guard(extractor_main_prog, extractor_startup_prog):\n paddle.disable_static()"
+ },
+ {
+ "comment": "This code builds a model, sets up the necessary parameters for execution, and loads pre-trained weights from a specified location. The model is built in inferencing mode for video tagging tasks, and it utilizes GPU or CPU resources based on the provided arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":111-133",
+ "content": " # build model\n extractor_model = models.get_model(args.extractor_name,\n extractor_infer_config,\n mode='infer',\n is_videotag=True)\n extractor_model.build_input(use_dataloader=False)\n extractor_model.build_model()\n extractor_feeds = extractor_model.feeds()\n extractor_fetch_list = extractor_model.fetches()\n place = paddle.CUDAPlace(0) if args.use_gpu else paddle.CPUPlace()\n exe = static.Executor(place)\n exe.run(extractor_startup_prog)\n logger.info('load extractor weights from {}'.format(\n args.extractor_weights))\n extractor_model.load_pretrain_params(exe,\n args.extractor_weights,\n extractor_main_prog)\n # get reader and metrics"
+ },
+ {
+ "comment": "The code is setting up a reader and feeder for an extractor in PaddleVideo, iterating through data from the reader, running the extractor using static mode, and logging progress. It also measures and prints the time taken for extraction.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":134-153",
+ "content": " extractor_reader = get_reader(args.extractor_name, 'infer',\n extractor_infer_config)\n extractor_feeder = paddle.fluid.DataFeeder(place=place,\n feed_list=extractor_feeds)\n feature_list = []\n file_list = []\n for idx, data in enumerate(extractor_reader()):\n file_id = [item[-1] for item in data]\n feed_data = [item[:-1] for item in data]\n feature_out = exe.run(fetch_list=extractor_fetch_list,\n feed=extractor_feeder.feed(feed_data))\n feature_list.append(feature_out[0]) #get out from list\n file_list.append(file_id)\n logger.info(\n '========[Stage 1 Sample {} ] Extractor finished======'.\n format(idx))\n paddle.enable_static()\n extractor_end_time = time.time()\n print('extractor_time', extractor_end_time - extractor_start_time)"
+ },
+ {
+ "comment": "This code configures and prepares input data for a predictor model. It first parses the predictor configuration file, then merges it with command line arguments to create an inferencing configuration. Depending on the specified predictor model, it either extracts relevant segments from feature lists or uses the entire feature list. The resulting data is added to a list of inputs for the predictor model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":155-176",
+ "content": " predictor_config = parse_config(args.predictor_config)\n predictor_infer_config = merge_configs(predictor_config, 'infer',\n vars(args))\n # get Predictor input from Extractor output\n predictor_feed_list = []\n for i in range(len(feature_list)):\n feature_out = feature_list[i]\n if args.predictor_name == \"AttentionCluster\":\n extractor_seg_num = extractor_infer_config.INFER.seg_num\n predictor_seg_num = predictor_infer_config.MODEL.seg_num\n idxs = []\n stride = float(extractor_seg_num) / predictor_seg_num\n for j in range(predictor_seg_num):\n pos = (j + np.random.random()) * stride\n idxs.append(min(extractor_seg_num - 1, int(pos)))\n extractor_feature = feature_out[:, idxs, :].astype(\n float) # get from bs dim\n else:\n extractor_feature = feature_out.astype(float)\n predictor_feed_data = [extractor_feature]\n predictor_feed_list.append((predictor_feed_data, file_list[i]))"
+ },
+ {
+ "comment": "This code sets up a predictor model, builds its inputs, builds the model itself, initializes feeds, runs a startup program, loads test weights from a specified location, and performs these actions within scopes and programs for efficient execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":178-198",
+ "content": " predictor_start_time = time.time()\n predictor_scope = paddle.static.Scope()\n with static.scope_guard(predictor_scope):\n predictor_startup_prog = static.Program()\n predictor_main_prog = static.Program()\n with static.program_guard(predictor_main_prog, predictor_startup_prog):\n paddle.disable_static()\n # parse config\n predictor_model = models.get_model(args.predictor_name,\n predictor_infer_config,\n mode='infer')\n predictor_model.build_input(use_dataloader=False)\n predictor_model.build_model()\n predictor_feeds = predictor_model.feeds()\n exe.run(predictor_startup_prog)\n logger.info('load predictor weights from {}'.format(\n args.predictor_weights))\n predictor_model.load_test_weights(exe, args.predictor_weights,\n predictor_main_prog)"
+ },
+ {
+ "comment": "This code snippet is initializing a DataFeeder for predictor model, fetching the list of metrics for predictor model and resetting them. It then iterates over the feed data, runs the model with each data instance, accumulates the final results in the metrics object, and finally logs the output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":200-220",
+ "content": " predictor_feeder = paddle.fluid.DataFeeder(place=place,\n feed_list=predictor_feeds)\n predictor_fetch_list = predictor_model.fetches()\n predictor_metrics = get_metrics(args.predictor_name.upper(),\n 'infer', predictor_infer_config)\n predictor_metrics.reset()\n for idx, data in enumerate(predictor_feed_list):\n file_id = data[1]\n predictor_feed_data = data[0]\n final_outs = exe.run(\n fetch_list=predictor_fetch_list,\n feed=predictor_feeder.feed(predictor_feed_data))\n logger.info(\n '=======[Stage 2 Sample {} ] Predictor finished========'\n .format(idx))\n final_result_list = [item\n for item in final_outs] + [file_id]\n predictor_metrics.accumulate(final_result_list)\n predictor_metrics.finalize_and_log_out("
+ },
+ {
+ "comment": "The code measures the time taken for a predictor to run and outputs it. It also records the total time taken for inferencing and displays the result, indicating when the inference is finished. This code snippet appears within the main function of the script, suggesting that this timing information is used for performance analysis or optimization purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/videotag_test.py\":221-237",
+ "content": " savedir=args.save_dir, label_file=args.label_file)\n paddle.enable_static()\n predictor_end_time = time.time()\n print('predictor_time', predictor_end_time - predictor_start_time)\nif __name__ == '__main__':\n start_time = time.time()\n args = parse_args()\n print(args)\n check_cuda(args.use_gpu)\n check_version()\n logger.info(args)\n main()\n end_time = time.time()\n period = end_time - start_time\n print('[INFER] infer finished. cost time: {}'.format(period))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/34596d26-8de0-4e85-9571-2b797bdf81bf.json b/docs/doc/34596d26-8de0-4e85-9571-2b797bdf81bf.json
new file mode 100644
index 000000000..ee913c33f
--- /dev/null
+++ b/docs/doc/34596d26-8de0-4e85-9571-2b797bdf81bf.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code calculates precision, recall, and F1-score for a TransNetV2 metric machine learning model by handling scene location transformations and errors. It iterates through different thresholds before logging the results.",
+ "details": [
+ {
+ "comment": "This function, named \"predictions_to_scenes\", takes in a list of predictions and outputs a list of scene locations. The scenes are determined by identifying changes from 0 to 1 and vice versa. If all predictions are 1, the function adds a final scene ending at the last index. The code also includes error checking for cases where all predictions are 1 or when there is a disruption in prediction data flow.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/transnetv2_metric.py\":0-33",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ndef predictions_to_scenes(predictions):\n scenes = []\n t, t_prev, start = -1, 0, 0\n for i, t in enumerate(predictions):\n if t_prev == 1 and t == 0:\n start = i\n if t_prev == 0 and t == 1 and i != 0:\n scenes.append([start, i])\n t_prev = t\n if t == 0:\n scenes.append([start, i])\n # just fix if all predictions are 1"
+ },
+ {
+ "comment": "This function converts scene lists to transition lists. If there are no scenes, it returns a transition list with one element. The function is based on an external source and adapted for specific use cases. It can handle different tolerance margins, which affects how the pred_scenes and gt_scenes are transformed into prediction transitions (pred_trans) and ground truth transitions (gt_trans), respectively. A \"HIT\" or \"MISS\" status is determined based on these converted lists.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/transnetv2_metric.py\":34-56",
+ "content": " if len(scenes) == 0:\n return np.array([[0, len(predictions) - 1]], dtype=np.int32)\n return np.array(scenes, dtype=np.int32)\ndef evaluate_scenes(gt_scenes, pred_scenes, n_frames_miss_tolerance=2):\n \"\"\"\n Adapted from: https://github.com/gyglim/shot-detection-evaluation\n The original based on: http://imagelab.ing.unimore.it/imagelab/researchActivity.asp?idActivity=19\n n_frames_miss_tolerance:\n Number of frames it is possible to miss ground truth by, and still being counted as a correct detection.\n Examples of computation with different tolerance margin:\n n_frames_miss_tolerance = 0\n pred_scenes: [[0, 5], [6, 9]] -> pred_trans: [[5.5, 5.5]]\n gt_scenes: [[0, 5], [6, 9]] -> gt_trans: [[5.5, 5.5]] -> HIT\n gt_scenes: [[0, 4], [5, 9]] -> gt_trans: [[4.5, 4.5]] -> MISS\n n_frames_miss_tolerance = 1\n pred_scenes: [[0, 5], [6, 9]] -> pred_trans: [[5.0, 6.0]]\n gt_scenes: [[0, 5], [6, 9]] -> gt_trans: [[5.0, 6.0]] -> HIT\n gt_scenes: [[0, 4], [5, 9]] -> gt_trans: [[4.0, 5.0]] -> HIT"
+ },
+ {
+ "comment": "This code adjusts and transforms input frame scene and transition data, and then iterates through both to calculate true positives (TP), false positives (FP), and false negatives (FN) for evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/transnetv2_metric.py\":57-79",
+ "content": " gt_scenes: [[0, 3], [4, 9]] -> gt_trans: [[3.0, 4.0]] -> MISS\n n_frames_miss_tolerance = 2\n pred_scenes: [[0, 5], [6, 9]] -> pred_trans: [[4.5, 6.5]]\n gt_scenes: [[0, 5], [6, 9]] -> gt_trans: [[4.5, 6.5]] -> HIT\n gt_scenes: [[0, 4], [5, 9]] -> gt_trans: [[3.5, 5.5]] -> HIT\n gt_scenes: [[0, 3], [4, 9]] -> gt_trans: [[2.5, 4.5]] -> HIT\n gt_scenes: [[0, 2], [3, 9]] -> gt_trans: [[1.5, 3.5]] -> MISS\n Users should be careful about adopting these functions in any commercial matters.\n \"\"\"\n shift = n_frames_miss_tolerance / 2\n gt_scenes = gt_scenes.astype(np.float32) + np.array([[-0.5 + shift, 0.5 - shift]])\n pred_scenes = pred_scenes.astype(np.float32) + np.array([[-0.5 + shift, 0.5 - shift]])\n gt_trans = np.stack([gt_scenes[:-1, 1], gt_scenes[1:, 0]], 1)\n pred_trans = np.stack([pred_scenes[:-1, 1], pred_scenes[1:, 0]], 1)\n i, j = 0, 0\n tp, fp, fn = 0, 0, 0\n while i < len(gt_trans) or j < len(pred_trans):\n if j == len(pred_trans) or pred_trans[j, 0] > gt_trans[i, 1]:"
+ },
+ {
+ "comment": "This function calculates precision, recall, and F1-score for transnetv2 metric given ground truth (gt) and predicted (pred) transcript sequences. It iterates through the sequences to count true positives (tp), false negatives (fn), and false positives (fp). Afterwards, it computes precision, recall, and F1-score based on these counts. The function also asserts that the total number of true positives matches the length of gt_trans and the total number of false positives matches the length of pred_trans. It then returns the calculated metrics and the count of tp, fp, and fn. The create_scene_based_summaries function generates precision, recall, and F1-score for different thresholds using a numpy array. It initializes these metrics as well as the counts of true positives, false negatives, false positives, and false negatives to zero, then iterates over the thresholds to calculate the metric values for each threshold.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/transnetv2_metric.py\":80-119",
+ "content": " fn += 1\n i += 1\n elif i == len(gt_trans) or pred_trans[j, 1] < gt_trans[i, 0]:\n fp += 1\n j += 1\n else:\n i += 1\n j += 1\n tp += 1\n if tp + fp != 0:\n p = tp / (tp + fp)\n else:\n p = 0\n if tp + fn != 0:\n r = tp / (tp + fn)\n else:\n r = 0\n if p + r != 0:\n f1 = (p * r * 2) / (p + r)\n else:\n f1 = 0\n assert tp + fn == len(gt_trans)\n assert tp + fp == len(pred_trans)\n return p, r, f1, (tp, fp, fn)\ndef create_scene_based_summaries(one_hot_pred, one_hot_gt):\n thresholds = np.array([\n 0.02, 0.06, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\n ])\n precision, recall, f1, tp, fp, fn = np.zeros_like(thresholds), np.zeros_like(thresholds),\\\n np.zeros_like(thresholds), np.zeros_like(thresholds),\\\n np.zeros_like(thresholds), np.zeros_like(thresholds)\n gt_scenes = predictions_to_scenes(one_hot_gt)"
+ },
+ {
+ "comment": "This code is from the TransNetV2Metric class, which calculates metrics for a model's predictions. It iterates through different thresholds to compute precision, recall, F1 score, and true positive, false positive, and false negative counts. The update method appends predictions and computes metrics when a new file is encountered.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/transnetv2_metric.py\":120-151",
+ "content": " for i in range(len(thresholds)):\n pred_scenes = predictions_to_scenes(\n (one_hot_pred > thresholds[i]).astype(np.uint8)\n )\n precision[i], recall[i], f1[i], (tp[i], fp[i], fn[i]) = evaluate_scenes(gt_scenes, pred_scenes)\n best_idx = np.argmax(f1)\n return f1[best_idx]\n@METRIC.register\nclass TransNetV2Metric(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.predictions = []\n self.total_stats = {\"tp\": 0, \"fp\": 0, \"fn\": 0}\n def update(self, batch_id, data, one_hot):\n \"\"\"update metrics during each iter\n \"\"\"\n if isinstance(one_hot, tuple):\n one_hot = one_hot[0]\n one_hot = paddle.nn.functional.sigmoid(one_hot)[0]\n self.predictions.append(one_hot.numpy()[25:75])\n gt_scenes = data[1]\n is_new_file = data[2]\n if is_new_file:\n self.compute(gt_scenes)\n # preds ensemble"
+ },
+ {
+ "comment": "The code calculates precision, recall, and F1 score for a machine learning model. It accumulates the metrics after processing all batches and logs the results using logger. It also displays the Precision, Recall, and F1 Score at the end of computation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/transnetv2_metric.py\":152-173",
+ "content": " if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def compute(self, gt_scenes):\n predictions = np.concatenate(self.predictions, 0)[:len(frames)]\n _, _, _, (tp, fp, fn), fp_mistakes, fn_mistakes = evaluate_scenes(\n gt_scenes, predictions_to_scenes((predictions >= args.thr).astype(np.uint8)))\n self.total_stats[\"tp\"] += tp\n self.total_stats[\"fp\"] += fp\n self.total_stats[\"fn\"] += fn\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n p = self.total_stats[\"tp\"] / (self.total_stats[\"tp\"] + self.total_stats[\"fp\"])\n r = self.total_stats[\"tp\"] / (self.total_stats[\"tp\"] + self.total_stats[\"fn\"])\n f1 = (p * r * 2) / (p + r)\n logger.info('[TEST] finished, Precision= {:5.2f}, Recall= {:5.2f} , F1 Score= {:5.2f} '.format(\n p * 100, r * 100, f1 * 100))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/346792a9-3f14-4fe1-91d3-fa862c3ec732.json b/docs/doc/346792a9-3f14-4fe1-91d3-fa862c3ec732.json
new file mode 100644
index 000000000..0537160c7
--- /dev/null
+++ b/docs/doc/346792a9-3f14-4fe1-91d3-fa862c3ec732.json
@@ -0,0 +1,10 @@
+{
+ "summary": "Code snippet registers two models, AttentionLSTM and TSN, in the application's model registry using the functions regist_model() and get_model(). Models are sorted alphabetically for easy retrieval.",
+ "details": [
+ {
+ "comment": "Code snippet registers two models, AttentionLSTM and TSN, in the application's model registry using the functions regist_model() and get_model(). Models are sorted alphabetically for easy retrieval.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/__init__.py\":0-6",
+ "content": "from .model import regist_model, get_model\nfrom .attention_lstm import AttentionLSTM\nfrom .tsn import TSN\n# regist models, sort by alphabet\nregist_model(\"AttentionLSTM\", AttentionLSTM)\nregist_model(\"TSN\", TSN)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/34a9d680-3937-45e1-9d1d-379d0f3be01e.json b/docs/doc/34a9d680-3937-45e1-9d1d-379d0f3be01e.json
new file mode 100644
index 000000000..356cbb867
--- /dev/null
+++ b/docs/doc/34a9d680-3937-45e1-9d1d-379d0f3be01e.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code reads the YouTube-8M dataset, featuring three models (LSTM, attention cluster, nextVlad), and is used for table tennis action detection. It uses cPickle and numpy, and a feature reader initializes for training or inference batches, extracting image, audio, and pcm features.",
+ "details": [
+ {
+ "comment": "This code is a data reader for the YouTube-8M dataset, which contains features extracted by prior networks for three models: LSTM, attention cluster, and nextVlad. It uses cPickle to load data from storage and numpy for numerical operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/reader/feature_reader.py\":0-33",
+ "content": "\"\"\"\nattention-lstm feature reader\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport sys\ntry:\n import cPickle as pickle\n from cStringIO import StringIO\nexcept ImportError:\n import pickle\nimport numpy as np\nimport random\nimport code\nfrom .reader_utils import DataReader\nclass FeatureReader(DataReader):\n \"\"\"\n Data reader for youtube-8M dataset, which was stored as features extracted by prior networks\n This is for the three models: lstm, attention cluster, nextvlad"
+ },
+ {
+ "comment": "This code initializes a feature reader for table tennis action detection. It takes in parameters such as name, mode, configuration, and material. The reader creates lists of image, audio, and pcm features, reshapes the pcm_feature_list, and shuffles proposal list if in train mode. It then defines a reader function that iterates through proposal list to create batches for training or inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/reader/feature_reader.py\":35-70",
+ "content": " dataset cfg: num_classes\n batch_size\n list\n NextVlad only: eigen_file\n \"\"\"\n def __init__(self, name, mode, cfg, material=None):\n self.name = name\n self.mode = mode\n self.batch_size = cfg[self.name.upper()]['batch_size']\n self.feature = material['feature']\n self.proposal = material['proposal']\n self.fps = 5\n def create_reader(self):\n \"\"\"\n create_reader\n \"\"\"\n image_feature_list = self.feature['image_feature']\n audio_feature_list = self.feature['audio_feature']\n pcm_feature_list = self.feature['pcm_feature']\n pcm_feature_list = pcm_feature_list.reshape(\n (pcm_feature_list.shape[0] * 5, 640))\n fl = self.proposal\n if self.mode == 'train':\n random.shuffle(fl)\n def reader():\n \"\"\"\n reader\n \"\"\"\n batch_out = []\n for prop_info in fl:\n start_id = int(prop_info['start'])"
+ },
+ {
+ "comment": "This code segment is part of a feature reader for Table Tennis action prediction. It extracts image, audio, and pcm features from their respective lists based on start and end IDs. If batch size is reached, it yields the batch and resets the batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/reader/feature_reader.py\":71-90",
+ "content": " end_id = int(prop_info['end'])\n bmn_score = float(prop_info['score'])\n try:\n image_feature = image_feature_list[start_id:end_id]\n audio_feature = audio_feature_list[int(start_id / self.fps\n ):int(end_id /\n self.fps)]\n pcm_feature = pcm_feature_list[start_id:end_id]\n # image_feature = np.concatenate((image_feature, pcm_feature), axis=1)\n batch_out.append(\n (image_feature, audio_feature, 0, prop_info))\n if len(batch_out) == self.batch_size:\n yield batch_out\n batch_out = []\n except Exception as e:\n continue\n return reader"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/34da3e09-faf6-4ce8-83de-df2c137c5653.json b/docs/doc/34da3e09-faf6-4ce8-83de-df2c137c5653.json
new file mode 100644
index 000000000..4849562f4
--- /dev/null
+++ b/docs/doc/34da3e09-faf6-4ce8-83de-df2c137c5653.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code defines `ReaderZoo` class for handling errors in PaddleVideo TableTennis, allowing registration and retrieval of different types of readers using named parameters.",
+ "details": [
+ {
+ "comment": "This Python script defines a ReaderNotFoundError exception and a class for handling reader-related errors in the PaddleVideo TableTennis application. It includes license information and allows for checking if a specified reader is available by comparing it to a list of available readers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/reader/reader_utils.py\":0-32",
+ "content": "\"\"\"\nreader_util\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport numpy as np\nclass ReaderNotFoundError(Exception):\n \"\"\"\n \"Error: reader not found\"\n \"\"\"\n def __init__(self, reader_name, avail_readers):\n super(ReaderNotFoundError, self).__init__()\n self.reader_name = reader_name\n self.avail_readers = avail_readers\n def __str__(self):\n msg = \"Reader {} Not Found.\\nAvailiable readers:\\n\".format(\n self.reader_name)"
+ },
+ {
+ "comment": "This code defines a `DataReader` class and a `ReaderZoo` class. The `DataReader` class is a data reader for video input, with methods such as `create_reader`, which should be implemented but is currently empty, and `get_config_from_sec`, which gets configuration from a section in the given config file. The `ReaderZoo` class registers different types of readers based on their names and ensures they inherit from the `DataReader` class. The code also includes functionality to retrieve a reader given its name, mode, configuration, and optionally a material type.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/reader/reader_utils.py\":33-80",
+ "content": " for reader in self.avail_readers:\n msg += \" {}\\n\".format(reader)\n return msg\nclass DataReader(object):\n \"\"\"\n data reader for video input\n \"\"\"\n def __init__(self, model_name, mode, cfg):\n self.name = model_name\n self.mode = mode\n self.cfg = cfg\n def create_reader(self):\n \"\"\"\n Not implemented\n \"\"\"\n pass\n def get_config_from_sec(self, sec, item, default=None):\n \"\"\"\n get_config_from_sec\n \"\"\"\n if sec.upper() not in self.cfg:\n return default\n return self.cfg[sec.upper()].get(item, default)\nclass ReaderZoo(object):\n \"\"\"\n ReaderZoo\n \"\"\"\n def __init__(self):\n \"\"\"\n __init__\n \"\"\"\n self.reader_zoo = {}\n def regist(self, name, reader):\n \"\"\"\n regist\n \"\"\"\n assert reader.__base__ == DataReader, \"Unknow model type {}\".format(\n type(reader))\n self.reader_zoo[name] = reader\n def get(self, name, mode, cfg, material=None):"
+ },
+ {
+ "comment": "This code defines a singleton reader_zoo, allows for registration of readers using the regist_reader() function, and retrieves the registered reader using get_reader() function. The reader instance is created by calling create_reader() on the retrieved reader model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/reader/reader_utils.py\":81-106",
+ "content": " \"\"\"\n get\n \"\"\"\n for k, v in self.reader_zoo.items():\n if k == name:\n return v(name, mode, cfg, material)\n raise ReaderNotFoundError(name, self.reader_zoo.keys())\n# singleton reader_zoo\nreader_zoo = ReaderZoo()\ndef regist_reader(name, reader):\n \"\"\"\n regist_reader\n \"\"\"\n reader_zoo.regist(name, reader)\ndef get_reader(name, mode, cfg, material=None):\n \"\"\"\n get_reader\n \"\"\"\n reader_model = reader_zoo.get(name, mode, cfg, material)\n return reader_model.create_reader()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/34eb7bf8-5164-4b64-84b1-483de0790aa1.json b/docs/doc/34eb7bf8-5164-4b64-84b1-483de0790aa1.json
new file mode 100644
index 000000000..b3439e83d
--- /dev/null
+++ b/docs/doc/34eb7bf8-5164-4b64-84b1-483de0790aa1.json
@@ -0,0 +1,50 @@
+{
+ "summary": "The code defines an Ernie model configuration and initializes the ERNIE model using Paddle's embedding layer. It also includes a multimodal video tagging model, embeddings, data pre-processing, attention mask creation, encoder usage, and TextCNN for sequence feature extraction. The code creates 1D convolutional layers with specified parameters and returns the output.",
+ "details": [
+ {
+ "comment": "This code snippet contains the Ernie model class definition. It imports necessary modules, defines logging, and initializes a class for configuring the Ernie model. The class inherits from `object` and represents the configuration to be used in the Ernie model architecture.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":0-32",
+ "content": "# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Ernie model.\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nfrom __future__ import unicode_literals\nfrom __future__ import absolute_import\nimport json\nimport six\nimport logging\nimport paddle\nimport paddle.static as static\nfrom io import open\nfrom .transformer_encoder import encoder, pre_process_layer\nlog = logging.getLogger(__name__)\nclass ErnieConfig(object):"
+ },
+ {
+ "comment": "This code defines a class for an Ernie model configuration. It initializes the config with a given path, parses the config file using JSON, allows getting and setting items from/to the configuration dictionary, and provides a print_config method to display the configuration in a readable format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":33-72",
+ "content": " \"\"\"\n Erine model config\n \"\"\"\n def __init__(self, config_path):\n \"\"\"\n init\n \"\"\"\n self._config_dict = self._parse(config_path)\n def _parse(self, config_path):\n \"\"\"\n parse config\n \"\"\"\n try:\n with open(config_path, 'r', encoding='utf8') as json_file:\n config_dict = json.load(json_file)\n except Exception:\n raise IOError(\"Error in parsing Ernie model config file '%s'\" %\n config_path)\n else:\n return config_dict\n def __getitem__(self, key):\n \"\"\"\n get item\n \"\"\"\n return self._config_dict.get(key, None)\n def __setitem__(self, key, value):\n \"\"\"\n set item\n \"\"\"\n self._config_dict[key] = value\n def print_config(self):\n \"\"\"\n print config\n \"\"\"\n for arg, value in sorted(six.iteritems(self._config_dict)):\n log.info('%s: %s' % (arg, value))\n log.info('------------------------------------------------')"
+ },
+ {
+ "comment": "The code defines the ErnieModel class, which initializes an ERINE model with parameters such as source ids, position ids, sentence ids, task ids, input mask, configuration, weight sharing, and use of fp16. The class attributes are initialized based on the provided configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":75-105",
+ "content": "class ErnieModel(object):\n \"\"\"\n ERINE Model\n \"\"\"\n def __init__(self,\n src_ids,\n position_ids,\n sentence_ids,\n task_ids,\n input_mask,\n config,\n weight_sharing=True,\n use_fp16=False):\n \"\"\"\n init model\n \"\"\"\n self._emb_size = config['hidden_size']\n self._n_layer = config['num_hidden_layers']\n self._n_head = config['num_attention_heads']\n self._voc_size = config['vocab_size']\n self._max_position_seq_len = config['max_position_embeddings']\n if config['sent_type_vocab_size']:\n self._sent_types = config['sent_type_vocab_size']\n else:\n self._sent_types = config['type_vocab_size']\n self._use_task_id = config['use_task_id']\n if self._use_task_id:\n self._task_types = config['task_type_vocab_size']\n self._hidden_act = config['hidden_act']\n self._prepostprocess_dropout = config['hidden_dropout_prob']"
+ },
+ {
+ "comment": "This code initializes the ERNIE model parameters and builds the model. It sets various attributes such as attention dropout probability, embedding names for word, position, sentence, and task, data types, and initializer range. The _build_model function is then called to create the model using Paddle's embedding layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":106-131",
+ "content": " self._attention_dropout = config['attention_probs_dropout_prob']\n self._weight_sharing = weight_sharing\n self._word_emb_name = \"word_embedding\"\n self._pos_emb_name = \"pos_embedding\"\n self._sent_emb_name = \"sent_embedding\"\n self._task_emb_name = \"task_embedding\"\n self._dtype = \"float16\" if use_fp16 else \"float32\"\n self._emb_dtype = \"float32\"\n # Initialize all weigths by truncated normal initializer, and all biases\n # will be initialized by constant zero by default.\n self._param_initializer = paddle.nn.initializer.TruncatedNormal(\n std=config['initializer_range'])\n self._build_model(src_ids, position_ids, sentence_ids, task_ids,\n input_mask)\n def _build_model(self, src_ids, position_ids, sentence_ids, task_ids,\n input_mask):\n \"\"\"\n build model\n \"\"\"\n # padding id in vocabulary must be set to 0\n emb_out = static.nn.embedding(\n input=src_ids,"
+ },
+ {
+ "comment": "This code initializes and concatenates three embeddings - word, position, and sentence - in a multimodal video tagging model. The embeddings are defined with specific sizes and data types. Two embeddings (position_emb_out and sent_emb_out) are added to the original embedding (emb_out), and then these combined embeddings are returned.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":132-157",
+ "content": " size=[self._voc_size, self._emb_size],\n dtype=self._emb_dtype,\n param_attr=paddle.ParamAttr(\n name=self._word_emb_name, initializer=self._param_initializer),\n is_sparse=False)\n position_emb_out = static.nn.embedding(\n input=position_ids,\n size=[self._max_position_seq_len, self._emb_size],\n dtype=self._emb_dtype,\n param_attr=paddle.ParamAttr(\n name=self._pos_emb_name, initializer=self._param_initializer))\n sent_emb_out = static.nn.embedding(\n sentence_ids,\n size=[self._sent_types, self._emb_size],\n dtype=self._emb_dtype,\n param_attr=paddle.ParamAttr(\n name=self._sent_emb_name, initializer=self._param_initializer))\n # emb_out = emb_out + position_emb_out\n # emb_out = emb_out + sent_emb_out\n emb_out = paddle.add(x=emb_out, y=position_emb_out)\n emb_out = paddle.add(x=emb_out, y=sent_emb_out)\n if self._use_task_id:"
+ },
+ {
+ "comment": "This code initializes an embedding layer for task types, adds it to the embeddings, applies pre-processing with dropout if necessary, and casts the embeddings to the desired dtype. It also creates a self-attention mask, stacks it for each attention head, sets its gradient to stop during backpropagation, and passes the embeddings through an encoder.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":158-183",
+ "content": " task_emb_out = static.nn.embedding(\n task_ids,\n size=[self._task_types, self._emb_size],\n dtype=self._emb_dtype,\n param_attr=paddle.ParamAttr(\n name=self._task_emb_name,\n initializer=self._param_initializer))\n emb_out = emb_out + task_emb_out\n emb_out = pre_process_layer(\n emb_out, 'nd', self._prepostprocess_dropout, name='pre_encoder')\n if self._dtype == \"float16\":\n emb_out = paddle.cast(x=emb_out, dtype=self._dtype)\n input_mask = paddle.cast(x=input_mask, dtype=self._dtype)\n self_attn_mask = paddle.matmul(\n x=input_mask, y=input_mask, transpose_y=True)\n self_attn_mask = paddle.scale(\n x=self_attn_mask, scale=10000.0, bias=-1.0, bias_after_scale=False)\n n_head_self_attn_mask = paddle.stack(\n x=[self_attn_mask] * self._n_head, axis=1)\n n_head_self_attn_mask.stop_gradient = True\n self._enc_out = encoder("
+ },
+ {
+ "comment": "This code is defining and initializing a model for an encoder layer in a deep learning application. The model takes several parameters such as embedding size, number of layers and heads, dropout rates, activation function, etc. It then casts the output to the specified data type if necessary. The `get_sequence_output` method returns the sequence output from the encoder layer and `get_sequence_textcnn_output` takes in a feature sequence and an input mask to generate the output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":184-214",
+ "content": " enc_input=emb_out,\n attn_bias=n_head_self_attn_mask,\n n_layer=self._n_layer,\n n_head=self._n_head,\n d_key=self._emb_size // self._n_head,\n d_value=self._emb_size // self._n_head,\n d_model=self._emb_size,\n d_inner_hid=self._emb_size * 4,\n prepostprocess_dropout=self._prepostprocess_dropout,\n attention_dropout=self._attention_dropout,\n relu_dropout=0,\n hidden_act=self._hidden_act,\n preprocess_cmd=\"\",\n postprocess_cmd=\"dan\",\n param_initializer=self._param_initializer,\n name='encoder')\n if self._dtype == \"float16\":\n self._enc_out = paddle.cast(\n x=self._enc_out, dtype=self._emb_dtype)\n def get_sequence_output(self):\n \"\"\"\n get sequence output\n \"\"\"\n return self._enc_out\n def get_sequence_textcnn_output(self, sequence_feature, input_mask):\n \"\"\"\n get sequence output\n \"\"\""
+ },
+ {
+ "comment": "This code defines a TextCNN model for sequence feature extraction. It pads the input sequence, applies convolutions with various window sizes, and pools the results. The get_pooled_output function extracts the first feature of each sequence for classification by applying an FC layer with tanh activation. The textcnn function initializes a TextCNN model with specified window sizes and hidden dimensions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":215-242",
+ "content": " seq_len = paddle.sum(x=input_mask, axis=[1, 2])\n seq_len = paddle.cast(seq_len, 'int64')\n sequence_feature = paddle.static.nn.sequence_unpad(sequence_feature, seq_len)\n return self.textcnn(sequence_feature)\n def get_pooled_output(self):\n \"\"\"Get the first feature of each sequence for classification\"\"\"\n next_sent_feat = paddle.slice(\n input=self._enc_out, axes=[1], starts=[0], ends=[1])\n next_sent_feat = static.nn.fc(\n x=next_sent_feat,\n size=self._emb_size,\n activation=\"tanh\",\n weight_attr=paddle.ParamAttr(\n name=\"pooled_fc.w_0\", initializer=self._param_initializer),\n bias_attr=\"pooled_fc.b_0\")\n return next_sent_feat\n def textcnn(self, feature, name='text_cnn'):\n \"\"\"\n TextCNN sequence feature extraction\n \"\"\"\n win_sizes = [2, 3, 4]\n hid_dim = 256\n convs = []\n for win_size in win_sizes:\n conv_h = paddle.fluid.nets.sequence_conv_pool(input=feature,"
+ },
+ {
+ "comment": "This code is creating a 1D convolutional layer with specified parameters, including the number of filters, filter size, activation function, and pooling type. The resulting convolutional layers are appended to the `convs` list, and then concatenated along axis 1 to form `convs_out`. Finally, the function returns `convs_out`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/ernie.py\":243-249",
+ "content": " num_filters=hid_dim,\n filter_size=win_size,\n act=\"tanh\",\n pool_type=\"max\")\n convs.append(conv_h)\n convs_out = paddle.concat(x=convs, axis=1)\n return convs_out"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/354a97b8-a143-47ed-9284-36d917d0fd1a.json b/docs/doc/354a97b8-a143-47ed-9284-36d917d0fd1a.json
new file mode 100644
index 000000000..99a278517
--- /dev/null
+++ b/docs/doc/354a97b8-a143-47ed-9284-36d917d0fd1a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "Code sets the QEI_VIDEO_ROOT variable to the absolute path of the directory containing the current file. It also imports the version module from QEIVideo and assigns its __version__ attribute to a variable. This may be used for identifying the version of the QEIVideo application.",
+ "details": [
+ {
+ "comment": "Code sets the QEI_VIDEO_ROOT variable to the absolute path of the directory containing the current file. It also imports the version module from QEIVideo and assigns its __version__ attribute to a variable. This may be used for identifying the version of the QEIVideo application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/__init__.py\":0-12",
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/6 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport os\nQEI_VIDEO_ROOT = os.path.abspath(os.path.dirname(__file__))\nimport os\nfrom QEIVideo.version import __version__\nQEI_VIDEO_ROOT = os.path.abspath(os.path.dirname(__file__))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/355ad68e-66eb-428c-b635-fbad5b4f117a.json b/docs/doc/355ad68e-66eb-428c-b635-fbad5b4f117a.json
new file mode 100644
index 000000000..e92f66b0e
--- /dev/null
+++ b/docs/doc/355ad68e-66eb-428c-b635-fbad5b4f117a.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The MSR-VTT dataset contains 10K videos available on its website, organized in a \"data\" directory for ActBERT model use. The lock.mdb file is a database used for storing and managing data related to multi-modal transformers for video retrieval as described in a 2020 ECCV paper.",
+ "details": [
+ {
+ "comment": "This code provides an overview of the MSR-VTT dataset, its download process for T2VLAD and ActBERT applications, and references for more information. It consists of 10K video clips from 20 categories, each with 20 English sentences, and is available on the MSRVTT website.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/msrvtt.md\":0-28",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../zh-CN/dataset/msrvtt.md) | English\n# MSR-VTT Preparation\n- [Introduction](#1.1)\n- [Download for T2VLAD](#1.2)\n- [Download for ActBERT](#1.3)\n- [Reference](#1.4)\n\n## Introduction\nMSR-VTT(Microsoft Research Video to Text) is a large-scale dataset containing videos and subtitles, which is composed of 10000 video clips from 20 categories, and each video clip is annotated with 20 English sentences. We used 9000 video clips for training and 1000 for testing. For more details, please refer to the website: [MSRVTT](https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/)\n\n## Download for T2VLAD\n[T2VLAD doc](../../../applications/T2VLAD/README_en.md)\nFor ease of use, we provided extracted features of video.\nFirst, make sure to enter the following command in the `applications/T2VLAD/data` directory to download the dataset.\n```bash\nbash download_features.sh\n```\nAfter downloading, the files in the data directory are organized as follows:"
+ },
+ {
+ "comment": "Code provides the instructions to download and decompress data features required for ActBERT model, specifically for MSR-VTT dataset. The data is organized in the \"data\" directory with a .lmdb file and a CSV file containing JSFusion test data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/msrvtt.md\":30-72",
+ "content": "```\n\u251c\u2500\u2500 data\n| \u251c\u2500\u2500 MSR-VTT\n| \u2502 \u251c\u2500\u2500 raw-captions.pkl\n| \u2502 \u251c\u2500\u2500 train_list_jsfusion.txt\n| \u2502 \u251c\u2500\u2500 val_list_jsfusion.txt\n| \u2502 \u251c\u2500\u2500 aggregated_text_feats\n| | | \u251c\u2500\u2500 w2v_MSRVTT_openAIGPT.pickle\n| | \u251c\u2500\u2500 mmt_feats\n| \u2502 \u2502 \u251c\u2500\u2500 features.audio.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.face_agg.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.flos_agg.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.ocr.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.rgb_agg.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.s3d.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.scene.pkl\n| \u2502 \u2502 \u251c\u2500\u2500 features.speech.pkl\n```\n\n## Download for ActBERT\n[ActBERT doc](../model_zoo/multimodal/actbert.md)\nDownload data features:\n```\nwget https://videotag.bj.bcebos.com/Data/ActBERT/msrvtt_test.lmdb.tar\nwget https://videotag.bj.bcebos.com/Data/ActBERT/MSRVTT_JSFUSION_test.csv\n```\nDecompress the `msrvtt_test.lmdb.tar`\uff1a\n```\ntar -zxvf msrvtt_test.lmdb.tar\n```\nThe files in the data directory are organized as follows:\n```\n\u251c\u2500\u2500 data\n| \u251c\u2500\u2500 MSR-VTT\n| \u2502 \u251c\u2500\u2500 MSRVTT_JSFUSION_test.csv\n| \u2502 \u251c\u2500\u2500 msrvtt_test.lmdb\n| \u2502 \u251c\u2500\u2500 data.mdb"
+ },
+ {
+ "comment": "lock.mdb: Database file used for storing and managing data in codebase related to multi-modal transformer for video retrieval as described in the 2020 ECCV paper by Valentin Gabeur et al.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/msrvtt.md\":73-78",
+ "content": "| \u2502 \u251c\u2500\u2500 lock.mdb\n```\n\n## Reference\n- Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, 2020."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/35885247-4755-4325-8af8-b4d9692791e2.json b/docs/doc/35885247-4755-4325-8af8-b4d9692791e2.json
new file mode 100644
index 000000000..f8b7196ad
--- /dev/null
+++ b/docs/doc/35885247-4755-4325-8af8-b4d9692791e2.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code imports libraries, defines a function for parsing command line arguments, and sets up paths and licenses before building the model using PaddleVideo. It initializes segments (num_seg) and summarizes the model's parameters while calculating FLOPs if enabled.",
+ "details": [
+ {
+ "comment": "This code snippet is importing necessary libraries and defining a function for parsing command line arguments in the PaddleVideo project. The code also sets up some paths and licenses, ensuring compliance with the Apache License, Version 2.0.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/summary.py\":0-33",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport argparse\nimport os\nimport sys\nimport os.path as osp\nimport paddle\nimport paddle.nn.functional as F\nfrom paddle.jit import to_static\nimport paddleslim\n__dir__ = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(os.path.abspath(os.path.join(__dir__, '../')))\nfrom paddlevideo.modeling.builder import build_model\nfrom paddlevideo.utils import get_config\ndef parse_args():\n parser = argparse.ArgumentParser(\"PaddleVideo Summary\")"
+ },
+ {
+ "comment": "This code parses arguments for the config file path, image size, and number of segments. It then trims unnecessary attributes from the training configuration before building the model using the parsed arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/summary.py\":34-68",
+ "content": " parser.add_argument('-c',\n '--config',\n type=str,\n default='configs/example.yaml',\n help='config file path')\n parser.add_argument(\"--img_size\", type=int, default=224)\n parser.add_argument(\"--num_seg\", type=int, default=8)\n parser.add_argument(\"--FLOPs\",\n action=\"store_true\",\n help=\"whether to print FLOPs\")\n return parser.parse_args()\ndef _trim(cfg, args):\n \"\"\"\n Reuse the trainging config will bring useless attribute, such as: backbone.pretrained model. Trim it here.\n \"\"\"\n model_name = cfg.model_name\n cfg = cfg.MODEL\n cfg.backbone.pretrained = \"\"\n if 'num_seg' in cfg.backbone:\n cfg.backbone.num_seg = args.num_seg\n return cfg, model_name\ndef main():\n args = parse_args()\n cfg, model_name = _trim(get_config(args.config, show=False), args)\n print(f\"Building model({model_name})...\")\n model = build_model(cfg)\n img_size = args.img_size"
+ },
+ {
+ "comment": "This code snippet initializes the number of segments (num_seg) and currently only supports tsm. It generates a summary of the model's parameters using Paddle's summary function, and if FLOPs is enabled, it also calculates and prints the model's floating-point operations using paddleslim's analysis.flops function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/summary.py\":69-81",
+ "content": " num_seg = args.num_seg\n #NOTE: only support tsm now, will refine soon\n params_info = paddle.summary(model, (1, 1, num_seg, 3, img_size, img_size))\n print(params_info)\n if args.FLOPs:\n flops_info = paddleslim.analysis.flops(\n model, [1, 1, num_seg, 3, img_size, img_size])\n print(flops_info)\nif __name__ == \"__main__\":\n main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/35c844fa-68b5-4457-abfb-24c68e2bdf80.json b/docs/doc/35c844fa-68b5-4457-abfb-24c68e2bdf80.json
new file mode 100644
index 000000000..190caccb9
--- /dev/null
+++ b/docs/doc/35c844fa-68b5-4457-abfb-24c68e2bdf80.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The MultiCropMetric class in PaddleVideo computes top-1/5 accuracy using multi-crop metrics for each video label and logs average values.",
+ "details": [
+ {
+ "comment": "This code snippet defines a class for MultiCropMetric, which is part of the PaddleVideo library. It initializes instances with various parameters such as data_size, batch_size, num_ensemble_views, num_spatial_crops, and num_classes. The log_interval parameter determines how often to update logs during training. This metric appears to be related to slowfast video processing, as specified in the comment.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/multi_crop_metric.py\":0-34",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nfrom paddle.hapi.model import _all_gather\nfrom paddlevideo.utils import get_logger\nfrom .registry import METRIC\nfrom .base import BaseMetric\nlogger = get_logger(\"paddlevideo\")\n\"\"\" An example for metrics class.\n MultiCropMetric for slowfast.\n\"\"\"\n@METRIC.register\nclass MultiCropMetric(BaseMetric):\n def __init__(self,\n data_size,\n batch_size,\n num_ensemble_views,\n num_spatial_crops,\n num_classes,\n log_interval=1):"
+ },
+ {
+ "comment": "This code initializes a multi-crop metric class, which takes data size, batch size, log interval, number of ensemble views, and number of spatial crops as parameters. It calculates the number of videos and clips, creates arrays to store video predictions and labels, and initializes a clip_count dictionary. The update method is used to update metrics during each iteration by gathering data across multiple cards if needed, converting outputs to numpy arrays.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/multi_crop_metric.py\":35-60",
+ "content": " \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.num_ensemble_views = num_ensemble_views\n self.num_spatial_crops = num_spatial_crops\n self.num_classes = num_classes\n self.num_clips = self.num_ensemble_views * self.num_spatial_crops\n num_videos = self.data_size // self.num_clips\n self.video_preds = np.zeros((num_videos, self.num_classes))\n self.video_labels = np.zeros((num_videos, 1), dtype=\"int64\")\n self.clip_count = {}\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n labels = data[2]\n clip_ids = data[3]\n # gather mulit card, results of following process in each card is the same.\n if self.world_size > 1:\n outputs = _all_gather(outputs, self.world_size)\n labels = _all_gather(labels.cuda(), self.world_size)\n clip_ids = _all_gather(clip_ids.cuda(), self.world_size)\n # to numpy"
+ },
+ {
+ "comment": "The code loops through each prediction and label for video clips in a batch. It checks if the clip index has been encountered before for a particular video ID, and updates the count if it's a new clip or performs ensemble by summing predictions if it's not. If there are labels for a video, it asserts they match and updates the label accordingly. The code also logs processing information at log intervals.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/multi_crop_metric.py\":61-82",
+ "content": " preds = outputs.numpy()\n labels = labels.numpy().astype(\"int64\")\n clip_ids = clip_ids.numpy()\n # preds ensemble\n for ind in range(preds.shape[0]):\n vid_id = int(clip_ids[ind]) // self.num_clips\n ts_idx = int(clip_ids[ind]) % self.num_clips\n if vid_id not in self.clip_count:\n self.clip_count[vid_id] = []\n if ts_idx in self.clip_count[vid_id]:\n logger.info(\n \"[TEST] Passed!! read video {} clip index {} / {} repeatedly.\"\n .format(vid_id, ts_idx, clip_ids[ind]))\n else:\n self.clip_count[vid_id].append(ts_idx)\n self.video_preds[vid_id] += preds[ind] # ensemble method: sum\n if self.video_labels[vid_id].sum() > 0:\n assert self.video_labels[vid_id] == labels[ind]\n self.video_labels[vid_id] = labels[ind]\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format("
+ },
+ {
+ "comment": "This code defines a class that accumulates metrics when all iterations are finished. It checks if the number of clips and their counts match, logging an error if not. Then, it converts video predictions and labels to Paddle tensors and calculates top-1 and top-5 accuracy using Paddle's metric library.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/multi_crop_metric.py\":83-103",
+ "content": " batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n # check clip index of each video\n for key in self.clip_count.keys():\n if len(self.clip_count[key]) != self.num_clips or sum(\n self.clip_count[key]) != self.num_clips * (self.num_clips -\n 1) / 2:\n logger.info(\n \"[TEST] Count Error!! video [{}] clip count [{}] not match number clips {}\"\n .format(key, self.clip_count[key], self.num_clips))\n video_preds = paddle.to_tensor(self.video_preds)\n video_labels = paddle.to_tensor(self.video_labels)\n acc_top1 = paddle.metric.accuracy(input=video_preds,\n label=video_labels,\n k=1)\n acc_top5 = paddle.metric.accuracy(input=video_preds,"
+ },
+ {
+ "comment": "Calculates top-1 and top-5 accuracy using multi-crop metric for each video label, then logs the average accuracy values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/multi_crop_metric.py\":104-107",
+ "content": " label=video_labels,\n k=5)\n logger.info('[TEST] finished, avg_acc1= {}, avg_acc5= {} '.format(\n acc_top1.numpy(), acc_top5.numpy()))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3633f597-c414-410b-98c3-14cb08e86f41.json b/docs/doc/3633f597-c414-410b-98c3-14cb08e86f41.json
new file mode 100644
index 000000000..164b2a132
--- /dev/null
+++ b/docs/doc/3633f597-c414-410b-98c3-14cb08e86f41.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code implements max margin ranking loss and calculates cosine similarity between images and sentences, including a ContrastiveLoss class for contrastive learning. It also computes the cost for contrastive learning and video-level loss in T2VLAD models with masks, comparisons, and scalings.",
+ "details": [
+ {
+ "comment": "This code snippet contains an implementation of the max margin ranking loss, modified from a source code, and includes functions to calculate cosine similarity between images and sentences. The original code is licensed under the Apache License 2.0.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/loss.py\":0-27",
+ "content": "\"\"\"This module contains an implementation of the max margin ranking loss, slightly\nmodified from this code:\nhttps://github.com/antoine77340/Mixture-of-Embedding-Experts/blob/master/loss.py\nThe modification is the `fix_norm` conditional, which removes zero terms from the\ndiagonal when performing the averaging calculation.\nOriginal licence below.\n\"\"\"\n# Copyright 2021 Antoine Miech All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\ndef cosine_sim(im, s):"
+ },
+ {
+ "comment": "This code calculates cosine similarity between image and sentence pairs, and defines a ContrastiveLoss class to compute contrastive loss for contrastive learning.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/loss.py\":28-60",
+ "content": " '''cosine similarity between all the image and sentence pairs\n '''\n inner_prod = im.mm(s.t())\n im_norm = paddle.sqrt((im ** 2).sum(axis=1).reshape([-1, 1]) + 1e-18) \n s_norm = paddle.sqrt((s ** 2).sum(axis=1).reshape([-1, 1]) + 1e-18)\n sim = inner_prod / (im_norm * s_norm)\n return sim\nclass ContrastiveLoss(nn.Layer):\n '''compute contrastive loss\n '''\n def __init__(self, margin=0, max_violation=True, direction='bi', topk=1):\n '''Args:\n direction: i2t for negative sentence, t2i for negative image, bi for both\n '''\n super().__init__()\n self.margin = margin\n self.max_violation = max_violation\n self.direction = direction\n self.topk = topk\n def forward(self, scores, margin=None, average_batch=True):\n '''\n Args:\n scores: image-sentence score matrix, (batch, batch)\n the same row of im and s are positive pairs, different rows are negative pairs\n '''\n if margin is None:\n margin = self.margin\n batch_size = scores.shape[0] \n diagonal = paddle.diagonal(scores).reshape([batch_size, 1])"
+ },
+ {
+ "comment": "This code segment calculates the cost for negative pairs in a contrastive learning task. It first creates masks to clear diagonal values, then compares each diagonal score with scores within its column or row (depending on direction), and applies a margin to create positive pairs. The cost is calculated based on the max violation method and averaged according to specific conditions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/loss.py\":61-84",
+ "content": " # mask to clear diagonals which are positive pairs\n pos_masks = paddle.eye(batch_size).astype('bool') \n batch_topk = min(batch_size, self.topk)\n if self.direction == 'i2t' or self.direction == 'bi':\n d1 = diagonal.expand_as(scores) # same collumn for im2s (negative sentence)\n # compare every diagonal score to scores in its collumn\n # caption retrieval\n cost_s = (margin + scores - d1).clip(min=0)\n cost_s[pos_masks] = 0 \n if self.max_violation:\n cost_s, _ = paddle.topk(cost_s, batch_topk, axis=1)\n cost_s = cost_s / batch_topk\n if average_batch:\n cost_s = cost_s / batch_size\n else:\n if average_batch:\n cost_s = cost_s / (batch_size * (batch_size - 1))\n cost_s = paddle.sum(cost_s)\n if self.direction == 't2i' or self.direction == 'bi':\n d2 = diagonal.t().expand_as(scores) # same row for s2im (negative image)\n # compare every diagonal score to scores in its row\n cost_im = (margin + scores - d2).clip(min=0)"
+ },
+ {
+ "comment": "This code calculates the video-level loss in a T2VLAD model. It first sets the positions of the correct matches to 0, then applies various scaling operations based on parameters. Finally, it sums the resulting cost and returns the appropriate value depending on the direction (i2t or t2i).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/loss.py\":85-101",
+ "content": " cost_im[pos_masks] = 0 \n if self.max_violation:\n cost_im, _ = paddle.topk(cost_im, batch_topk, axis=0)\n cost_im = cost_im / batch_topk\n if average_batch:\n cost_im = cost_im / batch_size\n else:\n if average_batch:\n cost_im = cost_im / (batch_size * (batch_size - 1))\n cost_im = paddle.sum(cost_im)\n if self.direction == 'i2t':\n return cost_s\n elif self.direction == 't2i':\n return cost_im\n else:\n return cost_s + cost_im"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/36691932-c5e5-450a-a5f5-f7d207b048e3.json b/docs/doc/36691932-c5e5-450a-a5f5-f7d207b048e3.json
new file mode 100644
index 000000000..26f4ed27b
--- /dev/null
+++ b/docs/doc/36691932-c5e5-450a-a5f5-f7d207b048e3.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The script prepares PaddleVideo for TimeSformer benchmarking, downloads UCF101 dataset, and performs batch experiments with various configurations on one or eight GPUs.",
+ "details": [
+ {
+ "comment": "This code snippet is part of a script for running the TimeSformer model benchmark in PaddleVideo. It sets up the environment, installs required dependencies, and copies necessary configuration files and data to ensure stable performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/benchmark/TimeSformer/run_all.sh\":0-19",
+ "content": "# \u63d0\u4f9b\u53ef\u7a33\u5b9a\u590d\u73b0\u6027\u80fd\u7684\u811a\u672c\uff0c\u9ed8\u8ba4\u5728\u6807\u51c6docker\u73af\u5883\u5185py37\u6267\u884c\uff1a paddlepaddle/paddle:latest-gpu-cuda10.2-cudnn7 paddle=2.1.2 py=37\n# \u6267\u884c\u76ee\u5f55\uff1a\u9700\u8bf4\u660e\nsed -i '/set\\ -xe/d' run_benchmark.sh\ncd ../../ # cd\u5230PaddleVideo\u9879\u76ee\u6839\u76ee\u5f55\u4e0b\ngit checkout benchmark_dev\nlog_path=${LOG_PATH_INDEX_DIR:-$(pwd)} # benchmark\u7cfb\u7edf\u6307\u5b9a\u8be5\u53c2\u6570,\u4e0d\u9700\u8981\u8dd1profile\u65f6,log_path\u6307\u5411\u5b58speed\u7684\u76ee\u5f55\n# 1 \u5b89\u88c5\u8be5\u6a21\u578b\u9700\u8981\u7684\u4f9d\u8d56 (\u5982\u9700\u5f00\u542f\u4f18\u5316\u7b56\u7565\u8bf7\u6ce8\u660e)\npython -m pip install -r requirements.txt\n# 2 \u62f7\u8d1d\u8be5\u6a21\u578b\u9700\u8981\u6570\u636e\u3001\u9884\u8bad\u7ec3\u6a21\u578b\nunalias cp\ncp -f benchmark/TimeSformer/timesformer_ucf101_videos_benchmark_bs1.yaml configs/recognition/timesformer/\ncp -f benchmark/TimeSformer/timesformer_ucf101_videos_benchmark_bs1_mp.yaml configs/recognition/timesformer/\ncp -f benchmark/TimeSformer/timesformer_ucf101_videos_benchmark_bs14.yaml configs/recognition/timesformer/\ncp -f benchmark/TimeSformer/timesformer_ucf101_videos_benchmark_bs14_mp.yaml configs/recognition/timesformer/\nif [ ! -f \"data/ucf101/trainlist_benchmark_mp.txt\" ]; then\n wget -P data/ucf101/ https://videotag.bj.bcebos.com/PaddleVideo-release2.2/trainlist_benchmark_mp.txt\nfi\nwget -P "
+ },
+ {
+ "comment": "This script downloads and prepares the UCF101 dataset for the TimeSformer model in PaddleVideo. It checks if the user wants to download data or use local data, then proceeds accordingly. The script also sets up a loop to run batch experiments with different model modes (TimeSformer), floating point items (fp32, fp16), and batch sizes (1). The log name is based on the model mode, run mode (speed), batch size, and floating point item. It uses CUDA_VISIBLE_DEVICES=0 to specify the GPU for execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/benchmark/TimeSformer/run_all.sh\":19-46",
+ "content": "data/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ViT_base_patch16_224_pretrained.pdparams\nalias cp='cp -i'\ncd data/ucf101 # \u8fdb\u5165PaddleVideo/data/ucf101\nif [ $1 = \"down_data\" ];then\n wget --no-check-certificate \"https://www.crcv.ucf.edu/data/UCF101/UCF101.rar\" # \u4e0b\u8f7d\u8bad\u7ec3\u6570\u636e\n unrar x UCF101.rar # \u89e3\u538b\n mv ./UCF-101 ./videos # \u91cd\u547d\u540d\u6587\u4ef6\u5939\u4e3a./videos\n rm -rf ./UCF101.rar\nelse # \u4f7f\u7528\u672c\u5730\u6570\u636e\n rm -rf videos\n ln -s ${data_path}/dygraph_data/TSM/ucf101/videos ./videos\nfi\ncd ../../ # \u8fd4\u56dePaddleVideo\n# 3 \u6279\u91cf\u8fd0\u884c\uff08\u5982\u4e0d\u65b9\u4fbf\u6279\u91cf\uff0c1\uff0c2\u9700\u653e\u5230\u5355\u4e2a\u6a21\u578b\u4e2d\uff09\nmodel_mode_list=(TimeSformer)\nfp_item_list=(fp32 fp16)\nbs_item_list=(1) # 14\nfor model_mode in ${model_mode_list[@]}; do\n for fp_item in ${fp_item_list[@]}; do\n for bs_item in ${bs_item_list[@]}\n do\n run_mode=sp\n log_name=video_${model_mode}_${run_mode}_bs${bs_item}_${fp_item} # \u5982:clas_MobileNetv1_mp_bs32_fp32_8\n echo \"index is speed, 1gpus, begin, ${log_name}\"\n CUDA_VISIBLE_DEVICES=0 bash benchmark/${model_m"
+ },
+ {
+ "comment": "The script iterates over different model modes, batch sizes, and floating-point types. It first runs a benchmark with one GPU and logs the results, then sleeps for 60 seconds. Next, it repeats the process but uses eight GPUs in parallel. The script aims to test various configurations and collect performance data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/benchmark/TimeSformer/run_all.sh\":46-56",
+ "content": "ode}/run_benchmark.sh ${run_mode} ${bs_item} ${fp_item} ${model_mode} | tee ${log_path}/${log_name}_speed_1gpus 2>&1\n sleep 60\n run_mode=mp\n log_name=video_${model_mode}_${run_mode}_bs${bs_item}_${fp_item} # \u5982:clas_MobileNetv1_mp_bs32_fp32_8\n echo \"index is speed, 8gpus, run_mode is multi_process, begin, ${log_name}\"\n CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash benchmark/${model_mode}/run_benchmark.sh ${run_mode} ${bs_item} ${fp_item} ${model_mode} | tee ${log_path}/${log_name}_speed_8gpus8p 2>&1\n sleep 60\n done\n done\ndone"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/367d02ef-7ba3-457b-9be2-0fe49a2bf272.json b/docs/doc/367d02ef-7ba3-457b-9be2-0fe49a2bf272.json
new file mode 100644
index 000000000..7a578bc30
--- /dev/null
+++ b/docs/doc/367d02ef-7ba3-457b-9be2-0fe49a2bf272.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code provides instructions to detect UAVs in restricted areas using PaddleDetection, with data preparation and dependency installation steps. Users can customize the configuration file and trained model for specific use cases.",
+ "details": [
+ {
+ "comment": "This code is for the Paddle-Anti-UAV application that uses PaddleDetection to detect flying UAVs in restricted areas. It provides details on data preparation, where to download and unzip the dataset, and how to install PaddleDetection.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Anti-UAV/README.md\":0-20",
+ "content": "# Paddle-Anti-UAV\nAnti-UAV base on PaddleDetection\n## Background\nUAVs are very popular and we can see them in many public spaces, such as parks and playgrounds. Most people use UAVs for taking photos.\nHowever, many areas like airport forbiden UAVs since they are potentially dangerous. In this case, we need to detect the flying UAVs in\nthese areas.\nIn this repository, we show how to train a detection model using [PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection).\n## Data preparation\nThe dataset can be found [here](https://anti-uav.github.io/dataset/). We direcly download the ```test-dev``` split composed of 140 videos\ntrain the detection model.\n* Download the ```test-dev``` dataset.\n* Run `unzip Anti_UAV_test_dev.zip -d Anti_UAV`.\n* Run `python get_image_label.py`. In this step, you may change the path to the videos and the value of `interval`.\nAfter the above steps, you will get a MSCOCO-style datasst for object detection.\n## Install PaddleDetection\nPlease refer to this [link](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.3/docs/tutorials/INSTALL.md)."
+ },
+ {
+ "comment": "The code outlines the process to train and use PP-YOLO for UAV detection using PaddleDetection in a specific environment. It involves cloning a repository, moving dataset files, adjusting configurations, and running training and inference commands with specific arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Anti-UAV/README.md\":22-35",
+ "content": "We use `python=3.7`, `Paddle=2.2.1`, `CUDA=10.2`.\n## Train PP-YOLO\nWe use [PP-YOLO](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.3/configs/ppyolo) as the detector.\n* Run `git clone https://github.com/PaddlePaddle/PaddleDetection.git`. Note that you should finish this step when you install PaddleDetection.\n* Move the anti-UAV dataset to `dataset`.\n* Move `anti_uav.yml` to `configs/datasets`, move `ppyolo_r50vd_dcn_1x_antiuav.yml` to `configs/ppyolo` and move `ppyolo_r50vd_dcn_antiuav.yml`\nto `configs/ppyolo/_base`.\n* Keep the value of `anchors` in `configs/ppyolo/_base/ppyolo_reader.yml` the same as `ppyolo_r50vd_dcn_antiuav.yml`.\n* Run `python -m paddle.distributed.launch --log_dir=./ppyolo_dygraph/ --gpus 0,1,2,3,4,5,6,7 tools/train.py -c configs/ppyolo/ppyolo_r50vd_dcn_1x_antiuav.yml &>ppyolo_dygraph.log 2>&1 &`.\nNote that you may change the arguments, such as `batch_size` and `gups`.\n## Inference\nPlease refer to the infernce section on this [webpage](https://github.com/Paddle"
+ },
+ {
+ "comment": "The code snippet is referring to the README file of an anti-UAV application based on PaddleVideo. It demonstrates two GIFs showing the demo in action and mentions that users can customize the configuration file and trained model for their own use cases.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Anti-UAV/README.md\":35-38",
+ "content": "Paddle/PaddleDetection/blob/release/2.3/docs/tutorials/GETTING_STARTED.md). You can just switch the configeration file and trained model to your own files.\n\n"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3699dc05-7c89-49b1-93b8-78e4f66ac812.json b/docs/doc/3699dc05-7c89-49b1-93b8-78e4f66ac812.json
new file mode 100644
index 000000000..9bd1745ee
--- /dev/null
+++ b/docs/doc/3699dc05-7c89-49b1-93b8-78e4f66ac812.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code sets up benchmark tests for TimeSformer video classification models in PaddleVideo, allowing users to customize parameters and analyze logs. The train() function is used for model training with specified parameters.",
+ "details": [
+ {
+ "comment": "This script is a bash file for running benchmark tests on TimeSformer video classification models. It sets parameters such as single or multi-GPU mode, batch size, floating point precision, and model item. The function _train() will be used to train the model with specified parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/benchmark/TimeSformer/run_benchmark.sh\":0-27",
+ "content": "#!/usr/bin/env bash\nset -xe\n# \u8fd0\u884c\u793a\u4f8b\uff1aCUDA_VISIBLE_DEVICES=0 bash run_benchmark.sh ${run_mode} ${bs_item} ${fp_item} 500 ${model_mode}\n# \u53c2\u6570\u8bf4\u660e\nfunction _set_params(){\n run_mode=${1:-\"sp\"} # \u5355\u5361sp|\u591a\u5361mp\n batch_size=${2:-\"1\"}\n fp_item=${3:-\"fp32\"} # fp32|fp16\n model_item=${4:-\"model_item\"}\n run_log_path=${TRAIN_LOG_DIR:-$(pwd)} # TRAIN_LOG_DIR \u540e\u7eedQA\u8bbe\u7f6e\u8be5\u53c2\u6570\n# \u6dfb\u52a0benchmark\u65e5\u5fd7\u89e3\u6790\u6240\u9700\u53c2\u6570\n base_batch_size=${batch_size}\n mission_name=\"\u89c6\u9891\u5206\u7c7b\"\n direction_id=\"0\"\n ips_unit=\"instance/sec\"\n skip_steps=10 # \u89e3\u6790\u65e5\u5fd7\uff0c\u6709\u4e9b\u6a21\u578b\u524d\u51e0\u4e2astep\u8017\u65f6\u957f\uff0c\u9700\u8981\u8df3\u8fc7 (\u5fc5\u586b)\n keyword=\"ips:\" # \u89e3\u6790\u65e5\u5fd7\uff0c\u7b5b\u9009\u51fa\u6570\u636e\u6240\u5728\u884c\u7684\u5173\u952e\u5b57 (\u5fc5\u586b)\n index=\"1\"\n model_name=${model_item}_bs${batch_size}_${fp_item}\n# \u4ee5\u4e0b\u4e0d\u7528\u4fee\u6539 \n device=${CUDA_VISIBLE_DEVICES//,/ }\n arr=(${device})\n num_gpu_devices=${#arr[*]}\n log_file=${run_log_path}/${model_item}_${run_mode}_bs${batch_size}_${fp_item}_${num_gpu_devices}\n}\nfunction _train(){\n echo \"Train on ${num_gpu_devices} GPUs\""
+ },
+ {
+ "comment": "This code is running a benchmark for the TimeSformer model in PaddleVideo. It checks if fp_item is either 'fp32' or 'fp16', and then calls the main script with the appropriate configuration file, based on the mode (sp or mp) and the chosen precision. The output logs are directed to a specified directory for analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/benchmark/TimeSformer/run_benchmark.sh\":28-47",
+ "content": " echo \"current CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES, gpus=$num_gpu_devices, batch_size=$batch_size\"\n case ${run_mode} in\n sp) \n if [ ${fp_item} == 'fp32' ]; then\n train_cmd=\"python -u main.py -c configs/recognition/timesformer/timesformer_ucf101_videos_benchmark_bs${batch_size}.yaml\"\n elif [ ${fp_item} == 'fp16' ]; then\n train_cmd=\"python -u main.py --amp -c configs/recognition/timesformer/timesformer_ucf101_videos_benchmark_bs${batch_size}.yaml\"\n else\n echo \"choose fp_item(fp32 or fp16)\"\n exit 1\n fi;;\n mp)\n rm -rf ./mylog\n if [ ${fp_item} == 'fp32' ]; then\n train_cmd=\"python -u -B -m paddle.distributed.launch --gpus=$CUDA_VISIBLE_DEVICES --log_dir=./mylog main.py \\\n -c configs/recognition/timesformer/timesformer_ucf101_videos_benchmark_bs${batch_size}_mp.yaml\"\n log_parse_file=\"mylog/workerlog.0\"\n elif [ ${fp_item} == 'fp16' ]; then\n train_cmd=\"python -u -B -m paddle.distributed.launch --gpus=$CUDA_VISIBLE_DEVICES --log_dir=./mylog main.py --amp \\"
+ },
+ {
+ "comment": "This code is part of a shell script for benchmarking the TimeSformer model. It sets up the command to run the training and checks for specific parameters like batch size and precision. It then executes the command with timeout, logs the result as success or failure, and removes intermediate log files if running in multi-process mode. The script also sources a separate file for further analysis of the log data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/benchmark/TimeSformer/run_benchmark.sh\":48-76",
+ "content": " -c configs/recognition/timesformer/timesformer_ucf101_videos_benchmark_bs${batch_size}_mp.yaml\"\n log_parse_file=\"mylog/workerlog.0\"\n else\n echo \"choose fp_item(fp32 or fp16)\"\n exit 1\n fi;;\n *) echo \"choose run_mode(sp or mp)\"; exit 1;\n esac\n# \u4ee5\u4e0b\u4e0d\u7528\u4fee\u6539\n timeout 15m ${train_cmd} > ${log_file} 2>&1\n if [ $? -ne 0 ];then\n echo -e \"${model_name}, FAIL\"\n export job_fail_flag=1\n else\n echo -e \"${model_name}, SUCCESS\"\n export job_fail_flag=0\n fi\n kill -9 `ps -ef|grep 'python'|awk '{print $2}'`\n if [ $run_mode = \"mp\" -a -d mylog ]; then\n rm ${log_file}\n cp mylog/workerlog.0 ${log_file}\n fi\n}\nsource ${BENCHMARK_ROOT}/scripts/run_model.sh # \u5728\u8be5\u811a\u672c\u4e2d\u4f1a\u5bf9\u7b26\u5408benchmark\u89c4\u8303\u7684log\u4f7f\u7528analysis.py \u811a\u672c\u8fdb\u884c\u6027\u80fd\u6570\u636e\u89e3\u6790;\u8be5\u811a\u672c\u5728\u8fde\u8c03\u65f6\u53ef\u4ecebenchmark repo\u4e2d\u4e0b\u8f7dhttps://github.com/PaddlePaddle/benchmark/blob/master/scripts/run_model.sh;\u5982\u679c\u4e0d\u8054\u8c03\u53ea\u60f3\u8981\u4ea7\u51fa\u8bad\u7ec3log\u53ef\u4ee5\u6ce8\u6389\u672c\u884c,\u63d0\u4ea4\u65f6\u9700\u6253\u5f00\n_set_params $@\n# _train # \u5982\u679c\u53ea\u60f3\u4ea7\u51fa\u8bad\u7ec3log,\u4e0d\u89e3\u6790,\u53ef\u53d6\u6d88\u6ce8\u91ca\n_run # \u8be5\u51fd\u6570\u5728run_model.sh\u4e2d,\u6267\u884c\u65f6\u4f1a\u8c03\u7528_train; \u5982\u679c\u4e0d\u8054\u8c03\u53ea\u60f3\u8981\u4ea7\u51fa\u8bad\u7ec3log\u53ef\u4ee5\u6ce8\u6389\u672c\u884c,\u63d0\u4ea4\u65f6\u9700\u6253\u5f00"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/36a13f58-7b2d-47c3-8431-d877fccb0deb.json b/docs/doc/36a13f58-7b2d-47c3-8431-d877fccb0deb.json
new file mode 100644
index 000000000..9aba063e7
--- /dev/null
+++ b/docs/doc/36a13f58-7b2d-47c3-8431-d877fccb0deb.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a Python class for the UCF101 Skeleton Dataset in PaddleVideo, loading skeleton features and normalizing data for action recognition tasks. The dataset includes train and test methods for preparing frames with `prepare_train` and `prepare_test` functions.",
+ "details": [
+ {
+ "comment": "This code snippet is a Python class for UCF101 Skeleton Dataset in PaddleVideo. It loads skeleton features and applies normalization operations, registering the dataset for action recognition tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/ucf101_skeleton.py\":0-34",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport pickle\nimport paddle\nfrom paddle.io import Dataset\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass UCF101SkeletonDataset(BaseDataset):\n \"\"\"\n Skeleton dataset for action recognition.\n The dataset loads skeleton feature, and apply norm operatations."
+ },
+ {
+ "comment": "This code defines a class that loads annotation data from a file, specifically for the UCF101 dataset's skeleton information. It takes arguments such as the file path, pipeline object, and whether it's building a test dataset. The load_file method checks if the file is a .pkl file and calls load_pkl_annotations to get video information. If the split argument is provided, it only uses the specified part of the data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/ucf101_skeleton.py\":35-65",
+ "content": " Args:\n file_path (str): Path to the index file.\n pipeline(obj): Define the pipeline of data preprocessing.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n \"\"\"\n def __init__(self,\n file_path,\n pipeline,\n split,\n repeat_times,\n test_mode=False):\n self.split = split\n self.repeat_times = repeat_times\n super().__init__(file_path, pipeline, test_mode=test_mode)\n self._ori_len = len(self.info)\n self.start_index = 0\n self.modality = \"Pose\"\n def load_file(self):\n \"\"\"Load annotation file to get video information.\"\"\"\n assert self.file_path.endswith('.pkl')\n return self.load_pkl_annotations()\n def load_pkl_annotations(self):\n with open(self.file_path, \"rb\") as f:\n data = pickle.load(f)\n if self.split:\n split, data = data['split'], data['annotations']\n identifier = 'filename' if 'filename' in data[0] else 'frame_dir'"
+ },
+ {
+ "comment": "This code defines a dataset for PaddleVideo, containing train and test methods for preparing frames. The `prepare_train` and `prepare_test` functions create new results by copying the original information, setting modality and start index based on the given index. The `__len__` function returns the size of the dataset by multiplying the number of info items with repeat times.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/ucf101_skeleton.py\":66-88",
+ "content": " data = [x for x in data if x[identifier] in split[self.split]]\n return data\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training given the index.\"\"\"\n results = copy.deepcopy(self.info[idx % self._ori_len])\n results['modality'] = self.modality\n results['start_index'] = self.start_index\n return self.pipeline(results)\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for testing given the index.\"\"\"\n results = copy.deepcopy(self.info[idx % self._ori_len])\n results['modality'] = self.modality\n results['start_index'] = self.start_index\n return self.pipeline(results)\n def __len__(self):\n \"\"\"get the size of the dataset.\"\"\"\n return len(self.info) * self.repeat_times"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/37739699-d923-4eae-b76e-e27b23c261f6.json b/docs/doc/37739699-d923-4eae-b76e-e27b23c261f6.json
new file mode 100644
index 000000000..446c7ea23
--- /dev/null
+++ b/docs/doc/37739699-d923-4eae-b76e-e27b23c261f6.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The Bash script configures a model serving environment, sets up an API server, transfers the model using provided Python code, and handles cleanup tasks. It also adjusts alias names, logs paths, and CUDA visible devices while running video processing pipeline tests.",
+ "details": [
+ {
+ "comment": "This Bash script is parsing a configuration file and extracting various parameters for running model inference. It assigns values to variables such as 'model_name', 'python_list', and others that will be used later in the code. The purpose is to set up an environment for serving the model and potentially run inferences on videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_serving_infer_python.sh\":0-28",
+ "content": "#!/bin/bash\nsource test_tipc/common_func.sh\nFILENAME=$1\ndataline=$(awk 'NR==1, NR==18{print}' $FILENAME)\nMODE=$2\n# parser params\nIFS=$'\\n'\nlines=(${dataline})\n# parser serving\nmodel_name=$(func_parser_value \"${lines[1]}\")\npython_list=$(func_parser_value \"${lines[2]}\")\ntrans_model_py=$(func_parser_value \"${lines[3]}\")\ninfer_model_dir_key=$(func_parser_key \"${lines[4]}\")\ninfer_model_dir_value=$(func_parser_value \"${lines[4]}\")\nmodel_filename_key=$(func_parser_key \"${lines[5]}\")\nmodel_filename_value=$(func_parser_value \"${lines[5]}\")\nparams_filename_key=$(func_parser_key \"${lines[6]}\")\nparams_filename_value=$(func_parser_value \"${lines[6]}\")\nserving_server_key=$(func_parser_key \"${lines[7]}\")\nserving_server_value=$(func_parser_value \"${lines[7]}\")\nserving_client_key=$(func_parser_key \"${lines[8]}\")\nserving_client_value=$(func_parser_value \"${lines[8]}\")\nserving_dir_value=$(func_parser_value \"${lines[9]}\")\nweb_service_py=$(func_parser_value \"${lines[10]}\")\npipeline_py=$(func_parser_value \"${lines[11]}\")\nvideo_dir_key=$(func_parser_key \"${lines[12]}\")"
+ },
+ {
+ "comment": "The code defines a function `func_serving` that takes Python executable path, script, model directory, and sets various parameters for serving. It then executes a command to transfer the model to the specified server or client using a provided Python script. The output is logged in the `trans_log` file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_serving_infer_python.sh\":29-53",
+ "content": "video_dir_value=$(func_parser_value \"${lines[12]}\")\nLOG_PATH=\"./test_tipc/output/log/${model_name}/${MODE}\"\nmkdir -p ${LOG_PATH}\nstatus_log=\"${LOG_PATH}/results_serving.log\"\nfunction func_serving(){\n IFS='|'\n _python=$1\n _script=$2\n _model_dir=$3\n # python serving code\n set_dirname=$(func_set_params \"${infer_model_dir_key}\" \"${infer_model_dir_value}\")\n set_model_filename=$(func_set_params \"${model_filename_key}\" \"${model_filename_value}\")\n set_params_filename=$(func_set_params \"${params_filename_key}\" \"${params_filename_value}\")\n set_serving_server=$(func_set_params \"${serving_server_key}\" \"${serving_server_value}\")\n set_serving_client=$(func_set_params \"${serving_client_key}\" \"${serving_client_value}\")\n python_list=(${python_list})\n python=${python_list[0]}\n trans_log=\"${LOG_PATH}/python_trans_model.log\"\n trans_model_cmd=\"${python} ${trans_model_py} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_serving_server} ${set_serving_client} > ${trans_log} 2>&1 \""
+ },
+ {
+ "comment": "This code modifies alias names in configuration files, sets log paths and starts a web service using Python. It also checks the status of the service, sleeps for 30 seconds, and saves logs into a specific path. The code is executed within a specific directory and sets environment variables before running the commands.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_serving_infer_python.sh\":55-76",
+ "content": " eval ${trans_model_cmd}\n # modify the alias name of fetch_var to \"outputs\"\n server_fetch_var_line_cmd=\"sed -i '/fetch_var/,/is_lod_tensor/s/alias_name: .*/alias_name: \\\"outputs\\\"/' $serving_server_value/serving_server_conf.prototxt\"\n eval ${server_fetch_var_line_cmd}\n client_fetch_var_line_cmd=\"sed -i '/fetch_var/,/is_lod_tensor/s/alias_name: .*/alias_name: \\\"outputs\\\"/' $serving_client_value/serving_client_conf.prototxt\"\n eval ${client_fetch_var_line_cmd}\n cd ${serving_dir_value}\n echo 'PWD= '$PWD\n unset https_proxy\n unset http_proxy\n server_log_path=\"${LOG_PATH}/python_server_gpu.log\"\n web_service_cmd=\"${python} ${web_service_py} > ${server_log_path} 2>&1 &\"\n eval $web_service_cmd\n last_status=${PIPESTATUS[0]}\n status_check $last_status \"${web_service_cmd}\" \"${status_log}\" \"${model_name}\"\n sleep 30s # not too short is ok\n _save_log_path=\"../../${LOG_PATH}/python_server_infer_gpu_batchsize_1.log\"\n set_video_dir=$(func_set_params \"${video_dir_key}\" \"${video_dir_value}\")"
+ },
+ {
+ "comment": "This code is setting up the environment and running a test for a video processing pipeline. It sets the CUDA visible devices, runs the test using specified command, and performs clean-up by killing related processes after the test.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_serving_infer_python.sh\":77-104",
+ "content": " pipeline_cmd=\"${python} ${pipeline_py} ${set_video_dir} > ${_save_log_path} 2>&1 \"\n eval $pipeline_cmd\n last_status=${PIPESTATUS[0]}\n eval \"cat ${_save_log_path}\"\n cd ../../\n status_check $last_status \"${pipeline_cmd}\" \"${status_log}\" \"${model_name}\"\n ps ux | grep -E 'web_service|pipeline' | awk '{print $2}' | xargs kill -s 9\n}\n# set cuda device\nGPUID=$3\nif [ ${#GPUID} -le 0 ];then\n env=\" \"\nelse\n env=\"export CUDA_VISIBLE_DEVICES=${GPUID}\"\nfi\nset CUDA_VISIBLE_DEVICES\neval $env\necho \"################### run test ###################\"\nexport Count=0\nIFS=\"|\"\nfunc_serving \"${web_service_cmd}\""
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/37ab8cf2-e1d0-4276-9749-038da87d02f7.json b/docs/doc/37ab8cf2-e1d0-4276-9749-038da87d02f7.json
new file mode 100644
index 000000000..6bbac7992
--- /dev/null
+++ b/docs/doc/37ab8cf2-e1d0-4276-9749-038da87d02f7.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The code defines global parameters for the VGGish model, including architectural constants, hyperparameters, and optimizer settings. It extracts audio features from spectrogram patches using PCA quantization and embedding processing, with options to adjust STFT window and hop lengths, mel frequency bins, and learning rate.",
+ "details": [
+ {
+ "comment": "This code sets global parameters for the VGGish model. It defines architectural constants, hyperparameters for feature and example generation, embedding postprocessing, and training. The VGGish model is used to extract audio features from spectrogram patches, with options for PCA-based quantization and embedding processing. Hyperparameters control the STFT window and hop lengths, mel frequency bins, and learning rate for Adam optimizer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/mfcc/vgg_params.py\":0-28",
+ "content": "\"\"\"Global parameters for the VGGish model.\nSee vggish_slim.py for more information.\n\"\"\"\n# Architectural constants.\nNUM_FRAMES = 50 # Frames in input mel-spectrogram patch.\nNUM_BANDS = 64 # Frequency bands in input mel-spectrogram patch.\nEMBEDDING_SIZE = 128 # Size of embedding layer.\n# Hyperparameters used in feature and example generation.\nSAMPLE_RATE = 16000\nSTFT_WINDOW_LENGTH_SECONDS = 0.040\nSTFT_HOP_LENGTH_SECONDS = 0.020\nNUM_MEL_BINS = NUM_BANDS\nMEL_MIN_HZ = 125\nMEL_MAX_HZ = 7500\nLOG_OFFSET = 0.01 # Offset used for stabilized log of input mel-spectrogram.\nEXAMPLE_WINDOW_SECONDS = 1.00 # Each example contains 96 10ms frames\nEXAMPLE_HOP_SECONDS = 1.00 # with zero overlap.\n# Parameters used for embedding postprocessing.\nPCA_EIGEN_VECTORS_NAME = 'pca_eigen_vectors'\nPCA_MEANS_NAME = 'pca_means'\nQUANTIZE_MIN_VAL = -2.0\nQUANTIZE_MAX_VAL = +2.0\n# Hyperparameters used in training.\nINIT_STDDEV = 0.01 # Standard deviation used to initialize weights.\nLEARNING_RATE = 1e-4 # Learning rate for the Adam optimizer."
+ },
+ {
+ "comment": "This code sets the Adam optimizer's epsilon value to 1e-8, defines names for input and output operations, tensors, and features. It also assigns the name \"audio_embedding\" to a feature.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/mfcc/vgg_params.py\":29-36",
+ "content": "ADAM_EPSILON = 1e-8 # Epsilon for the Adam optimizer.\n# Names of ops, tensors, and features.\nINPUT_OP_NAME = 'vggish/input_features'\nINPUT_TENSOR_NAME = INPUT_OP_NAME + ':0'\nOUTPUT_OP_NAME = 'vggish/embedding'\nOUTPUT_TENSOR_NAME = OUTPUT_OP_NAME + ':0'\nAUDIO_EMBEDDING_FEATURE_NAME = 'audio_embedding'"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/382773ae-ed48-48ec-bf7c-7d0ca872f997.json b/docs/doc/382773ae-ed48-48ec-bf7c-7d0ca872f997.json
new file mode 100644
index 000000000..bb6e20e68
--- /dev/null
+++ b/docs/doc/382773ae-ed48-48ec-bf7c-7d0ca872f997.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This Python script loads Baidu Cloud models for video classification, extracts audio and pcm features, logs details, saves features in a pickle file, creates \"features\" directory if necessary, and classifies videos from a specified dataset directory.",
+ "details": [
+ {
+ "comment": "This Python script is for the Baidu Cloud action, loading and initializing image and audio models according to a given configuration file. It also provides a function to classify videos. The script logs information about model loading time and the progress of video classification.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/extractor/extract_feat.py\":0-49",
+ "content": "#!./python27-gcc482/bin/python\n# coding: utf-8\n\"\"\"\nBAIDU CLOUD action\n\"\"\"\nimport os\nimport sys\nimport pickle\nimport json\nimport time\nimport shutil\nimport numpy as np\nsys.path.append(\"../predict/action_detect\")\nimport models.pptsm_infer as image_model\nimport models.audio_infer as audio_model\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config, print_configs\nimport utils.config_utils as config_utils\nimport logger\nlogger = logger.Logger()\ndef load_model(cfg_file=\"configs/configs.yaml\"):\n \"\"\"\n load_model\n \"\"\"\n logger.info(\"load model ... \")\n global infer_configs\n infer_configs = parse_config(cfg_file)\n print_configs(infer_configs, \"Infer\")\n t0 = time.time()\n global image_model, audio_model\n image_model = image_model.InferModel(infer_configs)\n audio_model = audio_model.InferModel(infer_configs)\n t1 = time.time()\n logger.info(\"step0: load model time: {} min\\n\".format((t1 - t0) * 1.0 / 60))\ndef video_classify(video_name):\n \"\"\"\n extract_feature\n \"\"\"\n logger.info('predict ... ')"
+ },
+ {
+ "comment": "Extracting video features, specifically images and audio. Converting extracted features to numpy arrays. Logging shapes of the arrays and time taken for feature extraction.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/extractor/extract_feat.py\":50-73",
+ "content": " logger.info(video_name)\n imgs_path = video_name.replace(\".mp4\", \"\").replace(\"mp4\", \"frames\")\n pcm_path = video_name.replace(\".mp4\", \".pcm\").replace(\"mp4\", \"pcm\")\n # step 1: extract feature\n t0 = time.time()\n image_path_list = get_images(imgs_path)\n infer_configs['PPTSM']['frame_list'] = image_path_list\n infer_configs['AUDIO']['pcm_file'] = pcm_path\n image_features = image_model.predict(infer_configs)\n audio_features, pcm_features = audio_model.predict(infer_configs)\n np_image_features = np.array(image_features, dtype=np.float32)\n np_audio_features = np.array(audio_features, dtype=np.float32)\n np_pcm_features = np.array(pcm_features, dtype=np.float32)\n t1 = time.time()\n logger.info('{} {} {}'.format(np_image_features.shape,\n np_audio_features.shape,\n np_pcm_features.shape))\n logger.info(\"step1: feature extract time: {} min\".format(\n (t1 - t0) * 1.0 / 60))\n video_features = {\n 'image_feature': np_image_features,"
+ },
+ {
+ "comment": "The code extracts audio and pcm features from video files, saves them in a pickle file named after the original video, creates a \"features\" directory if it doesn't exist, then classifies each video based on its location in a specified dataset directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/extractor/extract_feat.py\":74-99",
+ "content": " 'audio_feature': np_audio_features,\n 'pcm_feature': np_pcm_features\n }\n # save feature\n feature_path = video_name.replace(\".mp4\", \".pkl\").replace(\"mp4\", \"features\")\n feat_pkl_str = pickle.dumps(video_features,\n protocol=pickle.HIGHEST_PROTOCOL)\n with open(feature_path, 'wb') as fout:\n fout.write(feat_pkl_str)\nif __name__ == '__main__':\n dataset_dir = \"../datasets/EuroCup2016\"\n if not os.path.exists(dataset_dir + '/features'):\n os.mkdir(dataset_dir + '/features')\n load_model()\n video_url = os.path.join(dataset_dir, 'url.list')\n with open(video_url, 'r') as f:\n lines = f.readlines()\n lines = [os.path.join(dataset_dir, k.strip()) for k in lines]\n for line in lines:\n video_classify(line)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3868dce9-c7e8-4031-a111-90b800d1f64e.json b/docs/doc/3868dce9-c7e8-4031-a111-90b800d1f64e.json
new file mode 100644
index 000000000..8dd782b66
--- /dev/null
+++ b/docs/doc/3868dce9-c7e8-4031-a111-90b800d1f64e.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a dataset class for loading video information, requires subclassing to define load_file and prepare_train/test methods, prepares data for training/testing, and addresses DataLoader's dict type handling limitation.",
+ "details": [
+ {
+ "comment": "Base class for datasets, subclass it. Subclasses should overwrite load_file (load info from index file), prepare_train (provide train data), and prepare_test (provide test data).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/dataset/base.py\":0-33",
+ "content": "\"\"\"\n# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport os.path as osp\nimport copy\nimport numpy as np\nfrom abc import ABC, abstractmethod\nimport paddle\nfrom paddle.io import Dataset\nclass BaseDataset(Dataset, ABC):\n \"\"\"Base class for datasets\n All datasets should subclass it.\n All subclass should overwrite:\n - Method: `load_file`, load info from index file.\n - Method: `prepare_train`, providing train data.\n - Method: `prepare_test`, providing test data."
+ },
+ {
+ "comment": "This code defines a class for loading video information from an index file path. It takes arguments such as the file_path, pipeline, data_prefix, and test_mode. The load_file method abstractly loads the video information from the index file. The prepare_train method prepares data for training/valid given the index. Note: DataLoader cannot support dict type retval, so it converts to list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/dataset/base.py\":35-61",
+ "content": " Args:\n file_path (str): index file path.\n pipeline (Sequence XXX)\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): whether to build test dataset. Default: False.\n \"\"\"\n def __init__(self, file_path, pipeline, data_prefix=None, test_mode=False):\n super().__init__()\n self.file_path = file_path\n self.data_prefix = osp.realpath(data_prefix) if \\\n data_prefix is not None and osp.isdir(data_prefix) else data_prefix\n self.test_mode = test_mode\n self.pipeline = pipeline\n self.info = self.load_file()\n @abstractmethod\n def load_file(self):\n \"\"\"load the video information from the index file path.\"\"\"\n pass\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n #Note: For now, paddle.io.DataLoader cannot support dict type retval, so convert to list here\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)"
+ },
+ {
+ "comment": "The code defines a dataset class with methods for preparing data for training and testing, as well as returning the size of the dataset. The test_mode flag is used to determine whether to use the prepare_test or prepare_train method when accessing the dataset. Paddle.io.DataLoader cannot currently handle dict type return values, so they are converted to lists within these methods.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/dataset/base.py\":62-82",
+ "content": " #unsqueeze label to list\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"TEST: Prepare the data for test given the index.\"\"\"\n #Note: For now, paddle.io.DataLoader cannot support dict type retval, so convert to list here\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n #unsqueeze label to list\n return results['imgs'], np.array([results['labels']])\n def __len__(self):\n \"\"\"get the size of the dataset.\"\"\"\n return len(self.info)\n def __getitem__(self, idx):\n \"\"\" Get the sample for either training or testing given index\"\"\"\n if self.test_mode:\n return self.prepare_test(idx)\n else:\n return self.prepare_train(idx)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/386fe09e-4bc7-49c0-96ec-e7f0c2a9ba17.json b/docs/doc/386fe09e-4bc7-49c0-96ec-e7f0c2a9ba17.json
new file mode 100644
index 000000000..6dfe2d467
--- /dev/null
+++ b/docs/doc/386fe09e-4bc7-49c0-96ec-e7f0c2a9ba17.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code file is a part of the PaddleVideo library and contains a function named \"build_metric\". It imports necessary modules, defines a metric registry, and provides a build function to construct metrics according to the specified configuration (cfg). The code is licensed under Apache License 2.0.",
+ "details": [
+ {
+ "comment": "This code file is a part of the PaddleVideo library and contains a function named \"build_metric\". It imports necessary modules, defines a metric registry, and provides a build function to construct metrics according to the specified configuration (cfg). The code is licensed under Apache License 2.0.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/build.py\":0-22",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .registry import METRIC\nfrom ..utils import build\ndef build_metric(cfg):\n \"\"\"build metric\"\"\"\n return build(cfg, METRIC)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/387159e6-5d8e-4b52-a71c-0d46a3d29883.json b/docs/doc/387159e6-5d8e-4b52-a71c-0d46a3d29883.json
new file mode 100644
index 000000000..f82b17c30
--- /dev/null
+++ b/docs/doc/387159e6-5d8e-4b52-a71c-0d46a3d29883.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This script sets environment variables for GPU usage and then runs the \"inference.py\" Python script from the \"scenario_lib\" directory, specifying a model name (AttentionLstmErnie), configuration file path (./conf/conf.txt), saving inference models path (inference_models_save), and output file for results (output.json).",
+ "details": [
+ {
+ "comment": "This script sets environment variables for GPU usage and then runs the \"inference.py\" Python script from the \"scenario_lib\" directory, specifying a model name (AttentionLstmErnie), configuration file path (./conf/conf.txt), saving inference models path (inference_models_save), and output file for results (output.json).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/inference.sh\":0-11",
+ "content": "# inference sh \nexport CUDA_VISIBLE_DEVICES=0\nexport FLAGS_eager_delete_tensor_gb=0.0\nexport FLAGS_sync_nccl_allreduce=1\nexport FLAGS_fast_eager_deletion_mode=1\nexport FLAGS_fraction_of_gpu_memory_to_use=0.5\nexport FLAGS_reallocate_gpu_memory_in_mb=0\nexport FLAGS_memory_fraction_of_eager_deletion=1\npython scenario_lib/inference.py --model_name=AttentionLstmErnie \\\n--config=./conf/conf.txt \\\n--save_inference_model=inference_models_save \\\n--output='output.json'"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/38e736e9-f0a8-46f8-9d49-1e3493acdee5.json b/docs/doc/38e736e9-f0a8-46f8-9d49-1e3493acdee5.json
new file mode 100644
index 000000000..2c252c417
--- /dev/null
+++ b/docs/doc/38e736e9-f0a8-46f8-9d49-1e3493acdee5.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code defines functions to build components of a computer vision model, including backbone, head, loss, recognizer, and a model builder that selects the appropriate builder based on framework type.",
+ "details": [
+ {
+ "comment": "The code is defining functions to build different components of a computer vision model. The `build_backbone`, `build_head`, and `build_loss` functions are used to build the backbone, head, and loss for the model respectively using the `build` function from the `utils` module. The `build_recognizer` function is used to build a recognizer component for the model using the specified framework key.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/builder.py\":0-35",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import BACKBONES, HEADS, LOSSES, RECOGNIZERS, LOCALIZERS\nfrom ..utils import build\ndef build_backbone(cfg):\n \"\"\"Build backbone.\"\"\"\n return build(cfg, BACKBONES)\ndef build_head(cfg):\n \"\"\"Build head.\"\"\"\n return build(cfg, HEADS)\ndef build_loss(cfg):\n \"\"\"Build loss.\"\"\"\n return build(cfg, LOSSES)\ndef build_recognizer(cfg):\n \"\"\"Build recognizer.\"\"\"\n return build(cfg, RECOGNIZERS, key='framework')"
+ },
+ {
+ "comment": "This code defines functions for building localizer and model, and a build_model function that selects the appropriate builder based on the framework type specified in the configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/builder.py\":38-51",
+ "content": "def build_localizer(cfg):\n \"\"\"Build localizer.\"\"\"\n return build(cfg, LOCALIZERS, key='framework')\ndef build_model(cfg):\n cfg_copy = cfg.copy()\n framework_type = cfg_copy.get('framework')\n if framework_type in RECOGNIZERS:\n return build_recognizer(cfg)\n elif framework_type in LOCALIZERS:\n return build_localizer(cfg)\n else:\n raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/398b9feb-19d9-447a-bdeb-ce6acf427f06.json b/docs/doc/398b9feb-19d9-447a-bdeb-ce6acf427f06.json
new file mode 100644
index 000000000..5f9e5a8c4
--- /dev/null
+++ b/docs/doc/398b9feb-19d9-447a-bdeb-ce6acf427f06.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code initializes a logger, defines functions for logging metrics such as loss and learning rate during training. It also creates a class for tracking various metrics with update method and logs batch metrics at specified batch IDs in log_batch function, while formatting log string with colors for clarity in video processing tasks.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and defines functions for logging metrics such as loss, learning rate during training. It also initializes a logger with the name \"paddlevideo\" and specifies the available classes or functions that can be accessed from this file. The build_record function takes a configuration file and creates an ordered dictionary of metrics to record based on the framework type specified in the configuration file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/record.py\":0-31",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom collections import OrderedDict\nimport paddle\nfrom .logger import coloring, get_logger\nlogger = get_logger(\"paddlevideo\")\n__all__ = ['AverageMeter', 'build_record', 'log_batch', 'log_epoch']\ndef build_record(cfg):\n framework_type = cfg.get('framework', '')\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"lr\", AverageMeter('lr', 'f', need_avg=False)),\n ]\n if 'Recognizer1D' in framework_type: #TODO: required specify str in framework"
+ },
+ {
+ "comment": "This code is part of a function that handles recording different metrics for various framework types. It appends specific metric names and instances of the AverageMeter class to the record_list depending on the framework type. If 'Recognizer' is in the framework type, it records 'top1' and 'top5' metrics. If 'FastRCNN' is present, it records a series of recall and precision metrics along with mAP@0.5IOU. The function continues with more conditions for different framework types after this snippet.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/record.py\":32-48",
+ "content": " record_list.append((\"hit_at_one\", AverageMeter(\"hit_at_one\", '.5f')))\n record_list.append((\"perr\", AverageMeter(\"perr\", '.5f')))\n record_list.append((\"gap\", AverageMeter(\"gap\", '.5f')))\n elif 'Recognizer' in framework_type:\n record_list.append((\"top1\", AverageMeter(\"top1\", '.5f')))\n record_list.append((\"top5\", AverageMeter(\"top5\", '.5f')))\n elif 'FastRCNN' in framework_type:\n record_list.append(\n (\"recall@thr=0.5\", AverageMeter(\"recall@thr=0.5\", '.5f')))\n record_list.append(\n (\"prec@thr=0.5\", AverageMeter(\"prec@thr=0.5\", '.5f')))\n record_list.append((\"recall@top3\", AverageMeter(\"recall@top3\", '.5f')))\n record_list.append((\"prec@top3\", AverageMeter(\"prec@top3\", '.5f')))\n record_list.append((\"recall@top5\", AverageMeter(\"recall@top5\", '.5f')))\n record_list.append((\"prec@top5\", AverageMeter(\"prec@top5\", '.5f')))\n record_list.append((\"mAP@0.5IOU\", AverageMeter(\"mAP@0.5IOU\", '.5f')))\n elif 'DepthEstimator' in cfg.framework:"
+ },
+ {
+ "comment": "This code defines a list of metrics to be tracked and an AverageMeter class that computes and stores the average and current value for each metric. The list includes various metrics like \"abs_rel\", \"sq_rel\", \"rmse\", \"rmse_log\", \"a1\", \"a2\", \"a3\", \"losses_day\", \"losses_night\", \"batch_time\", and \"reader_time\". The list is then converted to an OrderedDict.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/record.py\":49-71",
+ "content": " record_list.append((\"abs_rel\", AverageMeter(\"abs_rel\", '.5f')))\n record_list.append((\"sq_rel\", AverageMeter(\"sq_rel\", '.5f')))\n record_list.append((\"rmse\", AverageMeter(\"rmse\", '.5f')))\n record_list.append((\"rmse_log\", AverageMeter(\"rmse_log\", '.5f')))\n record_list.append((\"a1\", AverageMeter(\"a1\", '.5f')))\n record_list.append((\"a2\", AverageMeter(\"a2\", '.5f')))\n record_list.append((\"a3\", AverageMeter(\"a3\", '.5f')))\n record_list.append((\"losses_day\", AverageMeter(\"losses_day\", '.5f')))\n record_list.append(\n (\"losses_night\", AverageMeter(\"losses_night\", '.5f')))\n record_list.append((\"batch_time\", AverageMeter('batch_cost', '.5f')))\n record_list.append((\"reader_time\", AverageMeter('reader_cost', '.5f')))\n record_list = OrderedDict(record_list)\n return record_list\nclass AverageMeter(object):\n \"\"\"\n Computes and stores the average and current value\n \"\"\"\n def __init__(self, name='', fmt='f', need_avg=True):\n self.name = name"
+ },
+ {
+ "comment": "This code defines a class for tracking metrics such as sum, count, average, and total values. The `update` method allows updating the metric with a new value, while the `total`, `total_minute`, `mean`, and `value` properties retrieve the current metric value in different formats. The `log_batch` function logs batch metrics for a list of metrics at a specified batch ID.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/record.py\":72-112",
+ "content": " self.fmt = fmt\n self.need_avg = need_avg\n self.reset()\n def reset(self):\n \"\"\" reset \"\"\"\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def update(self, val, n=1):\n \"\"\" update \"\"\"\n if isinstance(val, paddle.Tensor):\n val = float(val)\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n @property\n def total(self):\n return '{self.name}_sum: {self.sum:{self.fmt}}'.format(self=self)\n @property\n def total_minute(self):\n return '{self.name}_sum: {s:{self.fmt}} min'.format(s=self.sum / 60,\n self=self)\n @property\n def mean(self):\n return '{self.name}_avg: {self.avg:{self.fmt}}'.format(\n self=self) if self.need_avg else ''\n @property\n def value(self):\n return '{self.name}: {self.val:{self.fmt}}'.format(self=self)\ndef log_batch(metric_list,\n batch_id,"
+ },
+ {
+ "comment": "This function logs epoch metrics and step information for a video processing task. It formats the log string with different colors for each section: epoch or iteration, step number, metric values, batch time, reader time, and ips (images per second). The logger outputs this formatted string to provide an informative summary of the task's progress.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/record.py\":113-140",
+ "content": " epoch_id,\n total_epoch,\n mode,\n ips,\n tot_step=None,\n max_iters=None):\n batch_cost = str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = str(metric_list['reader_time'].value) + ' sec,'\n metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].value)\n metric_str = ' '.join([str(v) for v in metric_values])\n if max_iters:\n epoch_str = \"iter:[{:>3d}/{:<3d}]\".format(tot_step, max_iters)\n else:\n epoch_str = \"epoch:[{:>3d}/{:<3d}]\".format(epoch_id, total_epoch)\n step_str = \"{:s} step:{:<4d}\".format(mode, batch_id)\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {}\".format(\n coloring(epoch_str, \"HEADER\") if batch_id == 0 else epoch_str,\n coloring(step_str, \"PURPLE\"), coloring(metric_str, 'OKGREEN'),\n coloring(batch_cost, \"OKGREEN\"), coloring(reader_cost, 'OKGREEN'),\n ips))\ndef log_epoch(metric_list, epoch, mode, ips):"
+ },
+ {
+ "comment": "This code is formatting and logging information at the end of an epoch. It calculates various metric values, constructs a formatted string with different colors, and then logs this information using logger.info(). The metrics include batch time, reader time, total cost, mode, and inference per second. The strings are color-coded for visual clarity.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/record.py\":141-156",
+ "content": " batch_cost = 'avg_' + str(metric_list['batch_time'].value) + ' sec,'\n reader_cost = 'avg_' + str(metric_list['reader_time'].value) + ' sec,'\n batch_sum = str(metric_list['batch_time'].total) + ' sec,'\n metric_values = []\n for m in metric_list:\n if not (m == 'batch_time' or m == 'reader_time'):\n metric_values.append(metric_list[m].mean)\n metric_str = ' '.join([str(v) for v in metric_values])\n end_epoch_str = \"END epoch:{:<3d}\".format(epoch)\n logger.info(\"{:s} {:s} {:s} {:s} {:s} {:s} {}\".format(\n coloring(end_epoch_str, \"RED\"), coloring(mode, \"PURPLE\"),\n coloring(metric_str, \"OKGREEN\"), coloring(batch_cost, \"OKGREEN\"),\n coloring(reader_cost, \"OKGREEN\"), coloring(batch_sum, \"OKGREEN\"), ips))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3a1c90c1-39ee-49d9-b34e-b8b77b32fbce.json b/docs/doc/3a1c90c1-39ee-49d9-b34e-b8b77b32fbce.json
new file mode 100644
index 000000000..1dd662775
--- /dev/null
+++ b/docs/doc/3a1c90c1-39ee-49d9-b34e-b8b77b32fbce.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code reads a file for model details, extracts lines with common functions, sets up inference directories and parameters, enables ONNX checker, converts using paddle2onnx, saves logs, runs inference, and checks status for the \"func_paddle2onnx\" function.",
+ "details": [
+ {
+ "comment": "Code is reading a file, parsing specific lines to extract model name, python path, and other parameters for paddle2onnx conversion. It's using common functions from \"common_func.sh\" and \"awk\" command for line extraction.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_paddle2onnx.sh\":0-31",
+ "content": "#!/bin/bash\nsource test_tipc/common_func.sh\nFILENAME=$1\nMODE=$2\ndataline=$(cat ${FILENAME})\nlines=(${dataline})\n# common params\nmodel_name=$(func_parser_value \"${lines[1]}\")\npython=$(func_parser_value \"${lines[2]}\")\n# parser params\ndataline=$(awk 'NR==1, NR==14{print}' $FILENAME)\nIFS=$'\\n'\nlines=(${dataline})\n# parser paddle2onnx\nmodel_name=$(func_parser_value \"${lines[1]}\")\npython=$(func_parser_value \"${lines[2]}\")\npadlle2onnx_cmd=$(func_parser_value \"${lines[3]}\")\ninfer_model_dir_key=$(func_parser_key \"${lines[4]}\")\ninfer_model_dir_value=$(func_parser_value \"${lines[4]}\")\nmodel_filename_key=$(func_parser_key \"${lines[5]}\")\nmodel_filename_value=$(func_parser_value \"${lines[5]}\")\nparams_filename_key=$(func_parser_key \"${lines[6]}\")\nparams_filename_value=$(func_parser_value \"${lines[6]}\")\nsave_file_key=$(func_parser_key \"${lines[7]}\")\nsave_file_value=$(func_parser_value \"${lines[7]}\")\nopset_version_key=$(func_parser_key \"${lines[8]}\")\nopset_version_value=$(func_parser_value \"${lines[8]}\")\nenable_onnx_checker_key=$(func_parser_key \"${lines[9]}\")"
+ },
+ {
+ "comment": "Creating function \"func_paddle2onnx\" with arguments _script, setting up log path and directories for paddle2onnx inference. It then sets parameters such as infer_model_dir_key, model_filename_key, params_filename_key, save_file_key, and opset_version_key.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_paddle2onnx.sh\":32-57",
+ "content": "enable_onnx_checker_value=$(func_parser_value \"${lines[9]}\")\n# parser onnx inference\ninference_py=$(func_parser_value \"${lines[10]}\")\nconfig_key=$(func_parser_key \"${lines[11]}\")\nconfig_value=$(func_parser_value \"${lines[11]}\")\nmodel_key=$(func_parser_key \"${lines[12]}\")\ninput_file_key=$(func_parser_key \"${lines[13]}\")\ninput_file_value=$(func_parser_value \"${lines[13]}\")\nLOG_PATH=\"./log/${model_name}/${MODE}\"\nmkdir -p ${LOG_PATH}\nstatus_log=\"${LOG_PATH}/results_paddle2onnx.log\"\nfunction func_paddle2onnx(){\n IFS='|'\n _script=$1\n # paddle2onnx\n _save_log_path=\"${LOG_PATH}/paddle2onnx_infer_cpu.log\"\n set_dirname=$(func_set_params \"${infer_model_dir_key}\" \"${infer_model_dir_value}\")\n set_model_filename=$(func_set_params \"${model_filename_key}\" \"${model_filename_value}\")\n set_params_filename=$(func_set_params \"${params_filename_key}\" \"${params_filename_value}\")\n set_save_model=$(func_set_params \"${save_file_key}\" \"${save_file_value}\")\n set_opset_version=$(func_set_params \"${opset_version_key}\" \"${opset_version_value}\")"
+ },
+ {
+ "comment": "The code sets enable_onnx_checker and uses it to execute paddle2onnx conversion, saves the log. Then, it runs inference using Python and saves the status check log.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_paddle2onnx.sh\":58-72",
+ "content": " set_enable_onnx_checker=$(func_set_params \"${enable_onnx_checker_key}\" \"${enable_onnx_checker_value}\")\n trans_log=\"${LOG_PATH}/trans_model.log\"\n trans_model_cmd=\"${padlle2onnx_cmd} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_save_model} ${set_opset_version} ${set_enable_onnx_checker} > ${trans_log} 2>&1 \"\n eval $trans_model_cmd\n last_status=${PIPESTATUS[0]}\n status_check $last_status \"${trans_model_cmd}\" \"${status_log}\" \"${model_name}\"\n # python inference\n set_gpu=$(func_set_params \"${use_gpu_key}\" \"${use_gpu_value}\")\n set_model_dir=$(func_set_params \"${model_key}\" \"${save_file_value}\")\n set_input_file=$(func_set_params \"${input_file_key}\" \"${input_file_value}\")\n set_config=$(func_set_params \"${config_key}\" \"${config_value}\")\n infer_model_cmd=\"${python} ${inference_py} ${set_config} ${set_input_file} ${set_model_dir} > ${_save_log_path} 2>&1 \"\n eval $infer_model_cmd\n last_status=${PIPESTATUS[0]}\n status_check $last_status \"${infer_model_cmd}\" \"${status_log}\" \"${model_name}\""
+ },
+ {
+ "comment": "This code segment is running a test for the function \"func_paddle2onnx\" by exporting Count variable, setting IFS to \"|\", and echoing a message.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_paddle2onnx.sh\":73-80",
+ "content": "}\necho \"################### run test ###################\"\nexport Count=0\nIFS=\"|\"\nfunc_paddle2onnx"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3a21a7b5-da83-452c-b74a-d98a60640456.json b/docs/doc/3a21a7b5-da83-452c-b74a-d98a60640456.json
new file mode 100644
index 000000000..58f46cc90
--- /dev/null
+++ b/docs/doc/3a21a7b5-da83-452c-b74a-d98a60640456.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is a part of PaddleVideo's VideoQualityAssessment module and it imports and defines functions for training and testing models.",
+ "details": [
+ {
+ "comment": "This code is a part of PaddleVideo's VideoQualityAssessment module and it imports and defines functions for training and testing models.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/tasks/__init__.py\":0-19",
+ "content": "\"\"\"\n# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .train import train_model\nfrom .test import test_model\n__all__ = ['train_model', 'test_model']"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3abbc625-0fe6-4346-b69e-b63e8e1e2634.json b/docs/doc/3abbc625-0fe6-4346-b69e-b63e8e1e2634.json
new file mode 100644
index 000000000..364af237a
--- /dev/null
+++ b/docs/doc/3abbc625-0fe6-4346-b69e-b63e8e1e2634.json
@@ -0,0 +1,30 @@
+{
+ "summary": "This code from PaddleVideo's VideoQualityAssessment module builds a record list for tracking metrics during training, appends metric names and AverageMeter instances for various frameworks and models, formats and logs epoch, mode, metric average, and image processing speed information with color-coded visual distinction.",
+ "details": [
+ {
+ "comment": "This code is from the PaddleVideo library's VideoQualityAssessment module. It imports necessary classes and functions, defines logger variables, and provides a function to build a record list for loss and learning rate metrics. The framework type is specified, and if Recognizer1D is part of the specified framework, additional steps may be required.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/record.py\":0-28",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom collections import OrderedDict\nfrom .logger import get_logger, coloring\nlogger = get_logger(\"paddlevideo\")\n__all__ = ['AverageMeter', 'build_record', 'build_rec_record', 'log_batch', 'log_epoch']\ndef build_record(cfg):\n framework_type = cfg.get('framework')\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"lr\", AverageMeter('lr', 'f', need_avg=False)),\n ]\n if 'Recognizer1D' in cfg.framework: #TODO: required specify str in framework"
+ },
+ {
+ "comment": "This code is building a record list for tracking metrics during the training process. It appends various metric names to the record list along with their corresponding AverageMeter instances for different frameworks and models. The AverageMeter keeps track of the average value over time, and each meter has its format specifier for displaying the values. The code also includes a function build_rec_record to create the record list based on the given configuration (cfg).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/record.py\":29-50",
+ "content": " record_list.append((\"hit_at_one\", AverageMeter(\"hit_at_one\", '.5f')))\n record_list.append((\"perr\", AverageMeter(\"perr\", '.5f')))\n record_list.append((\"gap\", AverageMeter(\"gap\", '.5f')))\n elif 'Recognizer' in cfg.framework:\n record_list.append((\"top1\", AverageMeter(\"top1\", '.5f')))\n record_list.append((\"top5\", AverageMeter(\"top5\", '.5f')))\n record_list.append((\"batch_time\", AverageMeter('elapse', '.3f')))\n record_list.append((\"reader_time\", AverageMeter('reader', '.3f')))\n record_list = OrderedDict(record_list)\n return record_list\ndef build_rec_record(cfg):\n \"\"\"build rec record\"\"\"\n framework_type = cfg.get('framework')\n record_list = [\n (\"loss\", AverageMeter('loss', '7.5f')),\n (\"lr\", AverageMeter('lr', 'f', need_avg=False)),\n ]\n if 'Recognizer1D' in cfg.framework: #TODO: required specify str in framework\n record_list.append((\"hit_at_one\", AverageMeter(\"hit_at_one\", '.5f')))\n record_list.append((\"perr\", AverageMeter(\"perr\", '.5f')))"
+ },
+ {
+ "comment": "This code defines a function `record_list` and a class `AverageMeter`. The function creates a list of record names and their corresponding AverageMeter objects, then converts the list to an OrderedDict. The AverageMeter class computes and stores average values, resets upon initialization, updates with new values, and provides properties for displaying total sum and total sum in minutes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/record.py\":51-89",
+ "content": " record_list.append((\"gap\", AverageMeter(\"gap\", '.5f')))\n record_list.append((\"batch_time\", AverageMeter('elapse', '.3f')))\n record_list.append((\"reader_time\", AverageMeter('reader', '.3f')))\n record_list = OrderedDict(record_list)\n return record_list\nclass AverageMeter(object):\n \"\"\"\n Computes and stores the average and current value\n \"\"\"\n def __init__(self, name='', fmt='f', need_avg=True):\n self.name = name\n self.fmt = fmt\n self.need_avg = need_avg\n self.reset()\n def reset(self):\n \"\"\" reset \"\"\"\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n def update(self, val, n=1):\n \"\"\" update \"\"\"\n if isinstance(val, paddle.Tensor):\n val = float(val)\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n @property\n def total(self):\n return '{self.name}_sum: {self.sum:{self.fmt}}'.format(self=self)\n @property\n def total_minute(self):"
+ },
+ {
+ "comment": "The code provides functions to log batch and epoch information for a video quality assessment task. The `log_batch` function takes in metric list, batch ID, epoch ID, total epochs, mode, and ips as input and logs the metrics, current epoch/total epochs, and step details. The `log_epoch` function calculates the mean of the metrics and logs the mean values along with the total batch time for an epoch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/record.py\":90-114",
+ "content": " return '{self.name}_sum: {s:{self.fmt}} min'.format(s=self.sum / 60,\n self=self)\n @property\n def mean(self):\n return '{self.name}_avg: {self.avg:{self.fmt}}'.format(\n self=self) if self.need_avg else ''\n @property\n def value(self):\n return '{self.name}: {self.val:{self.fmt}}'.format(self=self)\ndef log_batch(metric_list, batch_id, epoch_id, total_epoch, mode, ips):\n metric_str = ' '.join([str(m.value) for m in metric_list.values()])\n epoch_str = \"epoch:[{:>3d}/{:<3d}]\".format(epoch_id, total_epoch)\n step_str = \"{:s} step:{:<4d}\".format(mode, batch_id)\n logger.info(\"{:s} {:s} {:s}s {}\".format(\n coloring(epoch_str, \"HEADER\") if batch_id == 0 else epoch_str,\n coloring(step_str, \"PURPLE\"), coloring(metric_str, 'OKGREEN'), ips))\ndef log_epoch(metric_list, epoch, mode, ips):\n metric_avg = ' '.join([str(m.mean) for m in metric_list.values()] +\n [metric_list['batch_time'].total])"
+ },
+ {
+ "comment": "This code snippet is formatting and logging information related to an epoch, mode, metric average, and image processing speed. It uses the \"coloring\" function to color certain parts of the log text (RED, PURPLE, OKGREEN) for better visual distinction. The logger then logs this information with time stamp.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/record.py\":116-121",
+ "content": " end_epoch_str = \"END epoch:{:<3d}\".format(epoch)\n logger.info(\"{:s} {:s} {:s}s {}\".format(coloring(end_epoch_str, \"RED\"),\n coloring(mode, \"PURPLE\"),\n coloring(metric_avg, \"OKGREEN\"),\n ips))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3afd1d71-0f56-4c75-a411-549376b01b34.json b/docs/doc/3afd1d71-0f56-4c75-a411-549376b01b34.json
new file mode 100644
index 000000000..f3f58897d
--- /dev/null
+++ b/docs/doc/3afd1d71-0f56-4c75-a411-549376b01b34.json
@@ -0,0 +1,25 @@
+{
+ "summary": "This code sets up a colorful logging function for PaddleVideo, initializes logger with verbosity levels, and ensures non-propagation of logs. It configures logger for Python's logging module using different formats and handlers based on local rank.",
+ "details": [
+ {
+ "comment": "This code is from the \"logger.py\" file in the PaddleVideo project, and it sets up a coloring function for logging messages with optional colors using ANSI escape sequences. The function takes a message and an optional color parameter, which should be one of the defined colors in the Color dictionary. It asserts that the provided color is indeed a key in the dictionary, and then returns the message with the specified color applied. The function also checks the environment variable \"COLORING\" to determine whether coloring should be enabled or not (default is True).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py\":0-37",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport logging\nimport os\nimport sys\nimport datetime\nfrom paddle.distributed import ParallelEnv\nColor = {\n 'RED': '\\033[31m',\n 'HEADER': '\\033[35m', # deep purple\n 'PURPLE': '\\033[95m', # purple\n 'OKBLUE': '\\033[94m',\n 'OKGREEN': '\\033[92m',\n 'WARNING': '\\033[93m',\n 'FAIL': '\\033[91m',\n 'ENDC': '\\033[0m'\n}\ndef coloring(message, color=\"OKGREEN\"):\n assert color in Color.keys()\n if os.environ.get('COLORING', True):"
+ },
+ {
+ "comment": "This code initializes the PaddleVideo logger and sets its verbosity level to \"INFO\" or \"DEBUG\", depending on the input argument. It also defines a custom time zone converter for logging, and ensures that the logger does not propagate logs to its parent loggers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py\":38-70",
+ "content": " return Color[color] + str(message) + Color[\"ENDC\"]\n else:\n return message\nlogger_initialized = []\ndef setup_logger(output=None, name=\"paddlevideo\", level=\"INFO\"):\n \"\"\"\n Initialize the paddlevideo logger and set its verbosity level to \"INFO\".\n Args:\n output (str): a file name or a directory to save log. If None, will not save log file.\n If ends with \".txt\" or \".log\", assumed to be a file name.\n Otherwise, logs will be saved to `output/log.txt`.\n name (str): the root module name of this logger\n Returns:\n logging.Logger: a logger\n \"\"\"\n def time_zone(sec, fmt):\n real_time = datetime.datetime.now()\n return real_time.timetuple()\n logging.Formatter.converter = time_zone\n logger = logging.getLogger(name)\n if level == \"INFO\":\n logger.setLevel(logging.INFO)\n elif level==\"DEBUG\":\n logger.setLevel(logging.DEBUG)\n logger.propagate = False\n if level == \"DEBUG\":\n plain_formatter = logging.Formatter("
+ },
+ {
+ "comment": "This code configures a logger for Python's logging module. It uses different formats and handlers (stdout, file) based on the local rank of the process, creating separate log files for each worker ranked greater than 0. If the output is a .txt or .log file, it will be used as-is; otherwise, a .log.txt file with optional rank appended will be created. The code also ensures that missing directories for the log file are created beforehand.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py\":71-99",
+ "content": " \"[%(asctime)s] %(name)s %(levelname)s: %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n else:\n plain_formatter = logging.Formatter(\n \"[%(asctime)s] %(message)s\",\n datefmt=\"%m/%d %H:%M:%S\")\n # stdout logging: master only\n local_rank = ParallelEnv().local_rank\n if local_rank == 0:\n ch = logging.StreamHandler(stream=sys.stdout)\n ch.setLevel(logging.DEBUG)\n formatter = plain_formatter\n ch.setFormatter(formatter)\n logger.addHandler(ch)\n # file logging: all workers\n if output is not None:\n if output.endswith(\".txt\") or output.endswith(\".log\"):\n filename = output\n else:\n filename = os.path.join(output, \".log.txt\")\n if local_rank > 0:\n filename = filename + \".rank{}\".format(local_rank)\n # PathManager.mkdirs(os.path.dirname(filename))\n os.makedirs(os.path.dirname(filename), exist_ok=True)\n # fh = logging.StreamHandler(_cached_log_stream(filename)\n fh = logging.FileHandler(filename, mode='a')"
+ },
+ {
+ "comment": "This code initializes a logger object and sets its level to DEBUG, adds a file handler with a plain formatter, and appends the logger's name to an initialized list. The function returns the logger if it has been previously initialized for the given name; otherwise, it sets up the logger using the provided name and optional output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/logger.py\":100-112",
+ "content": " fh.setLevel(logging.DEBUG)\n fh.setFormatter(plain_formatter)\n logger.addHandler(fh)\n logger_initialized.append(name)\n return logger\ndef get_logger(name, output=None):\n logger = logging.getLogger(name)\n if name in logger_initialized:\n return logger\n return setup_logger(name=name, output=name)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3c27a33b-b94d-4fed-9cd1-3ef6fa629d86.json b/docs/doc/3c27a33b-b94d-4fed-9cd1-3ef6fa629d86.json
new file mode 100644
index 000000000..7636366ac
--- /dev/null
+++ b/docs/doc/3c27a33b-b94d-4fed-9cd1-3ef6fa629d86.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The code imports modules, sets up environment, creates an ONNX predictor for video object detection, and performs inference on batches of input files while supporting benchmarking if enabled.",
+ "details": [
+ {
+ "comment": "This code imports necessary modules and defines a function for parsing command-line arguments. It sets up the environment to execute PaddleVideo Inference model scripts. The code also includes license information, ensuring compliance with the Apache License, Version 2.0.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/paddle2onnx/predict_onnx.py\":0-30",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport argparse\nimport os\nimport sys\nfrom os import path as osp\n__dir__ = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(os.path.abspath(os.path.join(__dir__, '../../tools')))\nfrom utils import build_inference_helper, get_config\ndef parse_args():\n def str2bool(v):\n return v.lower() in (\"true\", \"t\", \"1\")\n # general params\n parser = argparse.ArgumentParser(\"PaddleVideo Inference model script\")"
+ },
+ {
+ "comment": "This code snippet is parsing command line arguments for config file, input file path, and ONNX model file path. It also includes parameters for ONNX prediction like batch size, use of GPU, precision, IR optimization, enable benchmark, and CPU threads.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/paddle2onnx/predict_onnx.py\":31-53",
+ "content": " parser.add_argument('-c',\n '--config',\n type=str,\n default='configs/example.yaml',\n help='config file path')\n parser.add_argument(\"-i\", \"--input_file\", type=str, help=\"input file path\")\n parser.add_argument(\"--onnx_file\", type=str, help=\"onnx model file path\")\n # params for onnx predict\n parser.add_argument(\"-b\", \"--batch_size\", type=int, default=1)\n parser.add_argument(\"--use_gpu\",\n type=str2bool,\n default=False,\n help=\"set to False when using onnx\")\n parser.add_argument(\"--precision\", type=str, default=\"fp32\")\n parser.add_argument(\"--ir_optim\", type=str2bool, default=True)\n parser.add_argument(\"--enable_benchmark\",\n type=str2bool,\n default=False,\n help=\"set to False when using onnx\")\n parser.add_argument(\"--cpu_threads\", type=int, default=4)\n return parser.parse_args()"
+ },
+ {
+ "comment": "The code defines a function to create an ONNX predictor by loading an ONNX file and setting configuration options. It also includes functions for parsing file paths and handling command-line arguments. This code is used for onnx model inference, specifically for video object detection tasks. The main function calls other utility functions to parse the input file path and load configuration settings before executing the actual prediction using the created ONNX predictor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/paddle2onnx/predict_onnx.py\":56-91",
+ "content": "def create_onnx_predictor(args, cfg=None):\n import onnxruntime as ort\n onnx_file = args.onnx_file\n config = ort.SessionOptions()\n if args.use_gpu:\n raise ValueError(\n \"onnx inference now only supports cpu! please set `use_gpu` to False.\"\n )\n else:\n config.intra_op_num_threads = args.cpu_threads\n if args.ir_optim:\n config.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL\n predictor = ort.InferenceSession(onnx_file, sess_options=config)\n return config, predictor\ndef parse_file_paths(input_path: str) -> list:\n if osp.isfile(input_path):\n files = [\n input_path,\n ]\n else:\n files = os.listdir(input_path)\n files = [\n file for file in files\n if (file.endswith(\".avi\") or file.endswith(\".mp4\"))\n ]\n files = [osp.join(input_path, file) for file in files]\n return files\ndef main():\n \"\"\"predict using onnx model\n \"\"\"\n args = parse_args()\n cfg = get_config(args.config, show=False)"
+ },
+ {
+ "comment": "This code builds an inference helper, creates an ONNX predictor, gets input and output names, processes file paths, performs benchmarking, and initializes an auto log for the given model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/paddle2onnx/predict_onnx.py\":93-121",
+ "content": " model_name = cfg.model_name\n print(f\"Inference model({model_name})...\")\n InferenceHelper = build_inference_helper(cfg.INFERENCE)\n inference_config, predictor = create_onnx_predictor(args)\n # get input_tensor and output_tensor\n input_names = predictor.get_inputs()[0].name\n output_names = predictor.get_outputs()[0].name\n # get the absolute file path(s) to be processed\n files = parse_file_paths(args.input_file)\n if args.enable_benchmark:\n test_video_num = 12\n num_warmup = 3\n # instantiate auto log\n try:\n import auto_log\n except ImportError as e:\n print(f\"{e}, [git+https://github.com/LDOUBLEV/AutoLog] \"\n f\"package and it's dependencies is required for \"\n f\"python-inference when enable_benchmark=True.\")\n pid = os.getpid()\n autolog = auto_log.AutoLogger(\n model_name=cfg.model_name,\n model_precision=args.precision,\n batch_size=args.batch_size,\n data_shape=\"dynamic\","
+ },
+ {
+ "comment": "Code snippet performs video inference on batches of input files using a predictor. It preprocesses the batch inputs, runs inference for each batch, and records pre-processing and inference time costs if benchmarking is enabled.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/paddle2onnx/predict_onnx.py\":122-152",
+ "content": " save_path=\"./output/auto_log.lpg\",\n inference_config=inference_config,\n pids=pid,\n process_name=None,\n gpu_ids=None,\n time_keys=['preprocess_time', 'inference_time', 'postprocess_time'],\n warmup=num_warmup)\n files = [args.input_file for _ in range(test_video_num + num_warmup)]\n # Inferencing process\n batch_num = args.batch_size\n for st_idx in range(0, len(files), batch_num):\n ed_idx = min(st_idx + batch_num, len(files))\n # auto log start\n if args.enable_benchmark:\n autolog.times.start()\n # Pre process batched input\n batched_inputs = InferenceHelper.preprocess_batch(files[st_idx:ed_idx])\n # get pre process time cost\n if args.enable_benchmark:\n autolog.times.stamp()\n # run inference\n batched_outputs = predictor.run(\n output_names=[output_names],\n input_feed={input_names: batched_inputs[0]})\n # get inference process time cost"
+ },
+ {
+ "comment": "The code segment is controlling the benchmark execution. If `args.enable_benchmark` is True, it stamps the current time using autolog, then calls postprocess function on batched outputs with `not args.enable_benchmark`. After that, it ends the timer using autolog and reports the benchmark log if `args.enable_benchmark` is still True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/paddle2onnx/predict_onnx.py\":153-170",
+ "content": " if args.enable_benchmark:\n autolog.times.stamp()\n InferenceHelper.postprocess(batched_outputs, not args.enable_benchmark)\n # get post process time cost\n if args.enable_benchmark:\n autolog.times.end(stamp=True)\n # time.sleep(0.01) # sleep for T4 GPU\n # report benchmark log if enabled\n if args.enable_benchmark:\n autolog.report()\nif __name__ == \"__main__\":\n main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3ca025f8-4e3a-47be-be41-e1cad067fd2a.json b/docs/doc/3ca025f8-4e3a-47be-be41-e1cad067fd2a.json
new file mode 100644
index 000000000..dedd441c9
--- /dev/null
+++ b/docs/doc/3ca025f8-4e3a-47be-be41-e1cad067fd2a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This file contains the definitions for BaseSegment and Manet classes, both part of PaddleVideo framework. These classes are likely used in video modeling or segmentation tasks. The code is licensed under Apache License 2.0 and distributed as-is without warranties or conditions.",
+ "details": [
+ {
+ "comment": "This file contains the definitions for BaseSegment and Manet classes, both part of PaddleVideo framework. These classes are likely used in video modeling or segmentation tasks. The code is licensed under Apache License 2.0 and distributed as-is without warranties or conditions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/__init__.py\":0-18",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .segment import BaseSegment, Manet\n__all__ = ['BaseSegment',\n 'Manet'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3dffe712-135b-480f-8096-a59d64202e9d.json b/docs/doc/3dffe712-135b-480f-8096-a59d64202e9d.json
new file mode 100644
index 000000000..f7d9555d6
--- /dev/null
+++ b/docs/doc/3dffe712-135b-480f-8096-a59d64202e9d.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This file is a Python module for video dataset loading and processing in PaddleVideo. It contains functions to build datasets, data loaders, and batch pipelines, along with the VideoDataset class.",
+ "details": [
+ {
+ "comment": "This file is a Python module for video dataset loading and processing in PaddleVideo. It contains functions to build datasets, data loaders, and batch pipelines, along with the VideoDataset class.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/__init__.py\":0-20",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .builder import build_dataset, build_dataloader, build_batch_pipeline\nfrom .dataset import VideoDataset\n__all__ = [\n 'build_dataset', 'build_dataloader', 'build_batch_pipeline', 'VideoDataset'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3e162e69-611f-4080-90c9-982004493277.json b/docs/doc/3e162e69-611f-4080-90c9-982004493277.json
new file mode 100644
index 000000000..bf6aedc31
--- /dev/null
+++ b/docs/doc/3e162e69-611f-4080-90c9-982004493277.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The NetVLAD class in PaddleVideo's \"T2VLAD\" model initializes neural network parameters, performs checks and calculations for VLAD representations with batch size x dimension K.",
+ "details": [
+ {
+ "comment": "NetVLAD is a class for implementing the NetVLAD algorithm. It takes parameters such as cluster_size, feature_size, ghost_clusters, and add_batch_norm. The feature_size represents the size of each feature, while the cluster_size represents the number of clusters. Ghost_clusters determines whether to include extra clusters for better performance. Add_batch_norm is a boolean value that decides whether or not to use batch normalization in the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/net_vlad.py\":0-32",
+ "content": "\"\"\"NetVLAD implementation.\n\"\"\"\n# Copyright 2021 Antoine Miech All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport numpy as np\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nclass NetVLAD(nn.Layer):\n def __init__(self, cluster_size, feature_size, ghost_clusters=0,\n add_batch_norm=True):\n super().__init__()\n self.feature_size = feature_size\n self.cluster_size = cluster_size\n self.ghost_clusters = ghost_clusters\n init_sc = (1 / math.sqrt(feature_size))"
+ },
+ {
+ "comment": "This code initializes the neural network parameters for a VLAD model. It creates two sets of cluster weights, and assigns random values within a certain range to these weights using Paddle's `paddle.randn` function with a specified initialization scale (`init_sc`). Additionally, it creates batch normalization layers (`BatchNorm1D`) for the clusters if `add_batch_norm` is True. The code also defines the output dimension as the product of cluster size and feature size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/net_vlad.py\":33-43",
+ "content": " init_sc = paddle.to_tensor(init_sc)\n clusters = cluster_size + ghost_clusters\n # The `clusters` weights are the `(w,b)` in the paper\n self.clusters = paddle.create_parameter([feature_size, clusters], dtype='float32', default_initializer=nn.initializer.Assign(paddle.randn([feature_size, clusters]) * init_sc))\n self.batch_norm1 = nn.BatchNorm1D(clusters) if add_batch_norm else None\n self.batch_norm2 = nn.BatchNorm1D(clusters) if add_batch_norm else None\n # The `clusters2` weights are the visual words `c_k` in the paper\n self.clusters1 = paddle.create_parameter([1, feature_size, cluster_size], dtype='float32', default_initializer=nn.initializer.Assign(paddle.randn([1, feature_size, cluster_size]) * init_sc))\n self.clusters2 = paddle.create_parameter([1, feature_size, cluster_size], dtype='float32', default_initializer=nn.initializer.Assign(paddle.randn([1, feature_size, cluster_size]) * init_sc)) \n self.out_dim = self.cluster_size * feature_size"
+ },
+ {
+ "comment": "The code snippet is a part of the \"T2VLAD\" model in PaddleVideo. It performs sanity checks to ensure there are no NaN inputs or clusters, and then proceeds with the forward pass. In the forward function, it reshapes input, applies batch normalization, and calculates the assignment between input features and clusters. This is used for aggregating feature maps into a fixed-size representation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/net_vlad.py\":45-75",
+ "content": " def sanity_checks(self, x):\n \"\"\"Catch any nans in the inputs/clusters\"\"\"\n if paddle.isnan(paddle.sum(x)):\n raise ValueError(\"nan inputs\")\n if paddle.isnan(self.clusters[0][0]): \n raise ValueError(\"nan clusters\")\n def forward(self, x, freeze=False, mask=None):\n \"\"\"Aggregates feature maps into a fixed size representation. In the following\n notation, B = batch_size, N = num_features, K = num_clusters, D = feature_size.\n Args:\n x (th.Tensor): B x N x D\n Returns:\n (th.Tensor): B x DK\n \"\"\"\n self.sanity_checks(x)\n max_sample = x.shape[1] \n x = x.reshape([-1, self.feature_size]) # B x N x D -> BN x D\n if freeze == True:\n clusters = self.clusters.detach()\n clusters2 = self.clusters1\n batch_norm = self.batch_norm1\n else:\n clusters = self.clusters\n clusters2 = self.clusters2\n batch_norm = self.batch_norm2\n assignment = paddle.matmul(x, clusters) # (BN x D) x (D x (K+G)) -> BN x (K+G)"
+ },
+ {
+ "comment": "In this code snippet, it performs batch normalization on the assignment matrix, applies softmax for normalization, reshapes the assignment matrix multiple times, calculates a sum of clusters and multiplies by cluster centers, performs matrix multiplication to generate a VLAD representation, normalizes the intra-cluster L2 norm, and finally reshapes and applies normalization for the final VLAD representation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/net_vlad.py\":76-98",
+ "content": " if batch_norm:\n assignment = batch_norm(assignment)\n assignment = F.softmax(assignment, axis=1) # BN x (K+G) -> BN x (K+G)\n save_ass = assignment.reshape([-1, max_sample, self.cluster_size+1])\n assignment = assignment[:, :self.cluster_size]\n assignment = assignment.reshape([-1, max_sample, self.cluster_size]) # -> B x N x K\n a_sum = paddle.sum(assignment, axis=1, keepdim=True) # B x N x K -> B x 1 x K\n a = a_sum * self.clusters2\n assignment = assignment.transpose([0, 2, 1]) # B x N x K -> B x K x N\n x = x.reshape([-1, max_sample, self.feature_size]) # BN x D -> B x N x D\n vlad = paddle.matmul(assignment, x) # (B x K x N) x (B x N x D) -> B x K x D\n vlad = vlad.transpose([0, 2, 1]) # -> B x D x K\n vlad = vlad - a\n # L2 intra norm\n vlad_ = F.normalize(vlad)\n # flattening + L2 norm\n vlad = vlad_.reshape([-1, self.cluster_size * self.feature_size]) # -> B x DK\n vlad = F.normalize(vlad)"
+ },
+ {
+ "comment": "The code is returning the VLAD (Vector of Locally Aggregated Descriptors) feature representations and their respective variables for Batch size x Dimension K.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/model/net_vlad.py\":99-99",
+ "content": " return vlad, vlad_, save_ass # B x DK"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/3f8ca3d4-39cf-43a3-a4bb-98ff2cd75d93.json b/docs/doc/3f8ca3d4-39cf-43a3-a4bb-98ff2cd75d93.json
new file mode 100644
index 000000000..9e08dbaf6
--- /dev/null
+++ b/docs/doc/3f8ca3d4-39cf-43a3-a4bb-98ff2cd75d93.json
@@ -0,0 +1,75 @@
+{
+ "summary": "The code utilizes an ArgumentParser to handle command line arguments, downloads and saves a model, initializes PaddleVideo with GPU/MKLDNN usage for video label prediction, and iterates through results to print top classes/scores/labels.",
+ "details": [
+ {
+ "comment": "This code block is a license notice for the Apache License, Version 2.0, which grants permission to use this file as long as it complies with the terms of the license.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":0-23",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,"
+ },
+ {
+ "comment": "This code imports necessary modules, defines paths and model names for PaddleVideo inference models, and includes a function to parse command line arguments. The code is setting up the environment for using different PaddleVideo models and downloading them if needed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":24-63",
+ "content": "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport sys\n__dir__ = os.path.dirname(__file__)\nsys.path.append(os.path.join(__dir__, ''))\nimport numpy as np\nimport tarfile\nimport requests\nfrom tqdm import tqdm\nimport shutil\nfrom paddle import inference\nfrom paddle.inference import Config, create_predictor\nfrom tools.utils import ppTSM_Inference_helper\n__all__ = ['PaddleVideo']\n# path of download model and data\nBASE_DIR = os.path.expanduser(\"~/.paddlevideo_inference/\")\nBASE_INFERENCE_MODEL_DIR = os.path.join(BASE_DIR, 'inference_model')\nBASE_VIDEOS_DIR = os.path.join(BASE_DIR, 'videos')\n# support Models\nMODELS = {\n 'ppTSM':\n 'https://videotag.bj.bcebos.com/PaddleVideo/InferenceModel/ppTSM_infer.tar',\n 'ppTSM_v2':\n 'https://videotag.bj.bcebos.com/PaddleVideo/InferenceModel/ppTSM_v2_infer.tar'\n}\nMODEL_NAMES = list(MODELS.keys())\ndef parse_args(mMain=True, add_help=True):"
+ },
+ {
+ "comment": "This code defines a function that creates an ArgumentParser object for command line arguments. It includes various argument types and default values, such as model name, video file, use GPU flag, number of segments, short and target sizes, and batch size. The function is intended to be used in the main section of a Python script when set to True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":64-92",
+ "content": " \"\"\"\n Args:\n mMain: bool. True for command args, False for python interface\n \"\"\"\n import argparse\n def str2bool(v):\n return v.lower() in (\"true\", \"t\", \"1\")\n if mMain == True:\n # general params\n parser = argparse.ArgumentParser(add_help=add_help)\n parser.add_argument(\"--model_name\", type=str, default='')\n parser.add_argument(\"-v\", \"--video_file\", type=str, default='')\n parser.add_argument(\"--use_gpu\", type=str2bool, default=True)\n # params for decode and sample\n parser.add_argument(\"--num_seg\", type=int, default=16)\n # params for preprocess\n parser.add_argument(\"--short_size\", type=int, default=256)\n parser.add_argument(\"--target_size\", type=int, default=224)\n # params for predict\n parser.add_argument(\"--model_file\", type=str, default='')\n parser.add_argument(\"--params_file\", type=str)\n parser.add_argument(\"-b\", \"--batch_size\", type=int, default=1)\n parser.add_argument(\"--use_fp16\", type=str2bool, default=False)"
+ },
+ {
+ "comment": "This code is initializing argument parser with default values for various options like ir_optim, use_tensorrt, gpu_mem, top_k and enable_mkldnn. It then parses the arguments using argparse and returns the resulting Namespace.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":93-114",
+ "content": " parser.add_argument(\"--ir_optim\", type=str2bool, default=True)\n parser.add_argument(\"--use_tensorrt\", type=str2bool, default=False)\n parser.add_argument(\"--gpu_mem\", type=int, default=8000)\n parser.add_argument(\"--top_k\", type=int, default=1)\n parser.add_argument(\"--enable_mkldnn\", type=bool, default=False)\n parser.add_argument(\"--label_name_path\", type=str, default='')\n return parser.parse_args()\n else:\n return argparse.Namespace(model_name='',\n video_file='',\n use_gpu=True,\n num_seg=16,\n short_size=256,\n target_size=224,\n model_file='',\n params_file='',\n batch_size=1,\n use_fp16=False,\n ir_optim=True,\n use_tensorrt=False,"
+ },
+ {
+ "comment": "Function `parse_file_paths` takes an input path as a parameter, checks if it is a file or a directory. If it's a file, it returns the file itself; otherwise, it lists all files in the directory, filters out those that don't end with \".avi\" or \".mp4\", and joins the input path with each filtered file to form an absolute path. These paths are then returned as a list.\n\nFunction `download_with_progressbar` downloads data from the given URL in chunks while providing progress updates using tqdm's progress bar. It sets the total size of the download based on the 'content-length' header from the response, and writes each chunk to the specified save path in a 'wb' mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":115-144",
+ "content": " gpu_mem=8000,\n top_k=1,\n enable_mkldnn=False,\n label_name_path='')\ndef parse_file_paths(input_path: str) -> list:\n if os.path.isfile(input_path):\n files = [\n input_path,\n ]\n else:\n files = os.listdir(input_path)\n files = [\n file for file in files\n if (file.endswith(\".avi\") or file.endswith(\".mp4\"))\n ]\n files = [os.path.join(input_path, file) for file in files]\n return files\ndef download_with_progressbar(url, save_path):\n response = requests.get(url, stream=True)\n total_size_in_bytes = int(response.headers.get('content-length', 0))\n block_size = 1024 # 1 Kibibyte\n progress_bar = tqdm(total=total_size_in_bytes, unit='iB', unit_scale=True)\n with open(save_path, 'wb') as file:\n for data in response.iter_content(block_size):\n progress_bar.update(len(data))\n file.write(data)"
+ },
+ {
+ "comment": "The code downloads an inference model from a given URL and saves it to the specified directory. It first checks if the required files ('inference.pdiparams' and 'inference.pdmodel') exist, then creates temporary directories for downloading, prints the download progress, extracts the tar archive containing the model files, and raises an exception if any issue occurs during the process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":145-167",
+ "content": " progress_bar.close()\n if total_size_in_bytes == 0 or progress_bar.n != total_size_in_bytes:\n raise Exception(\"Something went wrong while downloading models\")\ndef download_inference_model(model_storage_directory, url):\n # using custom model\n tar_file_name_list = [\n 'inference.pdiparams', 'inference.pdiparams.info', 'inference.pdmodel'\n ]\n if not os.path.exists(\n os.path.join(model_storage_directory,\n 'inference.pdiparams')) or not os.path.exists(\n os.path.join(model_storage_directory,\n 'inference.pdmodel')):\n tmp_path = os.path.join(model_storage_directory, url.split('/')[-1])\n print('download {} to {}'.format(url, tmp_path))\n os.makedirs(model_storage_directory, exist_ok=True)\n download_with_progressbar(url, tmp_path) #download\n #save to directory\n with tarfile.open(tmp_path, 'r') as tarObj:\n for member in tarObj.getmembers():"
+ },
+ {
+ "comment": "This code is initializing a Paddle predictor by reading arguments and configuring the model accordingly. It enables GPU use or MKLDNN based on the provided flags, sets the log level, and switches IR optimization if requested.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":168-196",
+ "content": " filename = None\n for tar_file_name in tar_file_name_list:\n if tar_file_name in member.name:\n filename = tar_file_name\n if filename is None:\n continue\n file = tarObj.extractfile(member)\n with open(os.path.join(model_storage_directory, filename),\n 'wb') as f:\n f.write(file.read())\n os.remove(tmp_path)\ndef create_paddle_predictor(args):\n config = Config(args.model_file, args.params_file)\n if args.use_gpu:\n config.enable_use_gpu(args.gpu_mem, 0)\n else:\n config.disable_gpu()\n if args.enable_mkldnn:\n # cache 10 different shapes for mkldnn to avoid memory leak\n config.set_mkldnn_cache_capacity(10)\n config.enable_mkldnn()\n config.disable_glog_info()\n config.switch_ir_optim(args.ir_optim) # default true\n if args.use_tensorrt:\n config.enable_tensorrt_engine("
+ },
+ {
+ "comment": "The code snippet is initializing a PaddleVideo object and creating a predictor. It sets the precision mode based on the `args.use_fp16` flag, enables memory optimization, and switches off zero copy operations. It also loads a label name dictionary from the specified path. The purpose of this code is to facilitate model inference using PaddleVideo and provide a user-friendly interface.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":197-231",
+ "content": " precision_mode=Config.Precision.Half\n if args.use_fp16 else Config.Precision.Float32,\n max_batch_size=args.batch_size)\n config.enable_memory_optim()\n # use zero copy\n config.switch_use_feed_fetch_ops(False)\n predictor = create_predictor(config)\n return predictor\ndef load_label_name_dict(path):\n result = {}\n if not os.path.exists(path):\n print(\n 'Warning: If want to use your own label_dict, please input legal path!\\nOtherwise label_names will be empty!'\n )\n else:\n for line in open(path, 'r'):\n partition = line.split('\\n')[0].partition(' ')\n try:\n result[int(partition[0])] = str(partition[-1])\n except:\n result = {}\n break\n return result\nclass PaddleVideo(object):\n def __init__(self, **kwargs):\n print(\n '\\nInference models that Paddle provides are listed as follows:\\n{}'\n .format(MODEL_NAMES), '\\n')\n process_params = parse_args(mMain=False, add_help=False)"
+ },
+ {
+ "comment": "This code checks if the model file exists, if not it prompts for a model name and downloads a pre-trained model from the provided URL if the model name is in the MODEL_NAMES list. It creates directories for the downloaded files and updates process_params with paths to the inference.pdmodel, inference.pdiparams, label_name_path files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":232-252",
+ "content": " process_params.__dict__.update(**kwargs)\n if not os.path.exists(process_params.model_file):\n if process_params.model_name is None:\n raise Exception('Please input model name that you want to use!')\n if process_params.model_name in MODEL_NAMES:\n url = MODELS[process_params.model_name]\n download_path = os.path.join(BASE_INFERENCE_MODEL_DIR,\n process_params.model_name)\n if not os.path.exists(download_path):\n os.makedirs(download_path)\n #create pretrained model download_path\n download_inference_model(model_storage_directory=download_path,\n url=url)\n process_params.model_file = os.path.join(\n download_path, 'inference.pdmodel')\n process_params.params_file = os.path.join(\n download_path, 'inference.pdiparams')\n process_params.label_name_path = os.path.join("
+ },
+ {
+ "comment": "The code initializes an object that can predict video labels using PaddleVideo. It checks for the presence of required parameters and allows user-specified models, then loads label name dictionary, and finally defines a \"predict\" method to classify videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":253-276",
+ "content": " __dir__, '../data/k400/Kinetics-400_label_list.txt')\n else:\n raise Exception(\n 'If you want to use your own model, Please input model_file as model path!'\n )\n else:\n print('Using user-specified model and params!')\n print(\"process params are as follows: \\n{}\".format(process_params))\n self.label_name_dict = load_label_name_dict(\n process_params.label_name_path)\n self.args = process_params\n self.predictor = create_paddle_predictor(process_params)\n def predict(self, video):\n \"\"\"\n predict label of video with paddlevideo\n Args:\n video:input video for clas, support single video , internet url, folder path containing series of videos\n Returns:\n list[dict:{videoname: \"\",class_ids: [], scores: [], label_names: []}],if label name path is None,label names will be empty\n \"\"\"\n video_list = []\n assert isinstance(video, (str))"
+ },
+ {
+ "comment": "The code fetches input and output tensor names from the predictor, then retrieves their handles. If the video is a URL, it downloads the internet video and saves it to the BASE_VIDEOS_DIR. The downloaded video file path replaces the original URL. It checks if the video is not legal (not a string) and outputs an error message.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":278-300",
+ "content": " # get input_tensor and output_tensor\n input_names = self.predictor.get_input_names()\n output_names = self.predictor.get_output_names()\n input_tensor_list = []\n output_tensor_list = []\n for item in input_names:\n input_tensor_list.append(self.predictor.get_input_handle(item))\n for item in output_names:\n output_tensor_list.append(self.predictor.get_output_handle(item))\n if isinstance(video, str):\n # download internet video\n if video.startswith('http'):\n if not os.path.exists(BASE_VIDEOS_DIR):\n os.makedirs(BASE_VIDEOS_DIR)\n video_path = os.path.join(BASE_VIDEOS_DIR, 'tmp.mp4')\n download_with_progressbar(video, video_path)\n print(\"Current using video from Internet:{}, renamed as: {}\".\n format(video, video_path))\n video = video_path\n files = parse_file_paths(video)\n else:\n print('Please input legal video!')"
+ },
+ {
+ "comment": "Looping over each chunk of files, preprocesses and runs inference on batched inputs, then post-processes the outputs to store in `batched_outputs`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":302-326",
+ "content": " # Inferencing process\n InferenceHelper = ppTSM_Inference_helper(\n num_seg=self.args.num_seg,\n short_size=self.args.short_size,\n target_size=self.args.target_size,\n top_k=self.args.top_k)\n batch_num = self.args.batch_size\n for st_idx in range(0, len(files), batch_num):\n ed_idx = min(st_idx + batch_num, len(files))\n # Pre process batched input\n batched_inputs = InferenceHelper.preprocess_batch(\n files[st_idx:ed_idx])\n # run inference\n for i in range(len(input_tensor_list)):\n input_tensor_list[i].copy_from_cpu(batched_inputs[i])\n self.predictor.run()\n batched_outputs = []\n for j in range(len(output_tensor_list)):\n batched_outputs.append(output_tensor_list[j].copy_to_cpu())\n results_list = InferenceHelper.postprocess(batched_outputs,\n print_output=False,"
+ },
+ {
+ "comment": "This code block is iterating through the 'results_list' and adding labels to each result. If the 'label_name_dict' is not empty, it assigns the corresponding label names from the dictionary to the results. It then prints various information about each result such as video file name and top classes/scores/labels. The main function initializes the PaddleVideo class and calls its 'predict' method with a specific video file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":327-352",
+ "content": " return_result=True)\n for res in results_list:\n classes = res[\"topk_class\"]\n label_names = []\n if len(self.label_name_dict) != 0:\n label_names = [self.label_name_dict[c] for c in classes]\n res[\"label_names\"] = label_names\n print(\"Current video file: {0}\".format(res[\"video_id\"]))\n print(\"\\ttop-{0} classes: {1}\".format(len(res[\"topk_class\"]),\n res[\"topk_class\"]))\n print(\"\\ttop-{0} scores: {1}\".format(len(res[\"topk_scores\"]),\n res[\"topk_scores\"]))\n print(\"\\ttop-{0} label names: {1}\".format(\n len(res[\"label_names\"]), res[\"label_names\"]))\ndef main():\n # for cmd\n args = parse_args(mMain=True)\n clas_engine = PaddleVideo(**(args.__dict__))\n clas_engine.predict(args.video_file)\nif __name__ == '__main__':"
+ },
+ {
+ "comment": "This line of code likely represents the entry point for the execution of the script, calling the main function to kick off the program's logic. The specific function or operations within this main() function will depend on the rest of the codebase.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/tools/wheel.py\":353-353",
+ "content": " main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/401cb792-b6b6-4218-b04f-42235071fca2.json b/docs/doc/401cb792-b6b6-4218-b04f-42235071fca2.json
new file mode 100644
index 000000000..fed534b83
--- /dev/null
+++ b/docs/doc/401cb792-b6b6-4218-b04f-42235071fca2.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The `mask_damager` function from PaddleVideo library takes a mask and maximum rotation angle, damages and scales it by applying rotations and translations to create damaged labels.",
+ "details": [
+ {
+ "comment": "This code defines a function, `mask_damager`, which randomly applies transformations to the input labels. It can make the entire label black with 20% probability or rotate and scale it by random values from predefined ranges. If morphology transformation is applied with 50% probability, it uses an open operation to modify the labels using a randomly generated kernel.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/mask_damaging.py\":0-35",
+ "content": "import numpy as np\nfrom scipy.ndimage import interpolation\ntry:\n from skimage import morphology, transform\nexcept ImportError as e:\n print(\n f\"{e}, [scikit-image] package and it's dependencies is required for MA-Net.\"\n )\nimport paddle\nimport cv2\nimport random\n####\ndef mask_damager(labels=None, p_black=0.2):\n scales = (0.8, 1.0, 1.2)\n kernel_size = random.randint(10, 15)\n kernel = np.ones((kernel_size, kernel_size), np.uint8)\n if random.random() < p_black:\n final_label = paddle.zeros_like(labels)\n final_label = final_label.squeeze().numpy()\n else:\n prot = random.randint(5, 15)\n nrot = random.randint(-15, -5)\n rots = [prot, nrot, 0]\n rot = rots[random.randint(0, 2)]\n sc = scales[random.randint(0, 2)]\n _, _, h, w = labels.shape\n tmp = labels.squeeze()\n tmp = tmp.unsqueeze(-1)\n tmp = tmp.numpy().astype(np.uint8)\n morph_p = random.random()\n if morph_p < 0.5:\n tmp = cv2.morphologyEx(tmp, cv2.MORPH_OPEN, kernel)"
+ },
+ {
+ "comment": "The code defines two functions, \"damage_masks\" and \"damage_masks_np\". The former applies mask damaging to a batch of input labels while the latter performs the actual mask damaging on individual numpy arrays. These functions can be used to alter the input masks by applying shifts, scales, and rotations. The output is then converted into a tensor for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/mask_damaging.py\":36-72",
+ "content": " else:\n tmp = cv2.morphologyEx(tmp, cv2.MORPH_CLOSE, kernel)\n tmp = tmp.astype(np.uint8)\n center = (w / 2, h / 2)\n M = cv2.getRotationMatrix2D(center, rot, sc)\n final_label = cv2.warpAffine(tmp, M, (w, h), cv2.INTER_NEAREST)\n return final_label\n#####\ndef damage_masks(labels, shift=True, scale=True, rotate=True):\n \"\"\"\n Args:\n labels: numpy array (batch_size * 1 * h * w)\n \"\"\"\n bs, _, h, w = labels.shape\n labels = labels.transpose([0, 2, 3, 1])\n labels = labels.numpy()\n final_label = []\n for i in range(bs):\n label = labels[i]\n damaged_label = damage_masks_np(label, shift, scale, rotate)\n final_label.append(damaged_label)\n final_label = np.array(final_label)\n final_label = paddle.to_tensor(final_label)\n final_label = final_label.transpose([0, 3, 1, 2])\n return final_label\ndef damage_masks_np(labels, shift=True, scale=True, rotate=True):\n \"\"\"Performs the actual mask damaging in numpy.\n Args:\n labels: Int32 numpy array of shape (height, width, 1)."
+ },
+ {
+ "comment": "This function takes a mask and applies various damage operations such as shifting, scaling, rotation, or dilation to it. It first extracts unique labels from the input mask, shuffles them for random depth ordering, and creates an empty damaged labels array. Then, for each unique label, it applies the single object mask damaging function to the corresponding mask region. The damaged masks are then combined with their original labels to create the final damaged labels array.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/mask_damaging.py\":73-97",
+ "content": " shift: Boolean, whether to damage the masks by shifting.\n scale: Boolean, whether to damage the masks by scaling.\n rotate: Boolean, whether to damage the masks by rotation.\n dilate: Boolean, whether to damage the masks by dilation.\n Returns:\n The damaged version of labels.\n \"\"\"\n unique_labels = np.unique(labels)\n unique_labels = np.setdiff1d(unique_labels, [0])\n # Shuffle to get random depth ordering when combining together.\n np.random.shuffle(unique_labels)\n damaged_labels = np.zeros_like(labels)\n for l in unique_labels:\n obj_mask = (labels == l)\n damaged_obj_mask = _damage_single_object_mask(obj_mask, shift, scale,\n rotate)\n damaged_labels[damaged_obj_mask] = l\n return damaged_labels\ndef _damage_single_object_mask(mask, shift, scale, rotate):\n \"\"\"Performs mask damaging in numpy for a single object.\n Args:\n mask: Boolean numpy array of shape(height, width, 1).\n shift: Boolean, whether to damage the masks by shifting."
+ },
+ {
+ "comment": "This code is from the PaddleVideo library and performs mask damage on a given input mask. The mask can be damaged by shifting, scaling, and/or rotation depending on the provided boolean parameters. The function _shift_mask() shifts the mask randomly based on the maximum shift factor. The returned mask is the damaged version of the original input mask.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/mask_damaging.py\":98-128",
+ "content": " scale: Boolean, whether to damage the masks by scaling.\n rotate: Boolean, whether to damage the masks by rotation.\n dilate: Boolean, whether to damage the masks by dilation.\n Returns:\n The damaged version of mask.\n \"\"\"\n if shift:\n mask = _shift_mask(mask)\n if scale:\n mask = _scale_mask(mask)\n if rotate:\n mask = _rotate_mask(mask)\n return mask\ndef _shift_mask(mask, max_shift_factor=0.05):\n \"\"\"Damages a mask for a single object by randomly shifting it in numpy.\n Args:\n mask: Boolean numpy array of shape(height, width, 1).\n max_shift_factor: Float scalar, the maximum factor for random shifting.\n Returns:\n The shifted version of mask.\n \"\"\"\n nzy, nzx, _ = mask.nonzero()\n h = nzy.max() - nzy.min()\n w = nzx.max() - nzx.min()\n size = np.sqrt(h * w)\n offset = np.random.uniform(-size * max_shift_factor,\n size * max_shift_factor, 2)\n shifted_mask = interpolation.shift(np.squeeze(mask, axis=2),\n offset,"
+ },
+ {
+ "comment": "The code contains three functions: _damage_mask, _scale_mask, and _rotate_mask. These functions are used to randomly damage a mask for a single object in the image. The _damage_mask function applies random noise to the mask by using random values from a uniform distribution and subtracting it from the original mask. The _scale_mask function scales the mask in numpy by applying a scale factor randomly generated within a specific range around 1.0. The _rotate_mask function rotates the mask by a random angle between -max_rot_degrees to max_rot_degrees degrees. These functions can be used together or separately depending on the desired damage to the mask.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/mask_damaging.py\":129-154",
+ "content": " order=0).astype('bool')[..., np.newaxis]\n return shifted_mask\ndef _scale_mask(mask, scale_amount=0.025):\n \"\"\"Damages a mask for a single object by randomly scaling it in numpy.\n Args:\n mask: Boolean numpy array of shape(height, width, 1).\n scale_amount: Float scalar, the maximum factor for random scaling.\n Returns:\n The scaled version of mask.\n \"\"\"\n nzy, nzx, _ = mask.nonzero()\n cy = 0.5 * (nzy.max() - nzy.min())\n cx = 0.5 * (nzx.max() - nzx.min())\n scale_factor = np.random.uniform(1.0 - scale_amount, 1.0 + scale_amount)\n shift = transform.SimilarityTransform(translation=[-cx, -cy])\n inv_shift = transform.SimilarityTransform(translation=[cx, cy])\n s = transform.SimilarityTransform(scale=[scale_factor, scale_factor])\n m = (shift + (s + inv_shift)).inverse\n scaled_mask = transform.warp(mask, m) > 0.5\n return scaled_mask\ndef _rotate_mask(mask, max_rot_degrees=3.0):\n \"\"\"Damages a mask for a single object by randomly rotating it in numpy."
+ },
+ {
+ "comment": "This function takes a boolean numpy array mask and maximum rotation angle, then returns a scaled version of the mask after applying random rotations and translations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/mask_damaging.py\":155-169",
+ "content": " Args:\n mask: Boolean numpy array of shape(height, width, 1).\n max_rot_degrees: Float scalar, the maximum number of degrees to rotate.\n Returns:\n The scaled version of mask.\n \"\"\"\n cy = 0.5 * mask.shape[0]\n cx = 0.5 * mask.shape[1]\n rot_degrees = np.random.uniform(-max_rot_degrees, max_rot_degrees)\n shift = transform.SimilarityTransform(translation=[-cx, -cy])\n inv_shift = transform.SimilarityTransform(translation=[cx, cy])\n r = transform.SimilarityTransform(rotation=np.deg2rad(rot_degrees))\n m = (shift + (r + inv_shift)).inverse\n scaled_mask = transform.warp(mask, m) > 0.5\n return scaled_mask"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4074085e-82d4-480b-a2cd-258f08196ddc.json b/docs/doc/4074085e-82d4-480b-a2cd-258f08196ddc.json
new file mode 100644
index 000000000..6829fa185
--- /dev/null
+++ b/docs/doc/4074085e-82d4-480b-a2cd-258f08196ddc.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code contains four functions that utilize the FFmpeg tool for handling video and audio files. \"ffmpeg_frames\" extracts frames from a given MP4 file, \"ffmpeg_pcm\" extracts audio in PCM format, \"ffmpeg_mp4\" downloads an MP4 file, and \"get_images\" lists the images inside a specified image directory.",
+ "details": [
+ {
+ "comment": "This code contains four functions that utilize the FFmpeg tool for handling video and audio files. \"ffmpeg_frames\" extracts frames from a given MP4 file, \"ffmpeg_pcm\" extracts audio in PCM format, \"ffmpeg_mp4\" downloads an MP4 file, and \"get_images\" lists the images inside a specified image directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/utils/preprocess.py\":0-34",
+ "content": "\"\"\" extract frames and pcm\"\"\"\nimport os\nimport sys\nimport shutil\ndef ffmpeg_frames(mp4_addr, frame_out_folder, fps=5):\n \"\"\"ffmpeg_frames\"\"\"\n if os.path.exists(frame_out_folder):\n shutil.rmtree(frame_out_folder)\n os.makedirs(frame_out_folder)\n cmd = './src/utils/ffmpeg -v 0 -i %s -r %d -q 0 %s/%s.jpg' % (mp4_addr, fps, frame_out_folder, '%08d')\n os.system(cmd)\ndef ffmpeg_pcm(mp4_addr, save_file_name):\n \"\"\"ffmpeg_pcm\"\"\"\n cmd = './src/utils/ffmpeg -y -i %s -acodec pcm_s16le -f s16le -ac 1 -ar 16000 %s -v 0' \\\n % (mp4_addr, save_file_name)\n os.system(cmd)\ndef ffmpeg_mp4(mp4_url, mp4_addr):\n \"\"\"ffmpeg_mp4\"\"\"\n cmd = \"wget %s -O %s -q\" % (mp4_url, mp4_addr)\n print (\"cmd = \", cmd)\n os.system(cmd)\ndef get_images(image_path):\n \"\"\"get_images\"\"\"\n images = sorted(os.listdir(image_path))\n images = images\n images_path_list = [image_path + '/' + im for im in images]\n return images_path_list"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/40c75bd6-da2e-49e5-90a6-02102a291481.json b/docs/doc/40c75bd6-da2e-49e5-90a6-02102a291481.json
new file mode 100644
index 000000000..02f6341a0
--- /dev/null
+++ b/docs/doc/40c75bd6-da2e-49e5-90a6-02102a291481.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This Python function uses PaddlePaddle to test models, enabling parallel processing and logging. It initializes the device, constructs model, dataset, and dataloader in test mode with adjustable parameters. The code builds a dataloader, loads state_dicts, sets up metrics, and iterates over batches for output or metric updates before accumulating the final result.",
+ "details": [
+ {
+ "comment": "The code is a Python function for testing a model using PaddlePaddle framework. It takes configuration (cfg) and weights path (weights) as inputs, and allows for parallel processing. The logger captures any log messages from the function execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/tasks/test.py\":0-31",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddlevideo.utils import get_logger, load\nfrom ..loader.builder import build_dataloader, build_dataset\nfrom ..metrics import build_metric\nfrom ..modeling.builder import build_model\nlogger = get_logger(\"paddlevideo\")\n@paddle.no_grad()\ndef test_model(cfg, weights, parallel=True):\n \"\"\"Test model entry\n Args:\n cfg (dict): configuration.\n weights (str): weights path to load.\n parallel (bool): Whether to do multi-cards testing. Default: True."
+ },
+ {
+ "comment": "This code block initializes the model's device, constructs and configures the model, dataset, and dataloader. It also sets test mode and adjusts batch size and number of workers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/tasks/test.py\":33-60",
+ "content": " \"\"\"\n if cfg.get('use_npu', False):\n places = paddle.set_device('npu')\n elif cfg.get('use_xpu', False):\n places = paddle.set_device('xpu')\n else:\n places = paddle.set_device('gpu')\n # 1. Construct model.\n if cfg.MODEL.get('backbone') and cfg.MODEL.backbone.get('pretrained'):\n cfg.MODEL.backbone.pretrained = '' # disable pretrain model init\n model = build_model(cfg.MODEL)\n if parallel:\n model = paddle.DataParallel(model)\n # 2. Construct dataset and dataloader.\n cfg.DATASET.test.test_mode = True\n dataset = build_dataset((cfg.DATASET.test, cfg.PIPELINE.test))\n batch_size = cfg.DATASET.get(\"test_batch_size\", 8)\n # default num worker: 0, which means no subprocess will be created\n num_workers = cfg.DATASET.get('num_workers', 0)\n num_workers = cfg.DATASET.get('test_num_workers', num_workers)\n dataloader_setting = dict(batch_size=batch_size,\n num_workers=num_workers,\n places=places,"
+ },
+ {
+ "comment": "The code builds a dataloader for the dataset, loads state_dicts into the model, and sets up metrics. It then iterates over batches of data from the dataloader to either update the metric directly or get outputs from the model before updating the metric. After processing all batches, it accumulates the final result in the metric.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/tasks/test.py\":61-89",
+ "content": " drop_last=False,\n shuffle=False)\n data_loader = build_dataloader(\n dataset, **dataloader_setting) if cfg.model_name not in ['CFBI'\n ] else dataset\n model.eval()\n state_dicts = load(weights)\n model.set_state_dict(state_dicts)\n # add params to metrics\n cfg.METRIC.data_size = len(dataset)\n cfg.METRIC.batch_size = batch_size\n Metric = build_metric(cfg.METRIC)\n if cfg.MODEL.framework == \"FastRCNN\":\n Metric.set_dataset_info(dataset.info, len(dataset))\n for batch_id, data in enumerate(data_loader):\n if cfg.model_name in [\n 'CFBI'\n ]: # for VOS task, dataset for video and dataloader for frames in each video\n Metric.update(batch_id, data, model)\n else:\n outputs = model(data, mode='test')\n Metric.update(batch_id, data, outputs)\n Metric.accumulate()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/40dbbb97-f039-42fd-a668-0f9b89bc7745.json b/docs/doc/40dbbb97-f039-42fd-a668-0f9b89bc7745.json
new file mode 100644
index 000000000..a7e256089
--- /dev/null
+++ b/docs/doc/40dbbb97-f039-42fd-a668-0f9b89bc7745.json
@@ -0,0 +1,20 @@
+{
+ "summary": "TSNHead class is defined for image classification, inheriting from BaseHead with num_classes, in_channels, and loss_cfg parameters. It uses AdaptiveAvgPool2D, Linear, Dropout layers and weight_init function for initialization. The function defines a head for TSN model that performs average pooling, reshapes, takes mean, applies dropout if enabled, and passes through fully connected layer for classification scores.",
+ "details": [
+ {
+ "comment": "The code is defining a TSNHead class for image classification, which inherits from the BaseHead class. It has parameters for num_classes, in_channels, and loss_cfg, and uses AdaptiveAvgPool2D, Linear, Dropout layers. The weight_init function is also imported for weight initialization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsn_head.py\":0-29",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle.nn import AdaptiveAvgPool2D, Linear, Dropout\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass TSNHead(BaseHead):\n \"\"\"TSN Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.\n loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss')."
+ },
+ {
+ "comment": "Initializes a TSN head with specified parameters, including num_classes, in_channels, loss_cfg, drop_ratio, std, and data_format. It creates an adaptive average pooling layer, a dropout layer if drop_ratio is non-zero, and a fully connected linear layer (fc). The fc layer weights are then initialized with normal distribution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsn_head.py\":30-62",
+ "content": " drop_ratio(float): drop ratio. Default: 0.4.\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n drop_ratio=0.4,\n std=0.01,\n data_format=\"NCHW\",\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.drop_ratio = drop_ratio\n self.std = std\n #NOTE: global pool performance\n self.avgpool2d = AdaptiveAvgPool2D((1, 1), data_format=data_format)\n if self.drop_ratio != 0:\n self.dropout = Dropout(p=self.drop_ratio)\n else:\n self.dropout = None\n self.fc = Linear(self.in_channels, self.num_classes)\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'Normal',"
+ },
+ {
+ "comment": "The function defines a head for the TSN model. It performs average pooling, reshapes the input, takes the mean along an axis, applies dropout if enabled, and passes the result through a fully connected layer to output scores for classification.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsn_head.py\":63-92",
+ "content": " 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.,\n std=self.std)\n def forward(self, x, num_seg):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n #XXX: check dropout location!\n # [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x)\n # [N * num_segs, in_channels, 1, 1]\n x = paddle.reshape(x, [-1, num_seg, x.shape[1]])\n # [N, num_seg, in_channels]\n x = paddle.mean(x, axis=1)\n # [N, in_channels]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N, in_channels]\n score = self.fc(x)\n # [N, num_class]\n #x = F.softmax(x) #NOTE remove\n return score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/413c1454-cc8c-4299-a143-292ac01c3855.json b/docs/doc/413c1454-cc8c-4299-a143-292ac01c3855.json
new file mode 100644
index 000000000..8c5bd562b
--- /dev/null
+++ b/docs/doc/413c1454-cc8c-4299-a143-292ac01c3855.json
@@ -0,0 +1,65 @@
+{
+ "summary": "This code initializes a reader class for PaddleVideo's MultimodalVideoTag application, preprocesses text, formats input sequences for BERT/ERNIE models, creates Record objects, generates batches with padding, and handles data generation for ERNIE models.",
+ "details": [
+ {
+ "comment": "This code is for the \"ernie\" reader, a part of PaddleVideo's MultimodalVideoTag application. It includes licensing information and various import statements for different functionalities like file handling, JSON parsing, random number generation, logging, numpy operations, and namedtuple creation. The log variable is initialized for error reporting.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":0-34",
+ "content": "\"\"\"\nernie reader\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nfrom __future__ import unicode_literals\nfrom __future__ import absolute_import\nimport sys\nimport os\nimport json\nimport random\nimport logging\nimport numpy as np\nimport six\nfrom io import open\nfrom collections import namedtuple\nfrom .tokenization import FullTokenizer, convert_to_unicode\nlog = logging.getLogger(__name__)"
+ },
+ {
+ "comment": "This code snippet defines a `BaseReader` class which initializes an object with various parameters related to text preprocessing, including maximum sequence length, tokenizer, and other properties. It also includes a utility function `csv_reader` that reads data from files in CSV format. The code adjusts the Python output stream encoding if running on Python 3, ensuring consistent text handling across all outputs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":36-73",
+ "content": "if six.PY3:\n import io\n sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')\n sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8')\ndef csv_reader(fd, delimiter='\\t'):\n \"\"\"csv_reader\n \"\"\"\n def gen():\n \"\"\"gen\n \"\"\"\n for i in fd:\n yield i.rstrip('\\n').split(delimiter)\n return gen()\nclass BaseReader(object):\n \"\"\"BaseReader\n \"\"\"\n def __init__(self,\n vocab_path,\n label_map_config=None,\n max_seq_len=512,\n do_lower_case=True,\n in_tokens=False,\n is_inference=False,\n random_seed=None,\n tokenizer=\"FullTokenizer\",\n is_classify=True,\n is_regression=False,\n for_cn=True,\n task_id=0):\n self.max_seq_len = max_seq_len\n self.tokenizer = FullTokenizer(vocab_file=vocab_path,\n do_lower_case=do_lower_case)\n self.vocab = self.tokenizer.vocab"
+ },
+ {
+ "comment": "This code initializes various attributes of the class and sets up some configurations for tokenizing input data. It also loads a label map from a file if provided, or sets it to None otherwise. The \"_truncate_seq_pair\" function truncates sequence pairs in place to the maximum length specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":74-101",
+ "content": " self.pad_id = self.vocab[\"[PAD]\"]\n self.cls_id = self.vocab[\"[CLS]\"]\n self.sep_id = self.vocab[\"[SEP]\"]\n self.in_tokens = in_tokens\n self.is_inference = is_inference\n self.for_cn = for_cn\n self.task_id = task_id\n np.random.seed(random_seed)\n self.is_classify = is_classify\n self.is_regression = is_regression\n self.current_example = 0\n self.current_epoch = 0\n self.num_examples = 0\n if label_map_config:\n with open(label_map_config, encoding='utf8') as f:\n self.label_map = json.load(f)\n else:\n self.label_map = None\n def _truncate_seq_pair(self, tokens_a, tokens_b, max_length):\n \"\"\"Truncates a sequence pair in place to the maximum length.\"\"\"\n # This is a simple heuristic which will always truncate the longer sequence\n # one token at a time. This makes more sense than truncating an equal percent\n # of tokens from each, since if one sequence is very short then each token"
+ },
+ {
+ "comment": "This function converts an example into a record. It tokenizes text_a and optionally text_b, then truncates the sequences if they exceed max_seq_length by popping tokens from either tokens_a or tokens_b.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":102-130",
+ "content": " # that's truncated likely contains more information than a longer sequence.\n while True:\n total_length = len(tokens_a) + len(tokens_b)\n if total_length <= max_length:\n break\n if len(tokens_a) > len(tokens_b):\n tokens_a.pop()\n else:\n tokens_b.pop()\n def _convert_example_to_record(self, example, max_seq_length, tokenizer):\n \"\"\"Converts a single `Example` into a single `Record`.\"\"\"\n text_a = convert_to_unicode(example.text_a)\n tokens_a = tokenizer.tokenize(text_a)\n tokens_b = None\n has_text_b = False\n if isinstance(example, dict):\n has_text_b = \"text_b\" in example.keys()\n else:\n has_text_b = \"text_b\" in example._fields\n if has_text_b:\n text_b = convert_to_unicode(example.text_b)\n tokens_b = tokenizer.tokenize(text_b)\n if tokens_b:\n # Modifies `tokens_a` and `tokens_b` in place so that the total"
+ },
+ {
+ "comment": "The code ensures that the input sequences for BERT/ERNIE models are formatted correctly. If the sequence length is less than the specified maximum length, it accounts for [CLS], [SEP], and [SEP] tokens with adjustments. If the sequence length exceeds the limit, it truncates the longer token sequence accordingly. The code also assigns type_ids to indicate whether it's the first or second sequence, as these are used in the model's embedding vectors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":131-150",
+ "content": " # length is less than the specified length.\n # Account for [CLS], [SEP], [SEP] with \"- 3\"\n self._truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)\n else:\n # Account for [CLS] and [SEP] with \"- 2\"\n if len(tokens_a) > max_seq_length - 2:\n tokens_a = tokens_a[0:(max_seq_length - 2)]\n # The convention in BERT/ERNIE is:\n # (a) For sequence pairs:\n # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1\n # (b) For single sequences:\n # tokens: [CLS] the dog is hairy . [SEP]\n # type_ids: 0 0 0 0 0 0 0\n #\n # Where \"type_ids\" are used to indicate whether this is the first\n # sequence or the second sequence. The embedding vectors for `type=0` and\n # `type=1` were learned during pre-training and are added to the wordpiece\n # embedding vector (and position vector). This is not *strictly* necessary"
+ },
+ {
+ "comment": "This code prepares input data for the ERNIE model by combining tokens from two input sequences (tokens_a and tokens_b) into a single sequence. It appends \"[CLS]\" at the start, \"[SEP]\" to separate the sequences, and assigns text_type_id 0 or 1 based on the source sequence. The code also converts the tokens to token ids and generates position ids for the input data. This is specifically designed for classification tasks where the \"[CLS]\" vector represents the overall sentence vector after fine-tuning the entire model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":151-178",
+ "content": " # since the [SEP] token unambiguously separates the sequences, but it makes\n # it easier for the model to learn the concept of sequences.\n #\n # For classification tasks, the first vector (corresponding to [CLS]) is\n # used as as the \"sentence vector\". Note that this only makes sense because\n # the entire model is fine-tuned.\n tokens = []\n text_type_ids = []\n tokens.append(\"[CLS]\")\n text_type_ids.append(0)\n for token in tokens_a:\n tokens.append(token)\n text_type_ids.append(0)\n tokens.append(\"[SEP]\")\n text_type_ids.append(0)\n if tokens_b:\n for token in tokens_b:\n tokens.append(token)\n text_type_ids.append(1)\n tokens.append(\"[SEP]\")\n text_type_ids.append(1)\n token_ids = tokenizer.convert_tokens_to_ids(tokens)\n position_ids = list(range(len(token_ids)))\n if self.is_inference:\n Record = namedtuple('Record',"
+ },
+ {
+ "comment": "This code defines a function to create a \"Record\" object, which contains token_ids, text_type_ids, position_ids (possibly label_id and qid depending on the example). It also includes another function _prepare_batch_data that generates batch records from examples. The batch size and phase are also taken as parameters in this function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":179-206",
+ "content": " ['token_ids', 'text_type_ids', 'position_ids'])\n record = Record(token_ids=token_ids,\n text_type_ids=text_type_ids,\n position_ids=position_ids)\n else:\n if self.label_map:\n label_id = self.label_map[example.label]\n else:\n label_id = example.label\n Record = namedtuple('Record', [\n 'token_ids', 'text_type_ids', 'position_ids', 'label_id', 'qid'\n ])\n qid = None\n if \"qid\" in example._fields:\n qid = example.qid\n record = Record(token_ids=token_ids,\n text_type_ids=text_type_ids,\n position_ids=position_ids,\n label_id=label_id,\n qid=qid)\n return record\n def _prepare_batch_data(self, examples, batch_size, phase=None):\n \"\"\"generate batch records\"\"\"\n batch_records, max_len = [], 0"
+ },
+ {
+ "comment": "This code iterates through examples and converts them to records. It then appends the records to a batch and pads the batch with zeros if it reaches the maximum size. It yields batches of records, ensuring that each batch is padded to the same length before being passed to the next step in the process. This class inherits from BaseReader and is used for getting Ernie embedding. The method _pad_batch_records pads the batch with zeros if it exceeds the maximum size, ensuring all batches are of equal length.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":207-234",
+ "content": " for index, example in enumerate(examples):\n if phase == \"train\":\n self.current_example = index\n record = self._convert_example_to_record(example, self.max_seq_len,\n self.tokenizer)\n max_len = max(max_len, len(record.token_ids))\n if self.in_tokens:\n to_append = (len(batch_records) + 1) * max_len <= batch_size\n else:\n to_append = len(batch_records) < batch_size\n if to_append:\n batch_records.append(record)\n else:\n yield self._pad_batch_records(batch_records)\n batch_records, max_len = [record], len(record.token_ids)\n if batch_records:\n yield self._pad_batch_records(batch_records)\nclass ExtractEmbeddingReader(BaseReader):\n \"\"\"\n data prepare for getting erine embedding \n \"\"\"\n def _pad_batch_records(self, batch_records):\n \"\"\"\n \u5bf9\u5b57\u6807\u53f7\uff0c\u4f4d\u7f6e\u6807\u53f7\u7279\u5f81\u8fdb\u884c\u56fa\u5b9a\u957f\u5ea6\u8865\u5168\n batch_records \u5305\u542b\u591a\u6761\u6587\u672c\u7684\u6807\u53f7"
+ },
+ {
+ "comment": "This code is processing a batch of records and padding token ids, text type ids, position ids, and task ids for an ERNIE (Enhanced Refined Network with Incremental Learning and Exploration) model. The processed data will be used as input for the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":235-256",
+ "content": " return [\u5b57\u6807\u53f7\u5217\u8868\uff0c\u6587\u672c\u7c7b\u578b\u5217\u8868\uff0c\u4f4d\u7f6e\u7279\u5f81\u5217\u8868\uff0c\u4efb\u52a1\u6807\u53f7\u5217\u8868\uff0c\u63a9\u7801\u5217\u8868]\n \"\"\"\n batch_token_ids = [record.token_ids for record in batch_records]\n batch_text_type_ids = [\n record.text_type_ids for record in batch_records\n ]\n batch_position_ids = [record.position_ids for record in batch_records]\n # padding\n padded_token_ids, input_mask, seq_lens = pad_batch_data(\n batch_token_ids,\n pad_idx=self.pad_id,\n return_input_mask=True,\n return_seq_lens=True,\n max_len=self.max_seq_len)\n padded_text_type_ids = pad_batch_data(batch_text_type_ids,\n pad_idx=self.pad_id,\n max_len=self.max_seq_len)\n padded_position_ids = pad_batch_data(batch_position_ids,\n pad_idx=self.pad_id,\n max_len=self.max_seq_len)\n padded_task_ids = np.ones_like(padded_token_ids,"
+ },
+ {
+ "comment": "This code is related to text processing and data generation for a specific task reader. It converts input texts into indexed representations and pads the data to ensure consistent sequence lengths. The function `data_generate_from_text` takes in a single text, converts it into a record, pads the batch of records, and returns the resulting one-hot encoded text representation. The `pad_batch_data` function is used for padding other types of data as well.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":257-288",
+ "content": " dtype=\"int64\") * self.task_id\n return_list = [\n padded_token_ids, padded_text_type_ids, padded_position_ids,\n padded_task_ids, input_mask\n ]\n return return_list\n def data_generate_from_text(self, text):\n \"\"\"\n trans text to idx\n input single text\n return 5*maxlen*1\n \"\"\"\n Example = namedtuple('Example', ['text_a', 'label'])\n example = Example(text, 0)\n records = [\n self._convert_example_to_record(example, self.max_seq_len,\n self.tokenizer)\n ]\n pad_records = self._pad_batch_records(records)\n text_one_hot = np.concatenate(pad_records, axis=0).astype('int64')\n return text_one_hot\ndef pad_batch_data(insts,\n pad_idx=0,\n max_len=None,\n return_pos=False,\n return_input_mask=False,\n return_max_len=False,\n return_num_token=False,"
+ },
+ {
+ "comment": "This function pads instances to the maximum sequence length in a batch. It first calculates the max_len based on instance lengths and then adds padding to shorter instances if necessary. It creates a 3D tensor of input data, position data (if required), and attention masks (if required). These tensors are added to a return list before being returned by the function. The padding is used to make no effect on parameter gradients by being masked out with weights.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":289-316",
+ "content": " return_seq_lens=False):\n \"\"\"\n Pad the instances to the max sequence length in batch, and generate the\n corresponding position data and attention bias.\n \"\"\"\n return_list = []\n if max_len is None:\n max_len = max(len(inst) for inst in insts)\n # Any token included in dict can be used to pad, since the paddings' loss\n # will be masked out by weights and make no effect on parameter gradients.\n inst_data = np.array(\n [inst + list([pad_idx] * (max_len - len(inst))) for inst in insts])\n return_list += [inst_data.astype(\"int64\").reshape([-1, max_len, 1])]\n # position data\n if return_pos:\n inst_pos = np.array([\n list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))\n for inst in insts\n ])\n return_list += [inst_pos.astype(\"int64\").reshape([-1, max_len, 1])]\n if return_input_mask:\n # This is used to avoid attention on paddings.\n input_mask_data = np.array(\n [[1] * len(inst) + [0] * (max_len - len(inst)) for inst in insts])"
+ },
+ {
+ "comment": "This code prepares a return list by adding various elements like input_mask_data, max_len (if required), number of tokens (if required), and sequence lengths (if required) before returning the final list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/ernie_task_reader.py\":317-333",
+ "content": " input_mask_data = np.expand_dims(input_mask_data, axis=-1)\n return_list += [input_mask_data.astype(\"float32\")]\n if return_max_len:\n return_list += [max_len]\n if return_num_token:\n num_token = 0\n for inst in insts:\n num_token += len(inst)\n return_list += [num_token]\n if return_seq_lens:\n seq_lens = np.array([len(inst) for inst in insts])\n return_list += [seq_lens.astype(\"int64\").reshape([-1])]\n return return_list if len(return_list) > 1 else return_list[0]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/417ea4e8-07a5-41cb-9904-3076ff6620e1.json b/docs/doc/417ea4e8-07a5-41cb-9904-3076ff6620e1.json
new file mode 100644
index 000000000..19e02f8ab
--- /dev/null
+++ b/docs/doc/417ea4e8-07a5-41cb-9904-3076ff6620e1.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This Python code defines a SegmentationSampler class in PaddleVideo, which samples data at a specified rate and registers it for the Action Segmentation Dataset. It is part of a video processing library's pipeline, likely for segmentation purposes.",
+ "details": [
+ {
+ "comment": "This Python code is from the PaddleVideo library and defines a SegmentationSampler class. It samples data at a specified rate, only keeps every nth element in a 1D array, and registers this pipeline operation for the Action Segmentation Dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/segmentation_pipline.py\":0-34",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport copy\nimport os\nimport numpy as np\nimport random\nimport paddle\nfrom ..registry import PIPELINES\n\"\"\"\npipeline ops for Action Segmentation Dataset.\n\"\"\"\n@PIPELINES.register()\nclass SegmentationSampler(object):\n def __init__(self, sample_rate):\n self.sample_rate = sample_rate\n def __call__(self, results):\n for key, data in results.items():\n if len(data.shape) == 1:\n data = data[::self.sample_rate]"
+ },
+ {
+ "comment": "This code segment appears to be part of a pipeline in a video processing library, possibly for segmentation. It selects specific data based on the sample rate and stores it in a results dictionary with deep copy.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/segmentation_pipline.py\":35-39",
+ "content": " results[key] = copy.deepcopy(data)\n else:\n data = data[:, ::self.sample_rate]\n results[key] = copy.deepcopy(data)\n return results"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/41f3e7a8-79e7-42b4-a57e-9533db88921c.json b/docs/doc/41f3e7a8-79e7-42b4-a57e-9533db88921c.json
new file mode 100644
index 000000000..2f08dafc6
--- /dev/null
+++ b/docs/doc/41f3e7a8-79e7-42b4-a57e-9533db88921c.json
@@ -0,0 +1,65 @@
+{
+ "summary": "This script prepares environment for benchmarking PaddleVideo model, trains with varying batch sizes and precisions, measures execution time, and processes log files to extract performance metrics.",
+ "details": [
+ {
+ "comment": "This script is a Bash function for running benchmark training on PaddlePaddle GPU. It sets environment variables, parses command line arguments, and executes the benchmark training using the provided configuration file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":0-41",
+ "content": "#!/bin/bash\nsource test_tipc/common_func.sh\n# set env\npython=python\nexport model_branch=`git symbolic-ref HEAD 2>/dev/null | cut -d\"/\" -f 3`\nexport model_commit=$(git log|head -n1|awk '{print $2}')\nexport str_tmp=$(echo `pip list|grep paddlepaddle-gpu|awk -F ' ' '{print $2}'`)\nexport frame_version=${str_tmp%%.post*}\nexport frame_commit=$(echo `${python} -c \"import paddle;print(paddle.version.commit)\"`)\n# BENCHMARK_ROOT='.' # only for self-test\n# run benchmark sh\n# Usage:\n# bash run_benchmark_train.sh config.txt params\n# or\n# bash run_benchmark_train.sh config.txt\nfunction func_parser_params(){\n strs=$1\n IFS=\"=\"\n array=(${strs})\n tmp=${array[1]}\n echo ${tmp}\n}\nfunction func_sed_params(){\n filename=$1\n line=$2\n param_value=$3\n params=`sed -n \"${line}p\" $filename`\n IFS=\":\"\n array=(${params})\n key=${array[0]}\n value=${array[1]}\n if [[ $value =~ 'benchmark_train' ]];then\n IFS='='\n _val=(${value})\n param_value=\"${param_value}\"\n fi\n new_params=\"${key}:${param_value}\""
+ },
+ {
+ "comment": "This code defines functions to modify parameters in a file and set GPU IDs. It then copies the input filename, sets the mode as \"benchmark_train\", and processes additional parameters. The script performs operations such as modifying lines in the file and replacing \"trainer:norm_train\" with \"trainer:to_static_train\". The purpose of this code seems to be related to manipulating configuration files for a program using PaddleVideo's test_tipc directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":42-85",
+ "content": " IFS=\";\"\n cmd=\"sed -i '${line}s/.*/${new_params}/' '${filename}'\"\n eval $cmd\n}\nfunction set_gpu_id(){\n string=$1\n _str=${string:1:6}\n IFS=\"C\"\n arr=(${_str})\n M=${arr[0]}\n P=${arr[1]}\n gn=`expr $P - 1`\n gpu_num=`expr $gn / $M`\n seq=`seq -s \",\" 0 $gpu_num`\n echo $seq\n}\nfunction get_repo_name(){\n IFS=\";\"\n cur_dir=$(pwd)\n IFS=\"/\"\n arr=(${cur_dir})\n echo ${arr[-1]}\n}\nFILENAME=$1\n# copy FILENAME as new\nnew_filename=\"./test_tipc/benchmark_train.txt\"\ncmd=`yes|cp $FILENAME $new_filename`\nFILENAME=$new_filename\n# MODE must be one of ['benchmark_train']\nMODE=$2\nPARAMS=$3\nREST_ARGS=$4\n# bash test_tipc/benchmark_train.sh /workspace/PaddleVideo/test_tipc/configs/BMN/train_infer_python.txt benchmark_train dynamicTostatic_bs8_fp32_DP_N1C8\nto_static=\"\"\n# parse \"to_static\" options and modify trainer into \"to_static_trainer\"\nif [[ $PARAMS =~ \"dynamicTostatic\" ]] ;then\n to_static=\"d2sT_\"\n sed -i 's/trainer:norm_train/trainer:to_static_train/g' $FILENAME\n # clear PARAM contents\n if [ $PARAMS = \"to_static\" ] ;then"
+ },
+ {
+ "comment": "The code is parsing parameters from the \"train_benchmark.txt\" file and setting variables such as model name, batch size, fp_items, epoch, profile option key, profile option parameters, flags value, and max_iters value for training purposes. These values will be used to train a specific model with given parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":86-122",
+ "content": " PARAMS=\"\"\n fi\nfi\nIFS=$'\\n'\n# parser params from train_benchmark.txt\ndataline=`cat $FILENAME`\n# parser params\nIFS=$'\\n'\nlines=(${dataline})\nmodel_name=$(func_parser_value \"${lines[1]}\")\n# \u83b7\u53d6'train_benchmark_params'\u6240\u5728\u7684\u884c\u6570\nline_num=`grep -n -w \"train_benchmark_params\" $FILENAME | cut -d \":\" -f 1`\n# for train log parser\nbatch_size=$(func_parser_value \"${lines[line_num]}\")\nline_num=`expr $line_num + 1`\nfp_items=$(func_parser_value \"${lines[line_num]}\")\nline_num=`expr $line_num + 1`\nepoch=$(func_parser_value \"${lines[line_num]}\")\nline_num=`expr $line_num + 1`\nprofile_option_key=$(func_parser_key \"${lines[line_num]}\")\nprofile_option_params=$(func_parser_value \"${lines[line_num]}\")\nprofile_option=\"${profile_option_key}:${profile_option_params}\"\nline_num=`expr $line_num + 1`\nflags_value=$(func_parser_value \"${lines[line_num]}\")\n# \u8bbe\u7f6e\u6bcf\u4e2a\u6a21\u578bmax-iters\uff0c\u4ee5\u83b7\u53d6\u7a33\u5b9a\u7684ips\nline_num=`expr $line_num + 1`\nmax_iters_value=$(func_parser_value \"${lines[line_num]}\")\n# set flags\nIFS=\";\"\nflags_list=(${flags_value})\nfor _flag in ${flags_list[*]}; do"
+ },
+ {
+ "comment": "This code is setting environment variables, defining log file locations and names, and using sed commands to modify a configuration file. It then executes the modified configuration file with additional command line parameters. This is likely part of a benchmarking or training process for machine learning or video processing tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":123-157",
+ "content": " cmd=\"export ${_flag}\"\n eval $cmd\ndone\n# set log_name\nrepo_name=$(get_repo_name )\nSAVE_LOG=${BENCHMARK_LOG_DIR:-$(pwd)} # */benchmark_log\nmkdir -p \"${SAVE_LOG}/benchmark_log/\"\nstatus_log=\"${SAVE_LOG}/benchmark_log/results.log\"\n# get benchmark profiling params : PROFILING_TIMER_ONLY=no|True|False\nPROFILING_TIMER_ONLY=${PROFILING_TIMER_ONLY:-\"True\"}\n# The number of lines in which train params can be replaced.\nline_python=3\nline_gpuid=4\nline_precision=6\nline_epoch=7\nline_batchsize=9\nline_profile=12\nline_eval_py=24\nline_eval_py_2=25\nline_export_py=38\nline_export_py_2=28\nline_export_py_3=30\nline_norm_train=16\nfunc_sed_params \"$FILENAME\" \"${line_eval_py}\" \"null\"\nfunc_sed_params \"$FILENAME\" \"${line_eval_py_2}\" \"null\"\nfunc_sed_params \"$FILENAME\" \"${line_export_py}\" \"null\"\nfunc_sed_params \"$FILENAME\" \"${line_export_py_2}\" \"null\"\nfunc_sed_params \"$FILENAME\" \"${line_export_py_3}\" \"null\"\nfunc_sed_params \"$FILENAME\" \"${line_python}\" \"$python\"\n# \u672b\u5c3e\u52a0\u4e0a--max_iters=30\u548c--log_interval=1\uff0c\u4ee5\u4fbf\u8fd0\u884c\u5e76\u8f93\u51fa\u8db3\u91cf\u6570\u636e\nset_log_interval_cmd=\"sed -i '${line_norm_train}s/.*/& --max_iters=${max_iters_value} -o log_interval=1/' '${filename}'\""
+ },
+ {
+ "comment": "This code is parsing parameters and configuring the environment for benchmarking. It removes \"validate\" from the command, checks if the input is a dynamic or static parameter, and then assigns variables based on the type of model, batch size, precision, run mode, and device number. If the precision is null, it defaults to fp32.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":158-196",
+ "content": "eval $set_log_interval_cmd\n# \u53bb\u6389--validate\uff0cbenchmark\u4e0d\u9700\u8981validate\nremove_validate_cmd=\"sed -i '${line_norm_train}s/--validate//' '${filename}'\"\neval $remove_validate_cmd\n# if params\nif [ ! -n \"$PARAMS\" ] ;then\n # PARAMS input is not a word.\n IFS=\"|\"\n batch_size_list=(${batch_size})\n fp_items_list=(${fp_items})\n device_num_list=(N1C4)\n run_mode=\"DP\"\nelif [[ ${PARAMS} = \"dynamicTostatic\" ]] ;then\n IFS=\"|\"\n model_type=$PARAMS\n batch_size_list=(${batch_size})\n fp_items_list=(${fp_items})\n device_num_list=(N1C4)\n run_mode=\"DP\"\nelse\n # parser params from input: modeltype_bs${bs_item}_${fp_item}_${run_mode}_${device_num}\n IFS=\"_\"\n params_list=(${PARAMS})\n model_type=${params_list[0]}\n batch_size=${params_list[1]}\n batch_size=`echo ${batch_size} | tr -cd \"[0-9]\" `\n precision=${params_list[2]}\n run_mode=${params_list[3]}\n device_num=${params_list[4]}\n IFS=\";\"\n if [ ${precision} = \"null\" ];then\n precision=\"fp32\"\n fi\n fp_items_list=($precision)\n batch_size_list=($batch_size)"
+ },
+ {
+ "comment": "The code is iterating over different combinations of batch sizes and precisions to train the PaddleVideo model. It sets up various environment variables and uses sed to modify a file before running the training script on specific GPUs. The profile option determines if only timer information should be logged or if full profiling data should be collected.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":197-219",
+ "content": " device_num_list=($device_num)\nfi\nlog_interval='--log_interval 1'\nIFS=\"|\"\nfor batch_size in ${batch_size_list[*]}; do\n for precision in ${fp_items_list[*]}; do\n for device_num in ${device_num_list[*]}; do\n # sed batchsize and precision\n func_sed_params \"$FILENAME\" \"${line_precision}\" \"$precision\"\n func_sed_params \"$FILENAME\" \"${line_batchsize}\" \"$batch_size\"\n func_sed_params \"$FILENAME\" \"${line_epoch}\" \"$epoch\"\n gpu_id=$(set_gpu_id $device_num)\n if [ ${#gpu_id} -le 1 ];then\n func_sed_params \"$FILENAME\" \"${line_gpuid}\" \"0\" # sed used gpu_id \n if [[ ${PROFILING_TIMER_ONLY} != \"no\" ]];then\n echo \"run profile\"\n # The default value of profile_option's timer_only parameter is True\n if [[ ${PROFILING_TIMER_ONLY} = \"False\" ]];then\n profile_option=\"${profile_option};timer_only=False\"\n fi\n log_path=\"$SAVE_LOG/profiling_log\""
+ },
+ {
+ "comment": "Creates a directory for log storage, sets the name of the log file based on various parameters, modifies profile option settings if necessary (for TimeSformer models), and then runs test_train_inference_python.sh script with provided arguments, redirecting output to the specified log path.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":220-233",
+ "content": " mkdir -p $log_path\n log_name=\"${repo_name}_${model_name}_bs${batch_size}_${precision}_${run_mode}_${device_num}_${to_static}profiling\"\n # set profile_option params\n tmp=`sed -i \"${line_profile}s/.*/\\\"${profile_option}\\\"/\" \"${FILENAME}\"`\n # for models which need to accumulate gradient.\n if [[ ${model_name} =~ \"TimeSformer\" ]]; then\n global_bs=`expr ${batch_size} \\* ${device_num:3:4} \\* 8`\n modify_global_bs_cmd=\"sed -i '${line_norm_train}s/.*/& -o GRADIENT_ACCUMULATION.global_batch_size=${global_bs}/' '${filename}'\"\n eval $modify_global_bs_cmd\n fi\n # run test_train_inference_python.sh\n cmd=\"timeout 5m bash test_tipc/test_train_inference_python.sh ${FILENAME} benchmark_train > ${log_path}/${log_name} 2>&1 \"\n echo $cmd\n eval ${cmd}"
+ },
+ {
+ "comment": "This code snippet executes a script without profiling. It sets the log and speed log paths, creates directories if necessary, and then runs a command to execute the test_train_inference_python.sh script. The run time is measured and stored in model_run_time variable. Finally, it displays the execution log.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":234-252",
+ "content": " eval \"cat ${log_path}/${log_name}\"\n fi\n echo \"run without profile\" \n # without profile\n log_path=\"$SAVE_LOG/train_log\"\n speed_log_path=\"$SAVE_LOG/index\"\n mkdir -p $log_path\n mkdir -p $speed_log_path\n log_name=\"${repo_name}_${model_name}_bs${batch_size}_${precision}_${run_mode}_${device_num}_${to_static}log\"\n speed_log_name=\"${repo_name}_${model_name}_bs${batch_size}_${precision}_${run_mode}_${device_num}_${to_static}speed\"\n func_sed_params \"$FILENAME\" \"${line_profile}\" \"null\" # sed profile_id as null\n cmd=\"bash test_tipc/test_train_inference_python.sh ${FILENAME} benchmark_train > ${log_path}/${log_name} 2>&1 \"\n echo $cmd\n job_bt=`date '+%Y%m%d%H%M%S'`\n eval $cmd\n job_et=`date '+%Y%m%d%H%M%S'`\n export model_run_time=$((${job_et}-${job_bt}))\n eval \"cat ${log_path}/${log_name}\""
+ },
+ {
+ "comment": "This code section is using Python to execute an analysis script. The analysis script processes log files, extracting performance metrics like inference per second (ips) and loss convergence data. It also handles skipping steps during processing and considers the device used for computation. The resulting status is logged into a specified file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":254-273",
+ "content": " # parser log\n _model_name=\"${model_name}_bs${batch_size}_${precision}_${run_mode}\"\n cmd=\"${python} ${BENCHMARK_ROOT}/scripts/analysis.py --filename ${log_path}/${log_name} \\\n --speed_log_file '${speed_log_path}/${speed_log_name}' \\\n --model_name ${_model_name} \\\n --base_batch_size ${batch_size} \\\n --run_mode ${run_mode} \\\n --fp_item ${precision} \\\n --keyword ips: \\\n --skip_steps 5 \\\n --device_num ${device_num} \\\n --speed_unit instance/sec \\\n --convergence_key loss: \"\n echo $cmd\n eval $cmd\n last_status=${PIPESTATUS[0]}\n status_check $last_status \"${cmd}\" \"${status_log}\" \"${model_name}\"\n else\n IFS=\";\"\n unset_env=`unset CUDA_VISIBLE_DEVICES`"
+ },
+ {
+ "comment": "Creates log and speed directories, sets variable names for logging files. Uses sed to modify the config file with gpu_id, profile option as null, and adjusts global batch size for TimeSformer model that needs gradient accumulation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":274-287",
+ "content": " log_path=\"$SAVE_LOG/train_log\"\n speed_log_path=\"$SAVE_LOG/index\"\n mkdir -p $log_path\n mkdir -p $speed_log_path\n log_name=\"${repo_name}_${model_name}_bs${batch_size}_${precision}_${run_mode}_${device_num}_${to_static}log\"\n speed_log_name=\"${repo_name}_${model_name}_bs${batch_size}_${precision}_${run_mode}_${device_num}_${to_static}speed\"\n func_sed_params \"$FILENAME\" \"${line_gpuid}\" \"$gpu_id\" # sed used gpu_id\n func_sed_params \"$FILENAME\" \"${line_profile}\" \"null\" # sed --profile_option as null\n # for models which need to accumulate gradient.\n if [[ ${model_name} =~ \"TimeSformer\" ]]; then\n global_bs=`expr ${batch_size} \\* ${device_num:3:4} \\* 8`\n modify_global_bs_cmd=\"sed -i '${line_norm_train}s/.*/& -o GRADIENT_ACCUMULATION.global_batch_size=${global_bs}/' '${filename}'\"\n eval $modify_global_bs_cmd"
+ },
+ {
+ "comment": "This code segment is running a benchmark training script and logging the results. It measures the model run time, parses the log to extract information about the speed, and then passes this information to another script for further analysis. The script is designed to handle different batch sizes, precision types, and run modes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":288-307",
+ "content": " fi\n cmd=\"bash test_tipc/test_train_inference_python.sh ${FILENAME} benchmark_train > ${log_path}/${log_name} 2>&1 \"\n echo $cmd\n job_bt=`date '+%Y%m%d%H%M%S'`\n eval $cmd\n job_et=`date '+%Y%m%d%H%M%S'`\n export model_run_time=$((${job_et}-${job_bt}))\n eval \"cat ${log_path}/${log_name}\"\n # parser log\n _model_name=\"${model_name}_bs${batch_size}_${precision}_${run_mode}\"\n cmd=\"${python} ${BENCHMARK_ROOT}/scripts/analysis.py --filename ${log_path}/${log_name} \\\n --speed_log_file '${speed_log_path}/${speed_log_name}' \\\n --model_name ${_model_name} \\\n --base_batch_size ${batch_size} \\\n --run_mode ${run_mode} \\\n --fp_item ${precision} \\\n --keyword ips: \\\n --skip_steps 5 \\\n --device_num ${device_num} \\"
+ },
+ {
+ "comment": "This code iterates through different models and configurations, running them with specified parameters. It logs the commands and checks their status to ensure successful execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/benchmark_train.sh\":308-317",
+ "content": " --speed_unit instance/sec \\\n --convergence_key loss: \"\n echo $cmd\n eval $cmd\n last_status=${PIPESTATUS[0]}\n status_check $last_status \"${cmd}\" \"${status_log}\" \"${model_name}\"\n fi\n done\n done\ndone"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/425817af-113d-48b2-881e-e1becef9b76b.json b/docs/doc/425817af-113d-48b2-881e-e1becef9b76b.json
new file mode 100644
index 000000000..822efaaad
--- /dev/null
+++ b/docs/doc/425817af-113d-48b2-881e-e1becef9b76b.json
@@ -0,0 +1,60 @@
+{
+ "summary": "The ConvBNLayer class introduces PaddlePaddle's MobileNetV2 backbone model for image/video processing with pretrained weights and inverted residual units. It initializes and returns three models (PPTSM_MobileNetV2_x0_75, PPTSM_MobileNetV2_x1_5, PPTSM_MobileNetV2_x2_0).",
+ "details": [
+ {
+ "comment": "This code is part of the PaddlePaddle deep learning framework, specifically for the MobileNetV2 backbone model. It imports necessary libraries and defines functions for the architecture, weight initialization, and pre-trained model downloading. The commented sections provide licensing information and download URLs for pretrained models.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":0-29",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nfrom paddle import ParamAttr\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.nn import Conv2D, BatchNorm, Linear, Dropout\nfrom paddle.nn import AdaptiveAvgPool2D, MaxPool2D, AvgPool2D\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt\n# Download URL of pretrained model\n# {\n# \"MobileNetV2\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_ssld_pretrained.pdparams\","
+ },
+ {
+ "comment": "This code defines the ConvBNLayer class, which inherits from nn.Layer and contains a convolutional layer followed by a batch normalization layer. The constructor takes several parameters such as number of channels, filter size, etc., to define the specifics of the convolutional layer. The URLs provided indicate that pretrained models are available for MobileNetV2 with various scaling factors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":31-57",
+ "content": "# \"MobileNetV2_x0_25\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x0_25_pretrained.pdparams\",\n# \"MobileNetV2_x0_5\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x0_5_pretrained.pdparams\",\n# \"MobileNetV2_x0_75\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x0_75_pretrained.pdparams\",\n# \"MobileNetV2_x1_5\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x1_5_pretrained.pdparams\",\n# \"MobileNetV2_x2_0\":\n# \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV2_x2_0_pretrained.pdparams\"\n# }\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n num_channels,\n filter_size,\n num_filters,\n stride,\n padding,\n channels=None,\n num_groups=1,\n name=None,\n use_cudnn=True):\n super(ConvBNLayer, self).__init__()\n self._conv = Conv2D(in_channels=num_channels,"
+ },
+ {
+ "comment": "The code defines a class for an inverted residual unit with batch normalization. The unit takes input, performs convolution, applies batch normalization, and optionally applies activation if specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":58-84",
+ "content": " out_channels=num_filters,\n kernel_size=filter_size,\n stride=stride,\n padding=padding,\n groups=num_groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False)\n self._batch_norm = BatchNorm(\n num_filters,\n param_attr=ParamAttr(name=name + \"_bn_scale\"),\n bias_attr=ParamAttr(name=name + \"_bn_offset\"),\n moving_mean_name=name + \"_bn_mean\",\n moving_variance_name=name + \"_bn_variance\")\n def forward(self, inputs, if_act=True):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if if_act:\n y = F.relu6(y)\n return y\nclass InvertedResidualUnit(nn.Layer):\n def __init__(self, num_channels, num_in_filter, num_filters, stride,\n filter_size, padding, expansion_factor, name, num_seg):\n super(InvertedResidualUnit, self).__init__()"
+ },
+ {
+ "comment": "This code initializes and assigns class attributes for a backbone model. It defines two convolutional layers, one for expansion (num_channels to num_expfilter) and another for bottleneck (num_expfilter to num_expfilter), both followed by BN operations. The layers are named with the prefix \"name\" for future reference or identification.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":85-102",
+ "content": " self.num_seg = num_seg\n num_expfilter = int(round(num_in_filter * expansion_factor))\n self._expand_conv = ConvBNLayer(num_channels=num_channels,\n num_filters=num_expfilter,\n filter_size=1,\n stride=1,\n padding=0,\n num_groups=1,\n name=name + \"_expand\")\n self._bottleneck_conv = ConvBNLayer(num_channels=num_expfilter,\n num_filters=num_expfilter,\n filter_size=filter_size,\n stride=stride,\n padding=padding,\n num_groups=num_expfilter,\n use_cudnn=False,\n name=name + \"_dwise\")"
+ },
+ {
+ "comment": "This code defines a neural network layer, likely for image or video processing. It contains a series of convolutional layers and activation functions. The \"forward\" function applies temporal shift to the input based on the number of segments and performs convolutions in different stages. The \"InvresiBlocks\" class defines an Inverted Residual block with initial parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":104-131",
+ "content": " self._linear_conv = ConvBNLayer(num_channels=num_expfilter,\n num_filters=num_filters,\n filter_size=1,\n stride=1,\n padding=0,\n num_groups=1,\n name=name + \"_linear\")\n def forward(self, inputs, ifshortcut):\n # add temporal shift module\n y = inputs\n if ifshortcut:\n y = F.temporal_shift(y, self.num_seg, 1.0 / self.num_seg)\n y = self._expand_conv(y, if_act=True)\n y = self._bottleneck_conv(y, if_act=True)\n y = self._linear_conv(y, if_act=False)\n if ifshortcut:\n y = paddle.add(inputs, y)\n return y\nclass InvresiBlocks(nn.Layer):\n def __init__(self, in_c, t, c, n, s, name, num_seg):\n super(InvresiBlocks, self).__init__()\n self._first_block = InvertedResidualUnit(num_channels=in_c,\n num_in_filter=in_c,"
+ },
+ {
+ "comment": "The code defines a function for the PPTSM_MV2 model, creating an InvertedResidualUnit with specified parameters and adding it to a list. The loop iterates from 1 to n-1, building multiple residual units with increasing indexes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":132-150",
+ "content": " num_filters=c,\n stride=s,\n filter_size=3,\n padding=1,\n expansion_factor=t,\n name=name + \"_1\",\n num_seg=num_seg)\n self._block_list = []\n for i in range(1, n):\n block = self.add_sublayer(name + \"_\" + str(i + 1),\n sublayer=InvertedResidualUnit(\n num_channels=c,\n num_in_filter=c,\n num_filters=c,\n stride=1,\n filter_size=3,\n padding=1,\n expansion_factor=t,"
+ },
+ {
+ "comment": "This code defines a PPTSM-MV2 backbone and MobileNet model for image processing. The `__init__` function initializes the model with class number, scaling factor, pretrained weights, prefix name, and number of segments. The `forward` function passes inputs through each block in sequence. The `MobileNet` class defines a convolutional neural network (CNN) architecture with specific parameters for each stage, including the number of filters and stride sizes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":151-186",
+ "content": " name=name + \"_\" + str(i + 1),\n num_seg=num_seg))\n self._block_list.append(block)\n def forward(self, inputs):\n y = self._first_block(inputs, ifshortcut=False)\n for block in self._block_list:\n y = block(y, ifshortcut=True)\n return y\nclass MobileNet(nn.Layer):\n def __init__(self,\n class_num=400,\n scale=1.0,\n pretrained=None,\n prefix_name=\"\",\n num_seg=8):\n super(MobileNet, self).__init__()\n self.scale = scale\n self.class_num = class_num\n self.pretrained = pretrained\n self.num_seg = num_seg\n bottleneck_params_list = [\n (1, 16, 1, 1),\n (6, 24, 2, 2),\n (6, 32, 3, 2),\n (6, 64, 4, 2),\n (6, 96, 3, 1),\n (6, 160, 3, 2),\n (6, 320, 1, 1),\n ]\n self.conv1 = ConvBNLayer(num_channels=3,\n num_filters=int(32 * scale),"
+ },
+ {
+ "comment": "This code initializes a PPTSM_MV2 backbone model. It adds a convolution layer with specific parameters, and creates a list of block layers using InvresiBlocks with varying settings. The scale value affects the number of input channels in each layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":187-206",
+ "content": " filter_size=3,\n stride=2,\n padding=1,\n name=prefix_name + \"conv1_1\")\n self.block_list = []\n i = 1\n in_c = int(32 * scale)\n for layer_setting in bottleneck_params_list:\n t, c, n, s = layer_setting\n i += 1\n block = self.add_sublayer(prefix_name + \"conv\" + str(i),\n sublayer=InvresiBlocks(in_c=in_c,\n t=t,\n c=int(c * scale),\n n=n,\n s=s,\n name=prefix_name +\n \"conv\" + str(i),\n num_seg=num_seg))"
+ },
+ {
+ "comment": "This code defines a class, appends blocks to block_list, sets output channels based on scale factor, initializes convolution and pooling layers, and defines an initialization function for the weights. It seems to be part of a deep learning model backbone implementation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":207-231",
+ "content": " self.block_list.append(block)\n in_c = int(c * scale)\n self.out_c = int(1280 * scale) if scale > 1.0 else 1280\n self.conv9 = ConvBNLayer(num_channels=in_c,\n num_filters=self.out_c,\n filter_size=1,\n stride=1,\n padding=0,\n name=prefix_name + \"conv9\")\n self.pool2d_avg = AdaptiveAvgPool2D(1)\n self.out = Linear(self.out_c,\n class_num,\n weight_attr=ParamAttr(name=prefix_name +\n \"fc10_weights\"),\n bias_attr=ParamAttr(name=prefix_name + \"fc10_offset\"))\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":"
+ },
+ {
+ "comment": "Initializes a PPTSM MobileNetV2 model with optional pretrained weights and customizable scale. Initializes the underlying MobileNet model and applies custom modifications. Iterates through sublayers, applying Kaiming Normal initialization for Conv2D layers and constant initialization for BatchNorm2D layers. Defines the forward pass of the PPTSM MobileNetV2 model. Registers multiple PPTSM MobileNetV2 variants with different scales.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":232-265",
+ "content": " for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n y = self.conv1(inputs, if_act=True)\n for block in self.block_list:\n y = block(y)\n y = self.conv9(y, if_act=True)\n y = self.pool2d_avg(y)\n y = paddle.reshape(y, [-1, self.num_seg, y.shape[1]])\n y = paddle.mean(y, axis=1)\n y = paddle.reshape(y, shape=[-1, self.out_c])\n y = self.out(y)\n return y\n@BACKBONES.register()\ndef PPTSM_MobileNetV2(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=1.0, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x0_25(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=0.25, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x0_5(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=0.5, **kwargs)"
+ },
+ {
+ "comment": "The code defines three functions, PPTSM_MobileNetV2_x0_75, PPTSM_MobileNetV2_x1_5, and PPTSM_MobileNetV2_x2_0. Each function creates a MobileNet model with different scales (0.75, 1.5, and 2.0) using the MobileNet class. The pretrained option allows for loading pre-trained weights if set to True. The functions return the created models.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/pptsm_mv2.py\":266-281",
+ "content": " return model\ndef PPTSM_MobileNetV2_x0_75(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=0.75, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x1_5(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=1.5, **kwargs)\n return model\ndef PPTSM_MobileNetV2_x2_0(pretrained=None, **kwargs):\n model = MobileNet(pretrained=pretrained, scale=2.0, **kwargs)\n return model"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/42da3399-dbce-42b4-8259-d88277b5384d.json b/docs/doc/42da3399-dbce-42b4-8259-d88277b5384d.json
new file mode 100644
index 000000000..11ead2a34
--- /dev/null
+++ b/docs/doc/42da3399-dbce-42b4-8259-d88277b5384d.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code provides instructions on how to download and organize the DAVIS2017 dataset for use in the Ma-Net application. It includes links to the necessary datasets, such as the DAVIS-2017-trainval-480p.zip and DAVIS-2017-scribbles-trainval.zip files, and provides a template directory structure for organizing the data within the PaddleVideo project.",
+ "details": [
+ {
+ "comment": "This code provides instructions on how to download and organize the DAVIS2017 dataset for use in the Ma-Net application. It includes links to the necessary datasets, such as the DAVIS-2017-trainval-480p.zip and DAVIS-2017-scribbles-trainval.zip files, and provides a template directory structure for organizing the data within the PaddleVideo project.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/dataloaders/DAVIS2017_cn.md\":0-26",
+ "content": "[English](../../en/dataset/DAVIS2017.md) | \u7b80\u4f53\u4e2d\u6587\n# DAVIS2017 \u6570\u636e\u96c6\u51c6\u5907\n## 1.\u6570\u636e\u4e0b\u8f7d\n\u4e0b\u8f7d [DAVIS2017](https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-trainval-480p.zip) \u548c [scribbles](https://data.vision.ee.ethz.ch/csergi/share/DAVIS-Interactive/DAVIS-2017-scribbles-trainval.zip)\u5230\u540c\u4e00\u4e2a\u6587\u4ef6\u5939\u4e2d\u3002\u8bf7\u53c2\u9605[DAVIS](https://davischallenge.org/davis2017/code.html).\n\u5982\u679c\u60a8\u9700\u8981\u6587\u4ef6\"DAVIS2017/ImageSets/2017/v_a_l_instances.txt\"\uff0c\u8bf7\u53c2\u9605[google](https://drive.google.com/file/d/1aLPaQ_5lyAi3Lk3d2fOc_xewSrfcrQlc/view?usp=sharing)\u94fe\u63a5\n## 2.\u76ee\u5f55\u7ed3\u6784\n\u6574\u4e2a\u9879\u76ee(Ma-Net)\u7684\u76ee\u5f55\u7ed3\u6784\u5982\u4e0b\u6240\u793a\uff1a\n```shell\nPaddleVideo\n\u251c\u2500\u2500 configs\n\u251c\u2500\u2500 paddlevideo\n\u251c\u2500\u2500 docs\n\u251c\u2500\u2500 tools\n\u251c\u2500\u2500 data\n\u2502 \t\u2514\u2500\u2500 DAVIS2017\n\u2502 \u2502 \t\u251c\u2500\u2500 Annotations\n\u2502 \u2502 \t\u251c\u2500\u2500 ImageSets\n\u2502 \u2502 \t\u251c\u2500\u2500 JPEGImages\n\u2502 \u2502 \t\u2514\u2500\u2500 Scribbles\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/42efd6d7-5026-4a7c-b967-1321ca165543.json b/docs/doc/42efd6d7-5026-4a7c-b967-1321ca165543.json
new file mode 100644
index 000000000..c418985cb
--- /dev/null
+++ b/docs/doc/42efd6d7-5026-4a7c-b967-1321ca165543.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The code is a Paddle Video inference script with argparse handling and an \"InferModel\" class for model loading and prediction. It supports GPU usage, multimodal video tagging, and customizable parameters. The inference function takes videos, labels, predicts, and outputs results to a JSON file.",
+ "details": [
+ {
+ "comment": "This code is a Paddle Video inference script for a specific model. It parses arguments such as the model name, config file path, output path, use_gpu, and save_inference_model flag. The script utilizes argparse to handle these command-line arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/inference.py\":0-37",
+ "content": "#!/usr/bin/env python\n# coding=utf-8\n\"\"\"\ninfer model\n\"\"\"\nimport sys\nimport os\nimport numpy as np\nimport json\nimport pickle\nimport argparse\nimport time\nimport numpy as np\nimport paddle\nfrom datareader import get_reader\nfrom config import merge_configs, parse_config, print_configs\ndef parse_args():\n \"\"\"parse_args\n \"\"\"\n parser = argparse.ArgumentParser(\"Paddle Video infer script\")\n parser.add_argument('--model_name',\n type=str,\n default='BaiduNet',\n help='name of model to train.')\n parser.add_argument('--config',\n type=str,\n default='configs/conf.txt',\n help='path to config file of model')\n parser.add_argument('--output', type=str, default=None, help='output path')\n parser.add_argument('--use_gpu',\n type=bool,\n default=True,\n help='default use gpu.')\n parser.add_argument('--save_inference_model',"
+ },
+ {
+ "comment": "The code is defining a class \"InferModel\" which initializes variables such as name, threshold, and label_map. It also defines a method \"load_inference_model\" that loads the model file, configures GPU usage, and creates a predictor for inference. The code takes input arguments like model_dir, use_gpu, and other configuration parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/inference.py\":38-68",
+ "content": " type=str,\n default=None,\n help='save inference path')\n args = parser.parse_args()\n return args\nclass InferModel(object):\n \"\"\"lstm infer\"\"\"\n def __init__(self, cfg, name='ACTION'): \n name = name.upper()\n self.name = name\n self.threshold = cfg.INFER.threshold\n self.cfg = cfg\n self.label_map = load_class_file(cfg.MODEL.class_name_file)\n def load_inference_model(self, model_dir, use_gpu=True):\n \"\"\"model_init\n \"\"\"\n model_file = os.path.join(model_dir, \"model\")\n params_file = os.path.join(model_dir, \"params\")\n config = paddle.inference.Config(model_file, params_file)\n if use_gpu:\n config.enable_use_gpu(1024)\n else:\n config.disable_gpu()\n config.switch_ir_optim(True) # default true\n config.enable_memory_optim()\n # use zero copy\n config.switch_use_feed_fetch_ops(False)\n self.predictor = paddle.inference.create_predictor(config)"
+ },
+ {
+ "comment": "This code initializes input and output tensors for a multimodal video tagging scenario. It builds the input tensor from RGB frames, audio data, and text data, and retrieves the output tensor. The `preprocess_for_lod_data` function converts input data into a list of arrays with length indicators (LOD) for efficient handling.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/inference.py\":69-97",
+ "content": " # build input tensor and output tensor\n self.build_input_output()\n def build_input_output(self):\n \"\"\"build_input_output\n \"\"\"\n input_names = self.predictor.get_input_names()\n # input\n self.input_rgb_tensor = self.predictor.get_input_handle(input_names[0])\n self.input_audio_tensor = self.predictor.get_input_handle(input_names[1])\n self.input_text_tensor = self.predictor.get_input_handle(input_names[2])\n # output\n output_names = self.predictor.get_output_names()\n self.output_tensor = self.predictor.get_output_handle(output_names[0])\n def preprocess_for_lod_data(self, input):\n \"\"\"pre process\"\"\"\n input_arr = []\n input_lod = [0]\n start_lod = 0\n end_lod = 0\n for sub_item in input:\n end_lod = start_lod + len(sub_item)\n input_lod.append(end_lod)\n input_arr.extend(sub_item)\n start_lod = end_lod\n input_arr = np.array(input_arr)\n return input_arr, [input_lod]"
+ },
+ {
+ "comment": "In this code, a function named \"predict\" is defined. It uses a reader to process data for inference and iterates through each instance of the data. The instances contain RGB images, audio, text, and video IDs. These instances are preprocessed for LOD (Level Of Detail) data using the preprocess_for_lod_data method. The preprocessed data is then copied to corresponding input tensors (RGB, audio, and text) for inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/inference.py\":99-121",
+ "content": " def predict(self):\n \"\"\"predict\"\"\"\n infer_reader = get_reader(self.name, 'infer', self.cfg)\n probs = []\n video_ids = []\n label_map_inverse = {value: key for key, value in self.label_map.items()}\n for infer_iter, data in enumerate(infer_reader()):\n # video_id = [[items[-2], items[-1]] for items in data]\n rgb = [items[0] for items in data]\n audio = [items[1] for items in data]\n text = np.array([items[2] for items in data])\n videos = np.array([items[3] for items in data])\n rgb_arr, rgb_lod = self.preprocess_for_lod_data(rgb)\n audio_arr, audio_lod = self.preprocess_for_lod_data(audio)\n self.input_rgb_tensor.copy_from_cpu(rgb_arr.astype('float32'))\n self.input_rgb_tensor.set_lod(rgb_lod)\n self.input_audio_tensor.copy_from_cpu(audio_arr.astype('float32'))\n self.input_audio_tensor.set_lod(audio_lod)\n self.input_text_tensor.copy_from_cpu(text.astype('int64'))"
+ },
+ {
+ "comment": "The code defines an inference function that takes a set of videos and their associated labels, and returns the predicted labels for each video. It also includes functions to load class information from a file and parse command-line arguments. The inference function first runs the predictor on the input data, then extracts the output probabilities and corresponding video IDs. It checks that the number of video IDs matches the number of probabilities. Then, for each video-probability pair, it identifies the indices where the probability is above a certain threshold and uses these to determine the predicted labels. The resulting dictionary contains the video ID, as well as a list of tuples containing the label name and corresponding probability for each detected label.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/inference.py\":123-160",
+ "content": " self.predictor.run()\n output = self.output_tensor.copy_to_cpu()\n probs.extend(list(output))\n video_ids.extend(videos)\n assert len(video_ids) == len(probs)\n result = []\n for video_id, prob in zip(video_ids, probs):\n label_idx = list(np.where(prob >= self.threshold)[0])\n result.append({\n \"video_id\": video_id,\n \"labels\": [\n (label_map_inverse[str(idx)], float(prob[idx])) for idx in label_idx\n ]\n })\n return result\ndef load_class_file(class_file):\n \"\"\"\n load_class_file\n \"\"\"\n class_lines = open(class_file, 'r', encoding='utf8').readlines()\n class_dict = {}\n for i, line in enumerate(class_lines):\n tmp = line.strip().split('\\t')\n word = tmp[0]\n index = str(i)\n if len(tmp) == 2:\n index = tmp[1]\n class_dict[word] = index\n return class_dict\ndef infer(args):\n \"\"\"\n infer main\n \"\"\"\n config = parse_config(args.config)"
+ },
+ {
+ "comment": "This code snippet is from the \"inference.py\" file in the PaddleVideo MultimodalVideoTag application. It defines a function `infer` that performs model inference on input data and outputs the results. The code merges configs for the infer stage, prints them out, creates an InferModel object with those configs, loads the inference model from a given file (if provided), runs inference, and finally saves the results to a JSON file if requested.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/inference.py\":161-172",
+ "content": " infer_config = merge_configs(config, 'infer', vars(args))\n print_configs(infer_config, 'infer')\n infer_obj = InferModel(infer_config, name=args.model_name)\n infer_obj.load_inference_model(args.save_inference_model, use_gpu=args.use_gpu)\n rt = infer_obj.predict()\n if args.output:\n with open(args.output, 'w') as f:\n json.dump(rt, f, ensure_ascii=False, indent=4)\nif __name__ == \"__main__\":\n args = parse_args()\n infer(args)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/437b6fc8-ae66-4984-9f7e-ab5d28eb045c.json b/docs/doc/437b6fc8-ae66-4984-9f7e-ab5d28eb045c.json
new file mode 100644
index 000000000..352093176
--- /dev/null
+++ b/docs/doc/437b6fc8-ae66-4984-9f7e-ab5d28eb045c.json
@@ -0,0 +1,80 @@
+{
+ "summary": "This code imports libraries, defines a class for metrics calculation, retrieves data from ActivityNet API, compares results, creates DataFrames, evaluates proposals using AUC-RC, and calculates average recall. It extracts videos, computes proposal scores, IOU scores, handles exceptions, determines recall with thresholds, and efficiently computes IoU for target and candidate segments.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries, defines a class for calculating AR@N and AUC, and sets the API URL for accessing ActivityNet data. The class uses ground truth fields and proposal fields to compare results. Code is transferred from the ActivityNet GitHub repository.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":0-28",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport json\nimport numpy as np\nimport pandas as pd\nimport urllib.request as urllib2\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\nclass ANETproposal(object):\n \"\"\"\n This class is used for calculating AR@N and AUC;\n Code transfer from ActivityNet Gitub repository](https://github.com/activitynet/ActivityNet.git)\n \"\"\"\n GROUND_TRUTH_FIELDS = ['database', 'taxonomy', 'version']\n PROPOSAL_FIELDS = ['results', 'version', 'external_data']\n API = 'http://ec2-52-25-205-214.us-west-2.compute.amazonaws.com/challenge19/api.py'"
+ },
+ {
+ "comment": "Initializing the class with ground truth and proposal filenames as required, setting default parameters, and checking if both files exist.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":30-53",
+ "content": " def __init__(self,\n ground_truth_filename=None,\n proposal_filename=None,\n ground_truth_fields=GROUND_TRUTH_FIELDS,\n proposal_fields=PROPOSAL_FIELDS,\n tiou_thresholds=np.linspace(0.5, 0.95, 10),\n max_avg_nr_proposals=None,\n subset='validation',\n verbose=False,\n check_status=True):\n if not ground_truth_filename:\n raise IOError('Please input a valid ground truth file.')\n if not proposal_filename:\n raise IOError('Please input a valid proposal file.')\n self.subset = subset\n self.tiou_thresholds = tiou_thresholds\n self.max_avg_nr_proposals = max_avg_nr_proposals\n self.verbose = verbose\n self.gt_fields = ground_truth_fields\n self.pred_fields = proposal_fields\n self.recall = None\n self.avg_recall = None\n self.proposals_per_video = None\n self.check_status = check_status"
+ },
+ {
+ "comment": "This code retrieves blocked videos from a server, imports ground truth and proposals, and checks if the ground truth file is well formatted. It also prints information about the number of ground truth instances and proposals, as well as the fixed threshold for tiou score.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":54-76",
+ "content": " # Retrieve blocked videos from server.\n if self.check_status:\n self.blocked_videos = self.get_blocked_videos()\n else:\n self.blocked_videos = list()\n # Import ground truth and proposals.\n self.ground_truth, self.activity_index = self._import_ground_truth(\n ground_truth_filename)\n self.proposal = self._import_proposal(proposal_filename)\n if self.verbose:\n print('[INIT] Loaded annotations from {} subset.'.format(subset))\n nr_gt = len(self.ground_truth)\n print('\\tNumber of ground truth instances: {}'.format(nr_gt))\n nr_pred = len(self.proposal)\n print('\\tNumber of proposals: {}'.format(nr_pred))\n print('\\tFixed threshold for tiou score: {}'.format(\n self.tiou_thresholds))\n def _import_ground_truth(self, ground_truth_filename):\n \"\"\"\n Reads ground truth file, checks if it is well formatted, and returns\n the ground truth instances and the activity classes."
+ },
+ {
+ "comment": "This function reads a ground truth JSON file and returns a DataFrame containing the instances. It also returns a dictionary of class indices. The function checks if the input file has the required fields, skips videos not in the specified subset, and ignores blocked videos. If an activity label is not found in the activity_index, it adds it to the index and increments the counter.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":78-101",
+ "content": " Parameters:\n ground_truth_filename (str): full path to the ground truth json file.\n Returns:\n ground_truth (df): Data frame containing the ground truth instances.\n activity_index (dict): Dictionary containing class index.\n \"\"\"\n with open(ground_truth_filename, 'r') as fobj:\n data = json.load(fobj)\n # Checking format\n if not all([field in data.keys() for field in self.gt_fields]):\n raise IOError('Please input a valid ground truth file.')\n # Read ground truth data.\n activity_index, cidx = {}, 0\n video_lst, t_start_lst, t_end_lst, label_lst = [], [], [], []\n for videoid, v in data['database'].items():\n if self.subset != v['subset']:\n continue\n if videoid in self.blocked_videos:\n continue\n for ann in v['annotations']:\n if ann['label'] not in activity_index:\n activity_index[ann['label']] = cidx\n cidx += 1"
+ },
+ {
+ "comment": "The code reads a proposal file, checks its format and returns proposal instances in the form of a data frame. It also generates ground truth data by appending video IDs, start and end times, and labels to lists before creating a DataFrame. The function takes a string as input for the full path to the proposal JSON file and returns a data frame containing the proposal instances.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":102-129",
+ "content": " video_lst.append(videoid)\n t_start_lst.append(float(ann['segment'][0]))\n t_end_lst.append(float(ann['segment'][1]))\n label_lst.append(activity_index[ann['label']])\n ground_truth = pd.DataFrame({\n 'video-id': video_lst,\n 't-start': t_start_lst,\n 't-end': t_end_lst,\n 'label': label_lst\n })\n return ground_truth, activity_index\n def _import_proposal(self, proposal_filename):\n \"\"\"\n Reads proposal file, checks if it is well formatted, and returns\n the proposal instances.\n Parameters:\n proposal_filename (str): Full path to the proposal json file.\n Returns:\n proposal (df): Data frame containing the proposal instances.\n \"\"\"\n with open(proposal_filename, 'r') as fobj:\n data = json.load(fobj)\n # Checking format...\n if not all([field in data.keys() for field in self.pred_fields]):\n raise IOError('Please input a valid proposal file.')"
+ },
+ {
+ "comment": "The code reads predictions from a data source, extracts relevant information (video IDs, start and end timestamps, scores), stores them in a DataFrame, and defines two functions: one for evaluating proposal files by computing area under the average recall vs average number of proposals per video curve. The evaluation function calls another function to compute this metric using ground truth data and the stored proposal data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":131-157",
+ "content": " # Read predictions.\n video_lst, t_start_lst, t_end_lst = [], [], []\n score_lst = []\n for videoid, v in data['results'].items():\n if videoid in self.blocked_videos:\n continue\n for result in v:\n video_lst.append(videoid)\n t_start_lst.append(float(result['segment'][0]))\n t_end_lst.append(float(result['segment'][1]))\n score_lst.append(result['score'])\n proposal = pd.DataFrame({\n 'video-id': video_lst,\n 't-start': t_start_lst,\n 't-end': t_end_lst,\n 'score': score_lst\n })\n return proposal\n def evaluate(self):\n \"\"\"\n Evaluates a proposal file. To measure the performance of a\n method for the proposal task, we computes the area under the\n average recall vs average number of proposals per video curve.\n \"\"\"\n recall, avg_recall, proposals_per_video = self.average_recall_vs_avg_nr_proposals(\n self.ground_truth,"
+ },
+ {
+ "comment": "Calculates the area under the curve of recall vs average number of proposals for ActivityNet proposal task, writes result to file and stores recall, average recall, and proposals per video in class attributes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":158-181",
+ "content": " self.proposal,\n max_avg_nr_proposals=self.max_avg_nr_proposals,\n tiou_thresholds=self.tiou_thresholds)\n area_under_curve = np.trapz(avg_recall, proposals_per_video)\n if self.verbose:\n print('[RESULTS] Performance on ActivityNet proposal task.')\n with open(\"data/bmn/BMN_Test_results/auc_result.txt\",\n \"a\") as text_file:\n text_file.write(\n '\\tArea Under the AR vs AN curve: {}% \\n'.format(\n 100. * float(area_under_curve) /\n proposals_per_video[-1]))\n print('\\tArea Under the AR vs AN curve: {}%'.format(\n 100. * float(area_under_curve) / proposals_per_video[-1]))\n self.recall = recall\n self.avg_recall = avg_recall\n self.proposals_per_video = proposals_per_video\n def average_recall_vs_avg_nr_proposals(self,\n ground_truth,\n proposals,"
+ },
+ {
+ "comment": "This code defines a function that computes average recall for given average number of proposals per video. It takes ground truth and proposal data frames as input, along with optional tiou_thresholds. It returns recall and average_recall arrays.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":182-201",
+ "content": " max_avg_nr_proposals=None,\n tiou_thresholds=np.linspace(\n 0.5, 0.95, 10)):\n \"\"\"\n Computes the average recall given an average number of\n proposals per video.\n Parameters:\n ground_truth(df): Data frame containing the ground truth instances.\n Required fields: ['video-id', 't-start', 't-end']\n proposal(df): Data frame containing the proposal instances.\n Required fields: ['video-id, 't-start', 't-end', 'score']\n tiou_thresholds(1d-array | optional): array with tiou thresholds.\n Returns:\n recall(2d-array): recall[i,j] is recall at ith tiou threshold at the jth\n average number of average number of proposals per video.\n average_recall(1d-array): recall averaged over a list of tiou threshold.\n This is equivalent to recall.mean(axis=0).\n proposals_per_video(1d-array): average number of proposals per video."
+ },
+ {
+ "comment": "This code retrieves a list of videos, calculates the maximum average number of proposals per video, groups proposals and ground truth by video ID, and then computes Tiou scores between ground-truth instances and retrieved proposals for each video.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":202-230",
+ "content": " \"\"\"\n # Get list of videos.\n video_lst = ground_truth['video-id'].unique()\n if not max_avg_nr_proposals:\n max_avg_nr_proposals = float(\n proposals.shape[0]) / video_lst.shape[0]\n ratio = max_avg_nr_proposals * float(\n video_lst.shape[0]) / proposals.shape[0]\n # Adaptation to query faster\n ground_truth_gbvn = ground_truth.groupby('video-id')\n proposals_gbvn = proposals.groupby('video-id')\n # For each video, computes tiou scores among the retrieved proposals.\n score_lst = []\n total_nr_proposals = 0\n for videoid in video_lst:\n # Get ground-truth instances associated to this video.\n ground_truth_videoid = ground_truth_gbvn.get_group(videoid)\n this_video_ground_truth = ground_truth_videoid.loc[:, [\n 't-start', 't-end'\n ]].values\n # Get proposals for this video.\n try:\n proposals_videoid = proposals_gbvn.get_group(videoid)"
+ },
+ {
+ "comment": "This code block is part of a function that handles exceptions when dealing with video proposals and ground truth. It appends a zero matrix to the score list if there are no video proposals or ground truth data for the current video. If there are proposals, it sorts them by score in descending order and expands dimensions as necessary before proceeding.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":231-254",
+ "content": " except:\n n = this_video_ground_truth.shape[0]\n score_lst.append(np.zeros((n, 1)))\n continue\n this_video_proposals = proposals_videoid.loc[:,\n ['t-start', 't-end'\n ]].values\n if this_video_proposals.shape[0] == 0:\n n = this_video_ground_truth.shape[0]\n score_lst.append(np.zeros((n, 1)))\n continue\n # Sort proposals by score.\n sort_idx = proposals_videoid['score'].argsort()[::-1]\n this_video_proposals = this_video_proposals[sort_idx, :]\n if this_video_proposals.ndim != 2:\n this_video_proposals = np.expand_dims(this_video_proposals,\n axis=0)\n if this_video_ground_truth.ndim != 2:\n this_video_ground_truth = np.expand_dims(\n this_video_ground_truth, axis=0)"
+ },
+ {
+ "comment": "This code calculates average recall for a set of video proposals. It sets the number of proposals based on a ratio, computes IOU scores, and stores the results in lists. The average recall is computed using a predetermined maximum number of proposals and the total number of proposals retrieved, considering the variable length of videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":256-277",
+ "content": " nr_proposals = np.minimum(\n int(this_video_proposals.shape[0] * ratio),\n this_video_proposals.shape[0])\n total_nr_proposals += nr_proposals\n this_video_proposals = this_video_proposals[:nr_proposals, :]\n # Compute tiou scores.\n tiou = self.wrapper_segment_iou(this_video_proposals,\n this_video_ground_truth)\n score_lst.append(tiou)\n # Given that the length of the videos is really varied, we\n # compute the number of proposals in terms of a ratio of the total\n # proposals retrieved, i.e. average recall at a percentage of proposals\n # retrieved per video.\n # Computes average recall.\n pcn_lst = np.arange(1, 101) / 100.0 * (max_avg_nr_proposals * float(\n video_lst.shape[0]) / total_nr_proposals)\n matches = np.empty((video_lst.shape[0], pcn_lst.shape[0]))\n positives = np.empty(video_lst.shape[0])\n recall = np.empty((tiou_thresholds.shape[0], pcn_lst.shape[0]))"
+ },
+ {
+ "comment": "Code iterates over different tiou thresholds and positive scores, computing the number of true positives based on threshold and percentage of proposals. It calculates matches per video and computes recall for each set of matches.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":278-297",
+ "content": " # Iterates over each tiou threshold.\n for ridx, tiou in enumerate(tiou_thresholds):\n # Inspect positives retrieved per video at different\n # number of proposals (percentage of the total retrieved).\n for i, score in enumerate(score_lst):\n # Total positives per video.\n positives[i] = score.shape[0]\n # Find proposals that satisfies minimum tiou threshold.\n true_positives_tiou = score >= tiou\n # Get number of proposals as a percentage of total retrieved.\n pcn_proposals = np.minimum(\n (score.shape[1] * pcn_lst).astype(int), score.shape[1])\n for j, nr_proposals in enumerate(pcn_proposals):\n # Compute the number of matches for each percentage of the proposals\n matches[i, j] = np.count_nonzero(\n (true_positives_tiou[:, :nr_proposals]).sum(axis=1))\n # Computes recall given the set of matches per video."
+ },
+ {
+ "comment": "The function calculates recall and average recall for detected objects in videos, based on the number of true positives and total proposals. It also returns the average number of proposals per video. The second function retrieves a list of blocked videos from an API. The third function computes intersection over union between target and candidate segments efficiently.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":298-323",
+ "content": " recall[ridx, :] = matches.sum(axis=0) / positives.sum()\n # Recall is averaged.\n avg_recall = recall.mean(axis=0)\n # Get the average number of proposals per video.\n proposals_per_video = pcn_lst * (float(total_nr_proposals) /\n video_lst.shape[0])\n return recall, avg_recall, proposals_per_video\n def get_blocked_videos(self, api=API):\n api_url = '{}?action=get_blocked'.format(api)\n req = urllib2.Request(api_url)\n response = urllib2.urlopen(req)\n return json.loads(response.read())\n def wrapper_segment_iou(self, target_segments, candidate_segments):\n \"\"\"\n Compute intersection over union btw segments\n Parameters:\n target_segments(nd-array): 2-dim array in format [m x 2:=[init, end]]\n candidate_segments(nd-array): 2-dim array in format [n x 2:=[init, end]]\n Returns:\n tiou(nd-array): 2-dim array [n x m] with IOU ratio.\n Note: It assumes that candidate-segments are more scarce that target-segments"
+ },
+ {
+ "comment": "This function calculates the temporal intersection over union (TIOU) between a target segment and multiple candidate segments. If the dimensions of arguments are not 2, it raises a ValueError. It loops through each candidate segment, compares their starting and ending times with the target segment's times using np.maximum, and stores the TIOU results in a 2D array.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":324-348",
+ "content": " \"\"\"\n if candidate_segments.ndim != 2 or target_segments.ndim != 2:\n raise ValueError('Dimension of arguments is incorrect')\n n, m = candidate_segments.shape[0], target_segments.shape[0]\n tiou = np.empty((n, m))\n for i in range(m):\n tiou[:, i] = self.segment_iou(target_segments[i, :],\n candidate_segments)\n return tiou\n def segment_iou(self, target_segment, candidate_segments):\n \"\"\"\n Compute the temporal intersection over union between a\n target segment and all the test segments.\n Parameters:\n target_segment(1d-array): Temporal target segment containing [starting, ending] times.\n candidate_segments(2d-array): Temporal candidate segments containing N x [starting, ending] times.\n Returns:\n tiou(1d-array): Temporal intersection over union score of the N's candidate segments.\n \"\"\"\n tt1 = np.maximum(target_segment[0], candidate_segments[:, 0])"
+ },
+ {
+ "comment": "Computes intersection over union (IoU) of two segments by finding the minimum endpoints, calculating intersection and union, and dividing intersection by union.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ActivityNet/anet_prop.py\":349-358",
+ "content": " tt2 = np.minimum(target_segment[1], candidate_segments[:, 1])\n # Intersection including Non-negative overlap score.\n segments_intersection = (tt2 - tt1).clip(0)\n # Segment union.\n segments_union = (candidate_segments[:, 1] - candidate_segments[:, 0]) \\\n + (target_segment[1] - target_segment[0]) - segments_intersection\n # Compute overlap as the ratio of the intersection\n # over union of two segments.\n tIoU = segments_intersection.astype(float) / segments_union\n return tIoU"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/43a2a3f1-ec47-40cf-aae2-e3c51154519a.json b/docs/doc/43a2a3f1-ec47-40cf-aae2-e3c51154519a.json
new file mode 100644
index 000000000..f71aff03a
--- /dev/null
+++ b/docs/doc/43a2a3f1-ec47-40cf-aae2-e3c51154519a.json
@@ -0,0 +1,110 @@
+{
+ "summary": "This code optimizes PaddleVideo model performance by configuring environment variables for efficient training or export tasks, evaluates models, saves trained models, logs, and runs evaluation scripts.",
+ "details": [
+ {
+ "comment": "The code reads a file containing training parameters and parses the values using different functions. These parameters include the model name, Python version, GPU list, auto-cast settings, epoch number, batch size, pre-trained model, and training model name. The parsed values will be used for further processing in the script.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":0-29",
+ "content": "#!/bin/bash\nsource test_tipc/common_func.sh\nFILENAME=$1\n# MODE be one of ['lite_train_lite_infer' 'lite_train_whole_infer' 'whole_train_whole_infer', 'whole_infer']\nMODE=$2\ndataline=$(cat ${FILENAME})\n# parser params\nIFS=$'\\n'\nlines=(${dataline})\n# The training params\nmodel_name=$(func_parser_value \"${lines[1]}\")\npython=$(func_parser_value \"${lines[2]}\")\ngpu_list=$(func_parser_value \"${lines[3]}\")\ntrain_use_gpu_key=$(func_parser_key \"${lines[4]}\")\ntrain_use_gpu_value=$(func_parser_value \"${lines[4]}\")\nautocast_list=$(func_parser_value \"${lines[5]}\")\nautocast_key=$(func_parser_key \"${lines[5]}\")\nepoch_key=$(func_parser_key \"${lines[6]}\")\nepoch_num=$(func_parser_value \"${lines[6]}\")\nsave_model_key=$(func_parser_key \"${lines[7]}\")\ntrain_batch_key=$(func_parser_key \"${lines[8]}\")\ntrain_batch_value=$(func_parser_value \"${lines[8]}\")\npretrain_model_key=$(func_parser_key \"${lines[9]}\")\npretrain_model_value=$(func_parser_value \"${lines[9]}\")\ntrain_model_name=$(func_parser_value \"${lines[10]}\")\ntrain_param_key1=$(func_parser_key \"${lines[12]}\")"
+ },
+ {
+ "comment": "This code is parsing key-value pairs from different lines and assigning them to specific variables. The variables are used for trainer, pact, fpgm, distill, amp, evaluator, save_infer, and export weight configurations. This information will likely be utilized in subsequent parts of the script or program.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":30-55",
+ "content": "train_param_value1=$(func_parser_value \"${lines[12]}\")\ntrain_param_key2=$(func_parser_key \"${lines[11]}\")\ntrain_param_value2=$(func_parser_value \"${lines[11]}\")\ntrainer_list=$(func_parser_value \"${lines[14]}\")\ntrainer_norm=$(func_parser_key \"${lines[15]}\")\nnorm_trainer=$(func_parser_value \"${lines[15]}\")\npact_key=$(func_parser_key \"${lines[16]}\")\npact_trainer=$(func_parser_value \"${lines[16]}\")\nfpgm_key=$(func_parser_key \"${lines[17]}\")\nfpgm_trainer=$(func_parser_value \"${lines[17]}\")\ndistill_key=$(func_parser_key \"${lines[18]}\")\ndistill_trainer=$(func_parser_value \"${lines[18]}\")\namp_key=$(func_parser_key \"${lines[19]}\")\namp_trainer=$(func_parser_value \"${lines[19]}\")\ntrainer_key2=$(func_parser_key \"${lines[20]}\")\ntrainer_value2=$(func_parser_value \"${lines[20]}\")\neval_py=$(func_parser_value \"${lines[23]}\")\neval_key1=$(func_parser_key \"${lines[24]}\")\neval_value1=$(func_parser_value \"${lines[24]}\")\nsave_infer_key=$(func_parser_key \"${lines[27]}\")\nsave_infer_value=$(func_parser_value \"${lines[27]}\")\nexport_weight=$(func_parser_key \"${lines[28]}\")"
+ },
+ {
+ "comment": "The code parses various configuration values and keys from the lines of a script. It extracts normalization, quantization, and distillation settings; inference directory path; model directories for inference; whether to use GPU, MKLDNN, specify CPU threads, and batch size. The variables are assigned with these parsed values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":56-78",
+ "content": "norm_export=$(func_parser_value \"${lines[29]}\")\npact_export=$(func_parser_value \"${lines[30]}\")\nfpgm_export=$(func_parser_value \"${lines[31]}\")\ndistill_export=$(func_parser_value \"${lines[32]}\")\nexport_key1=$(func_parser_key \"${lines[33]}\")\nexport_value1=$(func_parser_value \"${lines[33]}\")\nexport_key2=$(func_parser_key \"${lines[34]}\")\nexport_value2=$(func_parser_value \"${lines[34]}\")\ninference_dir=$(func_parser_value \"${lines[35]}\")\n# parser inference model\ninfer_model_dir_list=$(func_parser_value \"${lines[36]}\")\ninfer_export_list=$(func_parser_value \"${lines[37]}\")\ninfer_is_quant=$(func_parser_value \"${lines[38]}\")\n# parser inference\ninference_py=$(func_parser_value \"${lines[39]}\")\nuse_gpu_key=$(func_parser_key \"${lines[40]}\")\nuse_gpu_list=$(func_parser_value \"${lines[40]}\")\nuse_mkldnn_key=$(func_parser_key \"${lines[41]}\")\nuse_mkldnn_list=$(func_parser_value \"${lines[41]}\")\ncpu_threads_key=$(func_parser_key \"${lines[42]}\")\ncpu_threads_list=$(func_parser_value \"${lines[42]}\")\nbatch_size_key=$(func_parser_key \"${lines[43]}\")"
+ },
+ {
+ "comment": "This code is parsing the function parameters from a configuration file. The batch size, use_trt, precision, infer model value, video directory path, log saving flag, and benchmark are being assigned to respective variables. A specific line number is obtained using grep command for a keyword \"to_static_train_benchmark_params\". Then it checks if the mode is set to \"klquant_whole_infer\" and processes the first and 17th lines of the configuration file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":79-103",
+ "content": "batch_size_list=$(func_parser_value \"${lines[43]}\")\nuse_trt_key=$(func_parser_key \"${lines[44]}\")\nuse_trt_list=$(func_parser_value \"${lines[44]}\")\nprecision_key=$(func_parser_key \"${lines[45]}\")\nprecision_list=$(func_parser_value \"${lines[45]}\")\ninfer_model_key=$(func_parser_key \"${lines[46]}\")\ninfer_model_value=$(func_parser_value \"${lines[46]}\")\nvideo_dir_key=$(func_parser_key \"${lines[47]}\")\ninfer_video_dir=$(func_parser_value \"${lines[47]}\")\nsave_log_key=$(func_parser_key \"${lines[48]}\")\nbenchmark_key=$(func_parser_key \"${lines[49]}\")\nbenchmark_value=$(func_parser_value \"${lines[49]}\")\ninfer_key1=$(func_parser_key \"${lines[50]}\")\ninfer_value1=$(func_parser_value \"${lines[50]}\")\nline_num=`grep -n -w \"to_static_train_benchmark_params\" $FILENAME | cut -d \":\" -f 1`\nto_static_key=$(func_parser_key \"${lines[line_num]}\")\nto_static_trainer=$(func_parser_value \"${lines[line_num]}\")\n# parser klquant_infer\nif [ ${MODE} = \"klquant_whole_infer\" ]; then\n dataline=$(awk 'NR==1 NR==17{print}' $FILENAME)\n lines=(${dataline})"
+ },
+ {
+ "comment": "The code is parsing the configuration file to extract specific values for different variables like model name, python version, inference model directory list, and more. These values are used later in the script to execute specific commands related to the test and train operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":104-124",
+ "content": " model_name=$(func_parser_value \"${lines[1]}\")\n python=$(func_parser_value \"${lines[2]}\")\n # parser inference model\n infer_model_dir_list=$(func_parser_value \"${lines[3]}\")\n infer_export_list=$(func_parser_value \"${lines[4]}\")\n infer_is_quant=$(func_parser_value \"${lines[5]}\")\n # parser inference\n inference_py=$(func_parser_value \"${lines[6]}\")\n use_gpu_key=$(func_parser_key \"${lines[7]}\")\n use_gpu_list=$(func_parser_value \"${lines[7]}\")\n use_mkldnn_key=$(func_parser_key \"${lines[8]}\")\n use_mkldnn_list=$(func_parser_value \"${lines[8]}\")\n cpu_threads_key=$(func_parser_key \"${lines[9]}\")\n cpu_threads_list=$(func_parser_value \"${lines[9]}\")\n batch_size_key=$(func_parser_key \"${lines[10]}\")\n batch_size_list=$(func_parser_value \"${lines[10]}\")\n use_trt_key=$(func_parser_key \"${lines[11]}\")\n use_trt_list=$(func_parser_value \"${lines[11]}\")\n precision_key=$(func_parser_key \"${lines[12]}\")\n precision_list=$(func_parser_value \"${lines[12]}\")\n infer_model_key=$(func_parser_key \"${lines[13]}\")"
+ },
+ {
+ "comment": "This code sets variables for video directory, log path, inference functions, and other parameters. It then loops through various conditions to perform inferences with different combinations of GPU and MKLDNN usage, while keeping track of the results in the specified log file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":125-156",
+ "content": " video_dir_key=$(func_parser_key \"${lines[14]}\")\n infer_video_dir=$(func_parser_value \"${lines[14]}\")\n save_log_key=$(func_parser_key \"${lines[15]}\")\n benchmark_key=$(func_parser_key \"${lines[16]}\")\n benchmark_value=$(func_parser_value \"${lines[16]}\")\n infer_key1=$(func_parser_key \"${lines[17]}\")\n infer_value1=$(func_parser_value \"${lines[17]}\")\nfi\nLOG_PATH=\"./test_tipc/output/${model_name}/${MODE}\"\nmkdir -p ${LOG_PATH}\nstatus_log=\"${LOG_PATH}/results_python.log\"\nfunction func_inference(){\n IFS='|'\n _python=$1\n _script=$2\n _model_dir=$3\n _log_path=$4\n _video_dir=$5\n _flag_quant=$6\n _gpu=$7\n # inference\n for use_gpu in ${use_gpu_list[*]}; do\n if [ ${use_gpu} = \"False\" ] || [ ${use_gpu} = \"cpu\" ]; then\n for use_mkldnn in ${use_mkldnn_list[*]}; do\n if [[ ${use_mkldnn} = \"False\" ]] && [[ ${_flag_quant} = \"True\" ]]; then\n continue\n fi\n for threads in ${cpu_threads_list[*]}; do\n for batch_size in ${batch_size_list[*]}; do"
+ },
+ {
+ "comment": "This code is iterating over a list of precision values, checking conditions to decide whether to continue or skip the current iteration. It sets the appropriate log path, creates directories if needed, and calls functions to set parameters for inference data and benchmarking.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":157-169",
+ "content": " for precision in ${precision_list[*]}; do\n if [[ ${use_mkldnn} = \"False\" ]] && [[ ${precision} = \"fp16\" ]]; then\n continue\n fi # skip when enable fp16 but disable mkldnn\n if [[ ${_flag_quant} = \"True\" ]] && [[ ${precision} != \"int8\" ]]; then\n continue\n fi # skip when quant model inference but precision is not int8\n set_precision=$(func_set_params \"${precision_key}\" \"${precision}\")\n _save_log_path=\"${_log_path}/python_infer_cpu_gpus_${_gpu}_usemkldnn_${use_mkldnn}_threads_${threads}_precision_${precision}_batchsize_${batch_size}.log\"\n mkdir -p ${_log_path}\n set_infer_data=$(func_set_params \"${video_dir_key}\" \"${infer_video_dir}\")\n set_benchmark=$(func_set_params \"${benchmark_key}\" \"${benchmark_value}\")"
+ },
+ {
+ "comment": "The code is setting variables, constructing a command using environment variables, and executing it. It then checks the status of the execution and logs the output for later inspection. This appears to be part of a loop that's running multiple tests or experiments with varying parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":170-180",
+ "content": " set_batchsize=$(func_set_params \"${batch_size_key}\" \"${batch_size}\")\n set_cpu_threads=$(func_set_params \"${cpu_threads_key}\" \"${threads}\")\n set_model_dir=$(func_set_params \"${infer_model_key}\" \"${_model_dir}/${infer_model_value}\")\n set_infer_params1=$(func_set_params \"${infer_key1}\" \"${_model_dir}/${infer_value1}\")\n command=\"${_python} ${_script} ${use_gpu_key}=${use_gpu} ${use_mkldnn_key}=${use_mkldnn} ${set_cpu_threads} ${set_model_dir} ${set_batchsize} ${set_infer_data} ${set_benchmark} ${set_precision} ${set_infer_params1} > ${_save_log_path} 2>&1 \"\n eval $command\n last_status=${PIPESTATUS[0]}\n eval \"cat ${_save_log_path}\"\n status_check $last_status \"${command}\" \"${status_log}\" \"${model_name}\" \"${_save_log_path}\"\n done\n done"
+ },
+ {
+ "comment": "This code snippet is checking various conditions for model inference using different parameters like GPU usage, precision, and batch size. It iterates through a list of options to set up the necessary configurations for logging and execution. The purpose seems to be running inference tests with varying settings to optimize performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":181-197",
+ "content": " done\n done\n elif [ ${use_gpu} = \"True\" ] || [ ${use_gpu} = \"gpu\" ]; then\n for use_trt in ${use_trt_list[*]}; do\n for precision in ${precision_list[*]}; do\n if [[ ${_flag_quant} = \"False\" ]] && [[ ${precision} =~ \"int8\" ]]; then\n continue\n fi\n if [[ ${precision} =~ \"fp16\" || ${precision} =~ \"int8\" ]] && [[ ${use_trt} = \"False\" ]]; then\n continue\n fi\n if [[ ${use_trt} = \"False\" || ${precision} =~ \"int8\" ]] && [[ ${_flag_quant} = \"True\" ]]; then\n continue\n fi\n for batch_size in ${batch_size_list[*]}; do\n _save_log_path=\"${_log_path}/python_infer_gpu_gpus_${_gpu}_usetrt_${use_trt}_precision_${precision}_batchsize_${batch_size}.log\"\n set_infer_data=$(func_set_params \"${video_dir_key}\" \"${infer_video_dir}\")"
+ },
+ {
+ "comment": "This code sets parameters for benchmark, batch size, tensorrt usage, precision, model directory, and infer params1. It then executes a command with these parameters to run inference, saves the log, and checks the status of the execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":199-211",
+ "content": " set_benchmark=$(func_set_params \"${benchmark_key}\" \"${benchmark_value}\")\n set_batchsize=$(func_set_params \"${batch_size_key}\" \"${batch_size}\")\n set_tensorrt=$(func_set_params \"${use_trt_key}\" \"${use_trt}\")\n set_precision=$(func_set_params \"${precision_key}\" \"${precision}\")\n set_model_dir=$(func_set_params \"${infer_model_key}\" \"${_model_dir}/${infer_model_value}\")\n set_infer_params1=$(func_set_params \"${infer_key1}\" \"${_model_dir}/${infer_value1}\")\n command=\"${_python} ${_script} ${use_gpu_key}=${use_gpu} ${set_tensorrt} ${set_precision} ${set_model_dir} ${set_batchsize} ${set_infer_data} ${set_benchmark} ${set_infer_params1} > ${_save_log_path} 2>&1 \"\n eval $command\n last_status=${PIPESTATUS[0]}\n eval \"cat ${_save_log_path}\"\n status_check $last_status \"${command}\" \"${status_log}\" \"${model_name}\" \"${_save_log_path}\""
+ },
+ {
+ "comment": "This code is part of a script that tests and runs inference models using PaddleVideo. It checks the hardware being used (CPU or GPU) and sets appropriate environment variables accordingly. It then iterates through each inference model, running them with specific exported weights and saving the output logs for further analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":213-242",
+ "content": " done\n done\n done\n else\n echo \"Does not support hardware other than CPU and GPU Currently!\"\n fi\n done\n}\nif [ ${MODE} = \"whole_infer\" ] || [ ${MODE} = \"klquant_whole_infer\" ]; then\n GPUID=$3\n if [ ${#GPUID} -le 0 ];then\n env=\" \"\n else\n env=\"export CUDA_VISIBLE_DEVICES=${GPUID}\"\n fi\n set CUDA_VISIBLE_DEVICES\n eval $env\n export Count=0\n IFS=\"|\"\n infer_run_exports=(${infer_export_list})\n infer_quant_flag=(${infer_is_quant})\n for infer_model in ${infer_model_dir_list[*]}; do\n # run export\n if [ ${infer_run_exports[Count]} != \"null\" ];then\n save_infer_dir=$(dirname $infer_model)\n set_export_weight=$(func_set_params \"${export_weight}\" \"${infer_model}\")\n set_save_infer_key=$(func_set_params \"${save_infer_key}\" \"${save_infer_dir}\")\n export_log_path=\"${LOG_PATH}_export_${Count}.log\"\n export_cmd=\"${python} ${infer_run_exports[Count]} ${set_export_weight} ${set_save_infer_key} > ${export_log_path} 2>&1 \""
+ },
+ {
+ "comment": "This code is iterating through a list of GPUs, setting the visible CUDA devices accordingly and running inference for each GPU. It also checks if exporting is needed, saves the infer directory, runs quantized inference if it's a klquant_infer mode, and keeps track of the count to ensure all GPUs are considered. If a GPU list item is -1, it uses no GPU, otherwise, it sets the environment for that specific GPU. The code block also checks whether the GPU count is less than or equal to 15 to avoid potential issues with larger lists.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":243-273",
+ "content": " echo ${infer_run_exports[Count]}\n eval $export_cmd\n echo $export_cmd\n status_export=$?\n status_check $status_export \"${export_cmd}\" \"${status_log}\" \"${model_name}\" \"${export_log_path}\"\n else\n save_infer_dir=${infer_model}\n fi\n #run inference\n is_quant=${infer_quant_flag[Count]}\n if [ ${MODE} = \"klquant_infer\" ]; then\n is_quant=\"True\"\n fi\n func_inference \"${python}\" \"${inference_py}\" \"${save_infer_dir}\" \"${LOG_PATH}\" \"${infer_video_dir}\" ${is_quant} \"${gpu}\"\n Count=$(($Count + 1))\n done\nelse\n IFS=\"|\"\n export Count=0\n USE_GPU_KEY=(${train_use_gpu_value})\n for gpu in ${gpu_list[*]}; do\n train_use_gpu=${USE_GPU_KEY[Count]}\n Count=$(($Count + 1))\n ips=\"\"\n if [ ${gpu} = \"-1\" ];then\n env=\"\"\n elif [ ${#gpu} -le 1 ];then\n env=\"export CUDA_VISIBLE_DEVICES=${gpu}\"\n eval ${env}\n elif [ ${#gpu} -le 15 ];then"
+ },
+ {
+ "comment": "This code is setting up environment variables for parallel GPU usage, iterating through different autocast and trainer configurations to execute training or export tasks based on the provided key values. The flag_quant variable tracks whether quantization is required for a particular configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":274-302",
+ "content": " IFS=\",\"\n array=(${gpu})\n env=\"export CUDA_VISIBLE_DEVICES=${array[0]}\"\n IFS=\"|\"\n else\n IFS=\";\"\n array=(${gpu})\n ips=${array[0]}\n gpu=${array[1]}\n IFS=\"|\"\n env=\" \"\n fi\n for autocast in ${autocast_list[*]}; do\n if [ ${autocast} = \"fp16\" ]; then\n set_amp_config=\"--amp --amp_level 'O2'\"\n else\n set_amp_config=\" \"\n fi\n for trainer in ${trainer_list[*]}; do\n flag_quant=False\n if [ ${trainer} = ${pact_key} ]; then\n run_train=${pact_trainer}\n run_export=${pact_export}\n flag_quant=True\n elif [ ${trainer} = \"${fpgm_key}\" ]; then\n run_train=${fpgm_trainer}\n run_export=${fpgm_export}\n elif [ ${trainer} = \"${distill_key}\" ]; then\n run_train=${distill_trainer}"
+ },
+ {
+ "comment": "This code uses conditional statements to assign values to `run_train` and `run_export` variables based on the value of `trainer`. It handles multiple scenarios, including cases with specific keys, and triggers \"to_static\" logic in 'train.py' when needed. If `run_train` is assigned as null, it continues without executing further.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":303-324",
+ "content": " run_export=${distill_export}\n elif [ ${trainer} = ${amp_key} ]; then\n run_train=${amp_trainer}\n run_export=${norm_export}\n elif [[ ${trainer} = ${trainer_key2} ]]; then\n run_train=${trainer_value2}\n run_export=${export_value2}\n # In case of @to_static, we re-used norm_traier,\n # but append \"-o to_static=True\" for config\n # to trigger \"to_static\" logic in 'train.py'\n elif [ ${trainer} = \"${to_static_key}\" ]; then\n run_train=\"${norm_trainer} ${to_static_trainer}\"\n run_export=${norm_export}\n else\n run_train=${norm_trainer}\n run_export=${norm_export}\n fi\n if [ ${run_train} = \"null\" ]; then\n continue\n fi\n if [[ ${MODE} != \"benchmark_train\" ]] && [[ ! ${MODE} =~ \"whole_train\" ]]; then"
+ },
+ {
+ "comment": "This code is setting up parameters for model training and inference. It appends --max_iters=30 and --log_interval=1 to the run_train string for better data output, sets autocast, epoch, pretrain values, and checks if MODE includes \"whole_train\" to set certain variables to empty strings or nulls. The code also uses func_set_params to set batch size and train parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":325-346",
+ "content": " # \u8bad\u7ec3\u53c2\u6570\u672b\u5c3e\u52a0\u4e0a--max_iters=30\u548c--log_interval=1\uff0c\u4ee5\u4fbf\u8fd0\u884c\u5e76\u8f93\u51fa\u8db3\u91cf\u6570\u636e\n run_train=${run_train}\" --max_iters=30\"\n fi\n set_autocast=$(func_set_params \"${autocast_key}\" \"${autocast}\")\n set_epoch=$(func_set_params \"${epoch_key}\" \"${epoch_num}\")\n if [[ $MODE =~ \"whole_train\" ]]; then\n set_epoch=\"\"\n fi\n set_pretrain=$(func_set_params \"${pretrain_model_key}\" \"${pretrain_model_value}\")\n if [[ $MODE =~ \"whole_train\" ]]; then\n train_batch_key=\"\"\n train_batch_value=\"\"\n fi\n set_batchsize=$(func_set_params \"${train_batch_key}\" \"${train_batch_value}\")\n if [[ $MODE =~ \"whole_train\" ]]; then\n train_param_key1=\"\"\n train_param_value1=\"\"\n fi\n set_train_params1=$(func_set_params \"${train_param_key1}\" \"${train_param_value1}\")\n if [[ $MODE =~ \"whole_train\" ]]; then"
+ },
+ {
+ "comment": "This code sets up parameters for training and inference of a PaddleVideo model. It determines whether the training is on a single machine or multiple machines based on the number of IPs provided. Depending on this, it sets the number of nodes, logs information accordingly, and loads pre-training from normal training if the current trainer is PACT or FPGM.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":347-366",
+ "content": " train_param_key2=\"\"\n train_param_value2=\"\"\n fi\n set_train_params2=$(func_set_params \"${train_param_key2}\" \"${train_param_value2}\")\n set_use_gpu=$(func_set_params \"${train_use_gpu_key}\" \"${train_use_gpu}\")\n if [ ${#ips} -le 15 ];then\n # len(ips)<=15, single machine\n nodes=1\n save_log=\"${LOG_PATH}/${trainer}_gpus_${gpu}_autocast_${autocast}_nodes_${nodes}\"\n else\n # if length of ips > 15, then it is seen as multi-machine\n # 15 is the min length of ips info for multi-machine: 0.0.0.0,0.0.0.0\n IFS=\",\"\n ips_array=(${ips})\n IFS=\"|\"\n nodes=${#ips_array[@]}\n save_log=\"${LOG_PATH}/${trainer}_gpus_${gpu}_autocast_${autocast}_nodes_${nodes}\"\n fi\n # load pretrain from norm training if current trainer is pact or fpgm trainer"
+ },
+ {
+ "comment": "This code checks if the trainer is either 'pact_key' or 'fpgm_key', and if the number of nodes is less than or equal to 1. If true, it sets the 'set_pretrain' variable to 'load_norm_train_model'. The code then determines the appropriate command based on whether the number of GPUs is 2 or less (train with CPU or single GPU), up to 15 (train with multi-GPU), or more than 15 (train with multiple machines). The command uses PaddlePaddle's distributed training capabilities.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":367-377",
+ "content": " if ([ ${trainer} = ${pact_key} ] || [ ${trainer} = ${fpgm_key} ]) && [ ${nodes} -le 1 ]; then\n set_pretrain=\"${load_norm_train_model}\"\n fi\n set_save_model=$(func_set_params \"${save_model_key}\" \"${save_log}\")\n if [ ${#gpu} -le 2 ];then # train with cpu or single gpu\n cmd=\"${python} ${run_train} ${set_amp_config} ${set_use_gpu} ${set_save_model} ${set_epoch} ${set_pretrain} ${set_batchsize} ${set_train_params1} ${set_train_params2} > ${LOG_PATH}/train.log 2>&1\"\n elif [ ${#ips} -le 15 ];then # train with multi-gpu\n cmd=\"${python} -B -m paddle.distributed.launch --devices=\\\"${gpu}\\\" ${run_train} ${set_amp_config} ${set_use_gpu} ${set_save_model} ${set_epoch} ${set_pretrain} ${set_batchsize} ${set_train_params1} ${set_train_params2} > ${LOG_PATH}/train.log 2>&1\"\n else # train with multi-machine\n cmd=\"${python} -B -m paddle.distr"
+ },
+ {
+ "comment": "This code snippet is running a training script for a PaddleVideo model. It sets parameters, evaluates pre-trained models, and saves the trained models. The script also displays logs for benchmarking and checks the status of the operation. If there's a single node and trainer, it loads a norm-train model for further usage. Finally, it runs an evaluation script if specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":377-394",
+ "content": "ibuted.launch --ips=${ips} --devices=\\\"${gpu}\\\" ${run_train} ${set_amp_config} ${set_use_gpu} ${set_save_model} ${set_pretrain} ${set_epoch} ${set_batchsize} ${set_train_params1} ${set_train_params2} > ${LOG_PATH}/train.log 2>&1\"\n fi\n # run train\n eval $cmd\n # display log for benchmark train\n eval \"cat ${LOG_PATH}/train.log\"\n eval \"cat ${LOG_PATH}/train.log >> ${save_log}.log\"\n status_check $? \"${cmd}\" \"${status_log}\" \"${model_name}\" \"${save_log}.log\"\n # set_eval_pretrain=$(func_set_params \"${pretrain_model_key}\" \"${save_log}/${train_model_name}\")\n # save norm trained models to set pretrain for pact training and fpgm training\n if [ [${trainer} = ${trainer_norm}] ] && [ [${nodes} -le 1] ]; then\n load_norm_train_model=${set_eval_pretrain}\n fi\n # run eval\n if [ ${eval_py} != \"null\" ]; then\n real_model_name=${model_name/PP-/pp}"
+ },
+ {
+ "comment": "The code sets the evaluation parameters and prepares a command to evaluate a model using specified inputs. If the MODE includes \"lite_infer\" and train_param_key1 is not null, it appends additional parameters to the command. Finally, it runs the command and checks the status of the evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":395-409",
+ "content": " set_eval_params1=$(func_set_params \"${eval_key1}\" \"${save_log}/${real_model_name}_epoch_00001.pdparams\")\n eval_log_path=\"${LOG_PATH}/${trainer}_gpus_${gpu}_autocast_${autocast}_nodes_${nodes}_eval.log\"\n if [[ $MODE =~ \"lite_infer\" ]] && [[ ${train_param_key1} != \"null\" ]]; then\n eval_cmd=\"${python} ${eval_py} ${set_use_gpu} ${set_eval_params1} ${train_param_key1}=${train_param_value1} > ${eval_log_path} 2>&1 \"\n else\n eval_cmd=\"${python} ${eval_py} ${set_use_gpu} ${set_eval_params1} > ${eval_log_path} 2>&1 \"\n fi\n eval $eval_cmd\n status_check $? \"${eval_cmd}\" \"${status_log}\" \"${model_name}\" \"${eval_log_path}\"\n fi\n # run export model\n if [ ${run_export} != \"null\" ]; then\n save_infer_path=\"${save_log}\"\n real_model_name=${model_name/PP-/pp}\n "
+ },
+ {
+ "comment": "This code is setting up variables for exporting weights, saving inference key, and defining the export command. It then executes the export command and checks its status before running inference. If an inference directory is provided, it sets the inference model directory accordingly. Finally, it calls a function for inference processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":409-425",
+ "content": " set_export_weight=$(func_set_params \"${export_weight}\" \"${save_log}/${real_model_name}_epoch_00001.pdparams\")\n set_save_infer_key=$(func_set_params \"${save_infer_key}\" \"${save_log}\")\n export_log_path=\"${LOG_PATH}/${trainer}_gpus_${gpu}_autocast_${autocast}_nodes_${nodes}_export.log\"\n export_cmd=\"${python} ${run_export} ${set_export_weight} ${set_save_infer_key} > ${export_log_path} 2>&1 \"\n eval $export_cmd\n status_check $? \"${export_cmd}\" \"${status_log}\" \"${model_name}\" \"${export_log_path}\"\n #run inference\n eval $env\n save_infer_path=\"${save_log}\"\n if [ ${inference_dir} != \"null\" ] && [ ${inference_dir} != '##' ]; then\n infer_model_dir=${save_infer_path}\n else\n infer_model_dir=${save_infer_path}\n fi\n func_inference "
+ },
+ {
+ "comment": "This code snippet is a bash script that iterates through trainers, autocast options, and GPUs. It sets CUDA_VISIBLE_DEVICES to empty if the current mode is inference. The purpose is likely to set up an environment for training or inference based on different configurations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/test_train_inference_python.sh\":425-432",
+ "content": "\"${python}\" \"${inference_py}\" \"${infer_model_dir}\" \"${LOG_PATH}\" \"${infer_video_dir}\" \"${flag_quant}\" \"${gpu}\"\n eval \"unset CUDA_VISIBLE_DEVICES\"\n fi\n done # done with: for trainer in ${trainer_list[*]}; do\n done # done with: for autocast in ${autocast_list[*]}; do\n done # done with: for gpu in ${gpu_list[*]}; do\nfi # end if [ ${MODE} = \"infer\" ]; then"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/43c05170-0416-4c9f-b927-084bc4f984f2.json b/docs/doc/43c05170-0416-4c9f-b927-084bc4f984f2.json
new file mode 100644
index 000000000..4cfefe453
--- /dev/null
+++ b/docs/doc/43c05170-0416-4c9f-b927-084bc4f984f2.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The code documents a SlowFast_FasterRCNN model for action detection tasks, providing installation and processing instructions. It trains and tests the model with PaddleDetection and exports it for inference, using GPU acceleration and disabling TensorRT optimization.",
+ "details": [
+ {
+ "comment": "This code provides documentation for the SlowFast_FasterRCNN model, a high-precision video model used for action detection tasks. It takes human detection results and video frames as input, uses the SlowFast model to extract spatiotemporal features, and employs FasterRCNN's head to obtain the actions and positions of humans in the frame. Users need to install additional dependencies before getting started.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/detection/SlowFast_FasterRCNN_en.md\":0-23",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/detection/SlowFast_FasterRCNN.md) | English\n# SlowFast_FasterRCNN\n## Contents\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Inference](#Inference)\nBefore getting started, you need to install additional dependencies as follows:\n```bash\npython -m pip install moviepy\npython -m pip install et_xmlfile\npython -m pip install paddledet\n```\n## Introduction\nThe [SlowFast](https://github.com/PaddlePaddle/PaddleVideo/blob/develop/docs/zh-CN/model_zoo/recognition/slowfast.md) model is one of the high-precision models in the video field. For action detection task, it is also neccessary to detect the person in current frame. Therefore, the SlowFast_FasterRCNN model takes human detection results and video frames as input, extracts spatiotemporal features through the SlowFast model, and then uses FasterRCNN's head gets the actions and positions of humans in the frame.\nThe corresponding AI Studio Notebook Link\uff1a[\u57fa\u4e8eSlowFast+FasterRCNN\u7684\u52a8\u4f5c\u8bc6\u522b](https://aistudio.baidu.com/aistudio/projectdetail/3267637?contributionType=1)"
+ },
+ {
+ "comment": "The provided code is a set of instructions for downloading videos, annotations, and proposals as well as cutting videos and extracting frames from the AVA dataset. This dataset contains 430 videos annotated in 1 second intervals for action detection using SlowFast Networks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/detection/SlowFast_FasterRCNN_en.md\":25-63",
+ "content": "For details, please refer to the paper [SlowFast Networks for Video Recognition](https://arxiv.org/pdf/1812.03982.pdf).\n## Data\nWe use [AVA dataset](https://research.google.com/ava/download.html) for action detection. The AVA v2.2 dataset contains 430 videos split into 235 for training, 64 for validation, and 131 for test. Each video has 15 minutes annotated in 1 second intervals.\n### 1 Dowload Videos\n```\nbash download_videos.sh\n```\n### 2 Download Annotations\n```\nbash download_annotations.sh\n```\n### 3 Download Proposals\n```\nbash fetch_ava_proposals.sh\n```\n### 4 Cut Videos\n```\nbash cut_videos.sh\n```\n### 5 Extract Frames\n```\nbash extract_rgb_frames.sh\n```\nFor AVA v2.1, there is a simple introduction to some key files\uff1a\n* 'ava_videos_15min_frames' dir stores video frames extracted with FPS as the frame rate\uff1b\n* 'ava_train_v2.1.csv' file stores the trainning annotations\uff1b\n* 'ava_train_excluded_timestamps_v2.1.csv' file stores excluded timestamps\uff1b\n* 'ava_dense_proposals_train.FAIR.recall_93.9.pkl' file stores humans' bboxes and scores of key frames\uff1b"
+ },
+ {
+ "comment": "This code describes the training and testing procedures for a SlowFast model using Faster RCNN on AVA dataset. The training process requires a config file, pre-trained model weights, and evaluates the model during training with the --validate flag. Testing is done based on the best model provided with specifications like architecture, depth, pretrain model, frame length, sample rate, MAP, AVA version, and a link to the trained model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/detection/SlowFast_FasterRCNN_en.md\":64-89",
+ "content": "* 'ava_action_list_v2.1_for_activitynet_2018.pbtxt' file stores\u4e3a action list.\n## Train\n* `-c`: config file path;\n* `-w`: weights of model. The pretrained model can be downloaded from the table below;\n* `--validate`: evaluate model during training.\n```\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\npython -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=logdir.ava main.py --validate -w paddle.init_param.pdparams -c configs/detection/ava/ava.yaml\n```\n## Test\nTest model based on the best model:\n```\npython main.py --test \\\n -w output/AVA_SlowFast_FastRcnn/AVA_SlowFast_FastRcnn_best.pdparams \\\n -c configs/detection/ava/ava.yaml\n```\n| architecture | depth | Pretrain Model | frame length x sample rate | MAP | AVA version | model |\n| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |------------- |\n| SlowFast | R50 | [Kinetics 400](https://videotag.bj.bcebos.com/PaddleVideo/SlowFast/SlowFast_8*8.pdparams) | 8 x 8 | 23.2 | 2.1 | [`link`](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/SlowFastRCNN_AVA.pdparams) |"
+ },
+ {
+ "comment": "In this code, the inference process is outlined for an action detection project using SlowFast+FasterRCNN model. It requires installing PaddleDetection and downloading a detection model from provided URL. The \"export_model\" script prepares the model for inference, while the \"predict.py\" script performs inference based on the exported model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/detection/SlowFast_FasterRCNN_en.md\":92-125",
+ "content": "## Inference\nThe action detection of this project is divided into two stages. In the first stage, humans' proposals are obtained, and then input into the SlowFast+FasterRCNN model for action recognition.\nFor human detection\uff0cyou can use the trained model in [PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection).\nInstall PaddleDetection:\n```\ncd PaddleDetection/\npip install -r requirements.txt\n!python setup.py install\n```\nDownload detection model:\n```\n# faster_rcnn_r50_fpn_1x_coco as an example\nwget https://paddledet.bj.bcebos.com/models/faster_rcnn_r50_fpn_1x_coco.pdparams\n```\nexport model:\n```\npython tools/export_model.py \\\n -c configs/detection/ava/ava.yaml \\\n -o inference_output \\\n -p output/AVA_SlowFast_FastRcnn/AVA_SlowFast_FastRcnn_best.pdparams\n```\ninference based on the exported model:\n```\npython tools/predict.py \\\n -c configs/detection/ava/ava.yaml \\\n --input_file \"data/-IELREHXDEMO.mp4\" \\\n --model_file \"inference_output/AVA_SlowFast_FastRcnn.pdmodel\" \\\n --params_file \"inference_output/AVA_SlowFast_FastRcnn.pdiparams\" \\"
+ },
+ {
+ "comment": "The code sets `use_gpu` to True and `use_tensorrt` to False. This means the model will use GPU acceleration and not utilize TensorRT for optimizing performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/detection/SlowFast_FasterRCNN_en.md\":126-128",
+ "content": " --use_gpu=True \\\n --use_tensorrt=False\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4491a653-9505-4cce-9c8b-ca3be2d18b07.json b/docs/doc/4491a653-9505-4cce-9c8b-ca3be2d18b07.json
new file mode 100644
index 000000000..d825e1d95
--- /dev/null
+++ b/docs/doc/4491a653-9505-4cce-9c8b-ca3be2d18b07.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code employs machine learning and deep learning for Baidu Cloud's action detection system, including preprocessing, feature extraction, model application, and time-tracked execution. It configures PPTSM model, predicts features from data, creates a video feature dictionary, loads pre-existing features, checks shapes, and writes results to JSON file.",
+ "details": [
+ {
+ "comment": "This code is for the Baidu Cloud action detection system, which uses machine learning and deep learning models to classify actions from both audio and image inputs. It includes utilities for preprocessing data, extracting features, and applying various models including image, audio, and propensity models. The code also has a logger module to log processing time information. A class ActionDetection is defined which likely handles the overall action detection process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/action.py\":0-43",
+ "content": "#!./python27-gcc482/bin/python\n# coding: utf-8\n\"\"\"\nBAIDU CLOUD action\n\"\"\"\nimport os\nimport sys\nimport pickle\nimport json\nimport time\nimport functools\nimport numpy as np\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config, print_configs\nimport mfcc.feature_extractor as mfcc_extractor\nimport models.pptsm_infer as image_model\nimport models.audio_infer as audio_model\nimport models.bmn_infer as prop_model\nimport models.lstm_infer as classify_model\nimport logger\nlogger = logger.Logger()\ndef record_time_info(func):\n \"\"\"decorator func to log cost time for func\n \"\"\"\n @functools.wraps(func)\n def timer(*args):\n \"\"\"log cost time for func\n \"\"\"\n logger.info(\"function [{}] processing ...\".format(func.__name__))\n start_time = time.time()\n retval = func(*args)\n cost_time = round(time.time() - start_time, 5)\n logger.info(\"function [{}] run time: {:.2f} min\".format(func.__name__, cost_time / 60))\n return retval\n return timer\nclass ActionDetection(object):"
+ },
+ {
+ "comment": "This code initializes a ModelPredict object with various configuration settings and properties. It reads configuration data from a specified file, prints relevant information for debugging, and sets attributes related to model components such as BMN_ONLY, LSTM_ONLY, and PCM_ONLY. If LSTM_ONLY is set to true, it populates a prop_dict with video names and their corresponding BMN results for later use in the load_model function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/action.py\":44-70",
+ "content": " \"\"\"ModelPredict\"\"\"\n def __init__(self, cfg_file=\"configs/configs.yaml\"):\n cfg = parse_config(cfg_file)\n self.configs = cfg\n print_configs(self.configs, \"Infer\")\n name = 'COMMON'\n self.DEBUG = cfg[name]['DEBUG']\n self.BMN_ONLY = cfg[name]['BMN_ONLY']\n self.LSTM_ONLY = cfg[name]['LSTM_ONLY']\n self.PCM_ONLY = cfg[name]['PCM_ONLY']\n if self.LSTM_ONLY:\n self.prop_dict = {}\n for dataset in ['EuroCup2016']:\n prop_json = '/home/work/datasets/{}/feature_bmn/prop.json'.format(dataset)\n json_data = json.load(open(prop_json, 'r'))\n for item in json_data:\n basename = prop_json.replace('feature_bmn/prop.json', 'mp4')\n basename = basename + '/' + item['video_name'] + '.mp4'\n self.prop_dict[basename] = item['bmn_results']\n @record_time_info\n def load_model(self):\n \"\"\"\n load_model\n \"\"\""
+ },
+ {
+ "comment": "This code initializes models for image, audio, property prediction, and action classification. If DEBUG is not set, it creates InferModel instances for each model type. It then extracts features from input video, gets proposals using BMN (Bidirectional Mixture of Experts Network), and classifies the actions based on these features and proposals.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/action.py\":71-102",
+ "content": " if not self.DEBUG:\n self.image_model = image_model.InferModel(self.configs)\n if not self.PCM_ONLY:\n self.audio_model = audio_model.InferModel(self.configs)\n if not self.LSTM_ONLY:\n self.prop_model = prop_model.InferModel(self.configs)\n if not self.BMN_ONLY:\n self.classify_model = classify_model.InferModel(self.configs)\n logger.info(\"==> Action Detection prepared.\")\n @record_time_info\n def infer(self, imgs_path, pcm_path, fps=5):\n \"\"\"\n extract_feature\n \"\"\"\n self.imgs_path = imgs_path\n self.pcm_path = pcm_path\n self.configs['COMMON']['fps'] = fps\n logger.info(\"==> input video {}\".format(os.path.basename(self.imgs_path)))\n # step 1: extract feature\n video_features = self.extract_feature()\n # step2: get proposal\n bmn_results = self.extract_proposal(video_features)\n # step3: classify \n material = {'feature': video_features, 'proposal': bmn_results}"
+ },
+ {
+ "comment": "The code contains multiple methods: `video_classify`, `extract_proposal`, and `extract_feature`. The `video_classify` method predicts actions using a classification model, while the `extract_proposal` method extracts proposals (BMN results) for an input video. Both methods are decorated with the `@record_time_info` decorator to track execution time. The `extract_feature` method extracts features from images in the given path and is only executed if `DEBUG` flag is not set.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/action.py\":103-131",
+ "content": " action_results = self.video_classify(material)\n return bmn_results, action_results\n @record_time_info\n def video_classify(self, material):\n \"\"\"video classify\"\"\"\n if self.BMN_ONLY:\n return []\n action_results = self.classify_model.predict(self.configs, material=material) \n logger.info('action shape {}'.format(np.array(action_results).shape))\n return action_results\n @record_time_info\n def extract_proposal(self, video_features):\n \"\"\"extract proposal\"\"\"\n if self.LSTM_ONLY:\n basename = self.imgs_path.replace('frames', 'mp4') + '.mp4'\n bmn_results = self.prop_dict[basename]\n return bmn_results\n bmn_results = self.prop_model.predict(self.configs, material=video_features)\n logger.info('proposal shape {}'.format(np.array(bmn_results).shape))\n return bmn_results\n @record_time_info\n def extract_feature(self):\n \"\"\"extract feature\"\"\"\n if not self.DEBUG:\n image_path_list = get_images(self.imgs_path)"
+ },
+ {
+ "comment": "This code configures the PPTSM model with image and audio data, then predicts features for both. If PCM_ONLY is True, it extracts MFCC from pcm file. It creates a video feature dictionary containing the predicted image, audio, and (if applicable) pcm features. If no input images are given, it loads the corresponding features from the specified feature path.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/action.py\":132-150",
+ "content": " self.configs['PPTSM']['frame_list'] = image_path_list\n self.configs['AUDIO']['pcm_file'] = self.pcm_path\n image_features = self.image_model.predict(self.configs)\n if self.PCM_ONLY:\n sample_rate = self.configs['AUDIO']['sample_rate']\n pcm_features = mfcc_extractor.extract_pcm(self.pcm_path, sample_rate)\n audio_features = []\n else:\n audio_features, pcm_features = self.audio_model.predict(self.configs)\n np_image_features = np.array(image_features, dtype=np.float32)\n np_audio_features = np.array(audio_features, dtype=np.float32)\n np_pcm_features = np.array(pcm_features, dtype=np.float32)\n video_features = {'image_feature': np_image_features,\n 'audio_feature': np_audio_features,\n 'pcm_feature': np_pcm_features}\n else:\n feature_path = self.imgs_path.replace(\"frames\", \"features\") + '.pkl'"
+ },
+ {
+ "comment": "Code loads pre-existing video features from a file, checks their shapes and returns them for further processing. It then creates an instance of the ActionDetection model, loads the model, and calls infer function with image and audio paths. Finally, it writes results to a JSON file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/action_detect/action.py\":151-172",
+ "content": " video_features = pickle.load(open(feature_path, 'rb'))\n logger.info(\"feature shape {} {} {}\".format(video_features['image_feature'].shape,\n video_features['audio_feature'].shape,\n video_features['pcm_feature'].shape))\n return video_features\nif __name__ == '__main__':\n model_predict = ActionDetection(cfg_file=\"../configs/configs.yaml\")\n model_predict.load_model()\n imgs_path = \"/home/work/datasets/EuroCup2016/frames/1be705a8f67648da8ec4b4296fa80895\"\n pcm_path = \"/home/work/datasets/EuroCup2016/pcm/1be705a8f67648da8ec4b4296fa80895.pcm\"\n bmn_results, action_results = model_predict.infer(imgs_path, pcm_path)\n results = {'bmn_results': bmn_results, 'action_results': action_results}\n with open('results.json', 'w', encoding='utf-8') as f:\n data = json.dumps(results, indent=4, ensure_ascii=False)\n f.write(data)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/44af1e9d-a252-4491-9e4b-218ec8498ceb.json b/docs/doc/44af1e9d-a252-4491-9e4b-218ec8498ceb.json
new file mode 100644
index 000000000..38517f7b7
--- /dev/null
+++ b/docs/doc/44af1e9d-a252-4491-9e4b-218ec8498ceb.json
@@ -0,0 +1,40 @@
+{
+ "summary": "This code defines a BMN loss function for PaddleVideo, considering time-scale attributes and ratio of positive entries. It also includes a loss function for object detection models with weighted samples, position losses, and ground truth IoU masks. The code further defines a loss function for PEM and TEAM tasks by combining predicted and ground truth values using three loss functions.",
+ "details": [
+ {
+ "comment": "This code defines a BMN loss function for the PaddleVideo library. It is registered in the LOSSES registry and takes two arguments: dscale and tscale, which represent max duration length and sequence length respectively. The class extends BaseWeightedLoss, suggesting it combines multiple weighted losses.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":0-31",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass BMNLoss(BaseWeightedLoss):\n \"\"\"Loss for BMN model\n Args:\n tscale (int): sequence length, default 100.\n dscale (int): max duration length, default 100.\n \"\"\"\n def __init__(self, dscale, tscale):\n super().__init__()\n self.dscale = dscale"
+ },
+ {
+ "comment": "This code defines a class with a time-scale attribute, a method to create binary mask arrays, and a loss function for a specific task. The loss function takes in predicted start and end positions along with ground truth values and calculates a ratio between the number of entries and positive values. This ratio is then used to calculate a coefficient for the loss function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":32-54",
+ "content": " self.tscale = tscale\n def _get_mask(self, dscale, tscale):\n bm_mask = []\n for idx in range(dscale):\n mask_vector = [1 for i in range(tscale - idx)\n ] + [0 for i in range(idx)]\n bm_mask.append(mask_vector)\n bm_mask = np.array(bm_mask, dtype='float32')\n bm_mask = paddle.to_tensor(bm_mask)\n bm_mask.stop_gradient = True\n return bm_mask\n def tem_loss_func(self, pred_start, pred_end, gt_start, gt_end):\n def bi_loss(pred_score, gt_label, datatype):\n pred_score = paddle.reshape(x=pred_score, shape=[-1])\n gt_label = paddle.reshape(x=gt_label, shape=[-1])\n gt_label.stop_gradient = True\n pmask = paddle.cast(x=(gt_label > 0.5), dtype=datatype)\n num_entries = paddle.cast(paddle.shape(pmask), dtype=datatype)\n num_positive = paddle.cast(paddle.sum(pmask), dtype=datatype)\n ratio = num_entries / num_positive\n coef_0 = 0.5 * ratio / (ratio - 1)"
+ },
+ {
+ "comment": "The code defines a loss function for object detection models. It calculates the loss by considering positive and negative samples, applying weights to each sample based on their ratio, and then combines them. The bi_loss function is used to calculate losses for start and end positions. In another function, Pem_reg_loss_func, it separates ground truth IoU map into three masks: high (>0.7), medium (<=0.7 & >0.3), and low (<=0.3 & >=0). It then applies these masks to calculate the loss.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":55-76",
+ "content": " coef_1 = 0.5 * ratio\n epsilon = 0.000001\n loss_pos = paddle.multiply(paddle.log(pred_score + epsilon), pmask)\n loss_pos = coef_1 * paddle.mean(loss_pos)\n loss_neg = paddle.multiply(paddle.log(1.0 - pred_score + epsilon),\n (1.0 - pmask))\n loss_neg = coef_0 * paddle.mean(loss_neg)\n loss = -1 * (loss_pos + loss_neg)\n return loss\n loss_start = bi_loss(pred_start, gt_start, pred_start.dtype)\n loss_end = bi_loss(pred_end, gt_end, pred_start.dtype)\n loss = loss_start + loss_end\n return loss\n def pem_reg_loss_func(self, pred_score, gt_iou_map, mask):\n gt_iou_map = paddle.multiply(gt_iou_map, mask)\n u_hmask = paddle.cast(x=gt_iou_map > 0.7, dtype=pred_score.dtype)\n u_mmask = paddle.logical_and(gt_iou_map <= 0.7, gt_iou_map > 0.3)\n u_mmask = paddle.cast(x=u_mmask, dtype=pred_score.dtype)\n u_lmask = paddle.logical_and(gt_iou_map <= 0.3, gt_iou_map >= 0.)"
+ },
+ {
+ "comment": "Calculating the number of elements in different masks and using them to calculate ratios for later mask operations. Creating uniform masks and multiplying them with corresponding existing masks, then casting the results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":77-100",
+ "content": " u_lmask = paddle.cast(x=u_lmask, dtype=pred_score.dtype)\n u_lmask = paddle.multiply(u_lmask, mask)\n num_h = paddle.cast(paddle.sum(u_hmask), dtype=pred_score.dtype)\n num_m = paddle.cast(paddle.sum(u_mmask), dtype=pred_score.dtype)\n num_l = paddle.cast(paddle.sum(u_lmask), dtype=pred_score.dtype)\n r_m = num_h / num_m\n u_smmask = paddle.uniform(shape=[\n gt_iou_map.shape[1], gt_iou_map.shape[2]\n ],\n min=0.0,\n max=1.0).astype(pred_score.dtype)\n u_smmask = paddle.multiply(u_mmask, u_smmask)\n u_smmask = paddle.cast(x=(u_smmask > (1. - r_m)),\n dtype=pred_score.dtype)\n r_l = num_h / num_l\n u_slmask = paddle.uniform(shape=[\n gt_iou_map.shape[1], gt_iou_map.shape[2]\n ],\n min=0.0,\n max=1.0).astype(pred_score.dtype)\n u_slmask = paddle.multiply(u_lmask, u_slmask)"
+ },
+ {
+ "comment": "In this code, u_slmask is created by comparing r_l with 1 and casting the result to the dtype of pred_score. Then, weights are calculated by adding u_hmask, u_smmask, and u_slmask. The stop_gradient attribute of weights is set to True. Loss is calculated using square error cost between pred_score and gt_iou_map, multiplied by weights, averaged, and returned.\nIn the pem_cls_loss_func, gt_iou_map is multiplied by mask and marked as non-trainable (stop_gradient = True). Pmask and nmask are created based on conditions with gt_iou_map and mask. Num_positive and num_entries are calculated. Ratios are used to determine coef_0 and coef_1. Loss_pos is log(pred_score + epsilon) multiplied by pmask.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":101-125",
+ "content": " u_slmask = paddle.cast(x=(u_slmask > (1. - r_l)),\n dtype=pred_score.dtype)\n weights = u_hmask + u_smmask + u_slmask\n weights.stop_gradient = True\n loss = F.square_error_cost(pred_score, gt_iou_map)\n loss = paddle.multiply(loss, weights)\n loss = 0.5 * paddle.sum(loss) / paddle.sum(weights)\n return loss\n def pem_cls_loss_func(self, pred_score, gt_iou_map, mask):\n gt_iou_map = paddle.multiply(gt_iou_map, mask)\n gt_iou_map.stop_gradient = True\n pmask = paddle.cast(x=(gt_iou_map > 0.9), dtype=pred_score.dtype)\n nmask = paddle.cast(x=(gt_iou_map <= 0.9), dtype=pred_score.dtype)\n nmask = paddle.multiply(nmask, mask)\n num_positive = paddle.sum(pmask)\n num_entries = num_positive + paddle.sum(nmask)\n ratio = num_entries / num_positive\n coef_0 = 0.5 * ratio / (ratio - 1)\n coef_1 = 0.5 * ratio\n epsilon = 0.000001\n loss_pos = paddle.multiply(paddle.log(pred_score + epsilon), pmask)"
+ },
+ {
+ "comment": "Function `forward` takes in `pred_bm`, `pred_start`, `pred_end`, `gt_iou_map`, `gt_start`, and `gt_end`. It first extracts `pred_bm_reg` and `pred_bm_cls` by slicing `pred_bm` along the specified axes. Then, it calculates the `bm_mask` using `_get_mask` with given scales. The function returns the calculated loss from the input parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":126-146",
+ "content": " loss_pos = coef_1 * paddle.sum(loss_pos)\n loss_neg = paddle.multiply(paddle.log(1.0 - pred_score + epsilon),\n nmask)\n loss_neg = coef_0 * paddle.sum(loss_neg)\n loss = -1 * (loss_pos + loss_neg) / num_entries\n return loss\n def forward(self, pred_bm, pred_start, pred_end, gt_iou_map, gt_start,\n gt_end):\n pred_bm_reg = paddle.squeeze(paddle.slice(pred_bm,\n axes=[1],\n starts=[0],\n ends=[1]),\n axis=[1])\n pred_bm_cls = paddle.squeeze(paddle.slice(pred_bm,\n axes=[1],\n starts=[1],\n ends=[2]),\n axis=[1])\n bm_mask = self._get_mask(self.dscale, self.tscale)"
+ },
+ {
+ "comment": "This code calculates the loss for PEM and TEAM detection tasks by combining the predicted and ground truth values. It uses three loss functions: `pem_reg_loss_func`, `pem_cls_loss_func`, and `tem_loss_func`. The final loss is the sum of the temporal (TEM) loss, 10 times the PEM regression loss, and the PEM classification loss.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/bmn_loss.py\":148-154",
+ "content": " pem_reg_loss = self.pem_reg_loss_func(pred_bm_reg, gt_iou_map, bm_mask)\n pem_cls_loss = self.pem_cls_loss_func(pred_bm_cls, gt_iou_map, bm_mask)\n tem_loss = self.tem_loss_func(pred_start, pred_end, gt_start, gt_end)\n loss = tem_loss + 10 * pem_reg_loss + pem_cls_loss\n return loss"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/455ffff8-f746-4f1a-9660-e6765fa42cf5.json b/docs/doc/455ffff8-f746-4f1a-9660-e6765fa42cf5.json
new file mode 100644
index 000000000..bc6fd9e7e
--- /dev/null
+++ b/docs/doc/455ffff8-f746-4f1a-9660-e6765fa42cf5.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The DepthEstimator class inherits from BaseEstimator and contains a forward_net method for feature extraction. It has training, validation, testing, and inference methods with loss metrics calculated using the forward_net and head.loss.",
+ "details": [
+ {
+ "comment": "The code defines a DepthEstimator class that inherits from BaseEstimator. It has a forward_net method that takes inputs and optionally applies a backbone network for feature extraction. The results are stored in outputs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/estimators/depth_estimator.py\":0-30",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nfrom paddlevideo.modeling.framework.estimators.base import BaseEstimator\nfrom paddlevideo.modeling.registry import ESTIMATORS\nfrom paddlevideo.utils import get_logger\nfrom ... import builder\nlogger = get_logger(\"paddlevideo\")\n@ESTIMATORS.register()\nclass DepthEstimator(BaseEstimator):\n \"\"\"DepthEstimator\n \"\"\"\n def forward_net(self, inputs, day_or_night='day_and_night'):\n if self.backbone is not None:\n outputs = self.backbone(inputs, day_or_night)\n else:\n outputs = inputs"
+ },
+ {
+ "comment": "The code defines four methods: train_step, val_step, test_step, and infer_step. The main purpose of each step is to calculate the loss metrics from the input data. The 'forward_net' method is used in all steps to process the inputs and generate outputs, which are then passed to the 'head.loss' method to compute the loss metrics for each step.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/estimators/depth_estimator.py\":31-57",
+ "content": " return outputs\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n inputs, _ = data_batch\n outputs = self.forward_net(inputs, day_or_night='day_and_night')\n loss_metrics = self.head.loss(inputs, outputs)\n return loss_metrics\n def val_step(self, data_batch):\n inputs, day_or_night = data_batch\n outputs = self.forward_net(inputs, day_or_night=day_or_night)\n loss_metrics = self.head.loss(inputs, outputs)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n inputs, day_or_night = data_batch\n outputs = self.forward_net(inputs, day_or_night=day_or_night)\n loss_metrics = self.head.loss(inputs, outputs)\n return loss_metrics\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n inputs = data_batch[0]\n outputs = self.forward_net(inputs, day_or_night='day')"
+ },
+ {
+ "comment": "This code snippet returns the output results from a depth estimator model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/estimators/depth_estimator.py\":58-58",
+ "content": " return outputs"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/45cb0c72-1c7b-45ae-96be-26a6cf633a8a.json b/docs/doc/45cb0c72-1c7b-45ae-96be-26a6cf633a8a.json
new file mode 100644
index 000000000..4b257b5fe
--- /dev/null
+++ b/docs/doc/45cb0c72-1c7b-45ae-96be-26a6cf633a8a.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code trains and validates PoseC3D, a skeleton-based action recognition model, on the UCF101 dataset, using pre-trained weights. It details testing and inference processes without GPU acceleration or TensorRT.",
+ "details": [
+ {
+ "comment": "PoseC3D is a skeleton-based action recognition approach that utilizes 3D head pose features and aims to overcome the limitations of GCN-based methods in terms of robustness, interoperability, and scalability. It involves training on UCF101, testing on UCF101, exporting an inference model, and inferring using the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/posec3d.md\":0-23",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/recognition/posec3d.md) | English\n# PoseC3D\n---\n## Contents\n- [PoseC3D](#PoseC3D)\n - [Contents](#contents)\n - [Introduction](#introduction)\n - [Data](#data)\n - [Train](#train)\n - [Train on UCF101.](#train-on-ucf101)\n - [Test](#test)\n - [Test onf UCF101](#test-onf-ucf101)\n - [Inference](#inference)\n - [export inference model](#export-inference-model)\n - [infer](#infer)\n - [Reference](#reference)\n## Introduction\nHuman skeleton, as a compact representation of hu-man action, has received increasing attention in recentyears. Many skeleton-based action recognition methodsadopt graph convolutional networks (GCN) to extract fea-tures on top of human skeletons. Despite the positive re-sults shown in previous works, GCN-based methods aresubject to limitations in robustness, interoperability, andscalability. In this work, we propose PoseC3D, a new ap-proach to skeleton-based action recognition, which relieson a 3D hea"
+ },
+ {
+ "comment": "This code is for training the PoseC3D model on UCF101 dataset. It requires downloading pre-trained model weights from a specific URL. The command \"python3.7 main.py --validate -c configs/recognition/posec3d/posec3d.yaml --weights res3d_k400.pdparams\" is used to train the PoseC3D model using a provided configuration file and pre-trained weights. The trained model will be validated, likely to assess its performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/posec3d.md\":23-38",
+ "content": "tmap stack instead of a graph sequence asthe base representation of human skeletons. Compared toGCN-based methods, PoseC3D is more effective in learningspatiotemporal features, more robust against pose estima-tion noises, and generalizes better in cross-dataset settings.Also, PoseC3D can handle multiple-person scenarios with-out additional computation cost, and its features can be eas-ily integrated with other modalities at early fusion stages,which provides a great design space to further boost theperformance. On four challenging datasets, PoseC3D con-sistently obtains superior performance, when used alone onskeletons and in combination with the RGB modality.\n## Data\nPlease download UCF101 skeletons datasets and pretraind model weights.\n[https://aistudio.baidu.com/aistudio/datasetdetail/140593](https://aistudio.baidu.com/aistudio/datasetdetail/140593)\n## Train\n### Train on UCF101.\n- Train PoseC3D model:\n```bash\npython3.7 main.py --validate -c configs/recognition/posec3d/posec3d.yaml --weights res3d_k400.pdparams"
+ },
+ {
+ "comment": "This code provides instructions for testing and inference of the PoseC3D model on UCF101 dataset. The test script specifies the config file and weight path, while the inference steps explain how to export the model architecture and parameters for further usage. The link leads to additional information on model inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/posec3d.md\":39-81",
+ "content": "```\n## Test\n### Test onf UCF101\n- Test scripts\uff1a\n```bash\npython3.7 main.py --test -c configs/recognition/posec3d/posec3d.yaml -w output/PoseC3D/PoseC3D_epoch_0012.pdparams\n```\n- Specify the config file with `-c`, specify the weight path with `-w`.\nAccuracy on UCF101 dataset:\n| Test_Data | Top-1 | checkpoints |\n| :----: | :----: | :---- |\n| UCF101 test1 | 87.05 | [PoseC3D_ucf101.pdparams]() |\n## Inference\n### export inference model\n To get model architecture file `PoseC3D.pdmodel` and parameters file `PoseC3D.pdiparams`, use:\n```bash\npython3.7 tools/export_model.py -c configs/recognition/posec3d/posec3d.yaml \\\n -p data/PoseC3D_ucf101.pdparams \\\n -o inference/PoseC3D\n```\n- Args usage please refer to [Model Inference](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0/docs/zh-CN/start.md#2-%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86).\n### infer\n```bash\npython3.7 tools/predict.py --input_file data/example_UCF101_skeleton.pkl\\\n --config configs/recognition/posec3d/posec3d.yaml \\"
+ },
+ {
+ "comment": "Running PoseC3D model for inference with GPU acceleration and without TensorRT.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/posec3d.md\":82-99",
+ "content": " --model_file inference/PoseC3D/PoseC3D.pdmodel \\\n --params_file inference/PoseC3D/PoseC3D.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nexample of logs:\n```\nCurrent video file: data/example_UCF101_skeleton.pkl\n\ttop-1 class: 0\n\ttop-1 score: 0.6731489896774292\n```\n## Reference\n- [Revisiting Skeleton-based Action Recognition](https://arxiv.org/pdf/2104.13586v1.pdf), Haodong Duan, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, Bo Dai"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/45ce3ebf-27d8-4371-a3cd-e3850e588fe8.json b/docs/doc/45ce3ebf-27d8-4371-a3cd-e3850e588fe8.json
new file mode 100644
index 000000000..d39e58f2c
--- /dev/null
+++ b/docs/doc/45ce3ebf-27d8-4371-a3cd-e3850e588fe8.json
@@ -0,0 +1,185 @@
+{
+ "summary": "This function computes L2 distances, applies nearest neighbor attention and feature extraction, considers padding, uses local search windows and average pooling. It introduces a custom layer, calculates nearest neighbor features with embeddings, updates global map dictionaries, and processes inputs to return output dictionaries after calculations on local distance maps for each frame. The code segment updates global and local map dictionaries, calculates frame embeddings and masks, obtains segmentation predictions, and processes data for improved video processing accuracy.",
+ "details": [
+ {
+ "comment": "This code defines a function that calculates pairwise squared L2 distances between two tensors. It takes in two tensors, x and y, and optionally a third tensor ys. The function first computes the sum of squares for each row in tensor x and stores them in xs. If ys is None, it then computes the sum of squares for each row in tensor y and stores them in ys. Otherwise, it uses the provided ys. Finally, the function calculates the pairwise distances using the formula xs + ys - 2 * paddle.matmul(x, paddle.t(y)).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":0-36",
+ "content": "import numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom EIVideo.paddlevideo.utils.manet_utils import int_, float_, long_, load\nfrom EIVideo.paddlevideo.utils.manet_utils import kaiming_normal_\n#############################################################GLOBAL_DIST_MAP\nMODEL_UNFOLD = True\nWRONG_LABEL_PADDING_DISTANCE = 1e20\ndef _pairwise_distances(x, y, ys=None):\n \"\"\"Computes pairwise squared l2 distances between tensors x and y.\n Args:\n x: Tensor of shape [n, feature_dim].\n y: Tensor of shape [m, feature_dim].\n Returns:\n Float32 distances tensor of shape [n, m].\n \"\"\"\n xs = paddle.sum(x * x, 1)\n xs = xs.unsqueeze(1)\n if ys is None:\n ys = paddle.sum(y * y, 1)\n ys = ys.unsqueeze(0)\n else:\n ys = ys\n d = xs + ys - 2. * paddle.matmul(x, paddle.t(y))\n return d, ys\n##################\ndef _flattened_pairwise_distances(reference_embeddings, query_embeddings, ys):\n \"\"\"Calculates flattened tensor of pairwise distances between ref and query."
+ },
+ {
+ "comment": "This function takes reference and query embeddings as input, calculates pairwise distances between them using the _pairwise_distances function, and returns the distances in dists and ys. The _nn_features_per_object_for_chunk function extracts features for each object using nearest neighbor attention, taking reference embeddings, query embeddings, wrong_label_mask, k_nearest_neighbors, and ys as input.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":37-59",
+ "content": " Args:\n reference_embeddings: Tensor of shape [..., embedding_dim],\n the embedding vectors for the reference frame\n query_embeddings: Tensor of shape [n_query_images, height, width,\n embedding_dim], the embedding vectors for the query frames.\n Returns:\n A distance tensor of shape [reference_embeddings.size / embedding_dim,\n query_embeddings.size / embedding_dim]\n \"\"\"\n embedding_dim = query_embeddings.shape[-1]\n reference_embeddings = reference_embeddings.reshape([-1, embedding_dim])\n first_dim = -1\n query_embeddings = query_embeddings.reshape([first_dim, embedding_dim])\n dists, ys = _pairwise_distances(query_embeddings, reference_embeddings, ys)\n return dists, ys\ndef _nn_features_per_object_for_chunk(reference_embeddings, query_embeddings,\n wrong_label_mask, k_nearest_neighbors,\n ys):\n \"\"\"Extracts features for each object using nearest neighbor attention.\n Args:\n reference_embeddings: Tensor of shape [n_chunk, embedding_dim],"
+ },
+ {
+ "comment": "This function calculates the pairwise distances between reference and query embeddings, selects the nearest neighbors based on those distances, and returns the nearest neighbor features. It handles cases with different numbers of reference and query embeddings by padding with a specified distance value for missing embeddings.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":60-82",
+ "content": " the embedding vectors for the reference frame.\n query_embeddings: Tensor of shape [m_chunk, embedding_dim], the embedding\n vectors for the query frames.\n wrong_label_mask:\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m_chunk, n_objects, feature_dim].\n \"\"\"\n # reference_embeddings_key = reference_embeddings\n # query_embeddings_key = query_embeddings\n dists, ys = _flattened_pairwise_distances(reference_embeddings,\n query_embeddings, ys)\n dists = (paddle.unsqueeze(dists, 1) +\n paddle.unsqueeze(float_(wrong_label_mask), 0) *\n WRONG_LABEL_PADDING_DISTANCE)\n if k_nearest_neighbors == 1:\n features = paddle.min(dists, 2, keepdim=True)\n else:\n dists, _ = paddle.topk(-dists, k=k_nearest_neighbors, axis=2)\n dists = -dists\n valid_mask = (dists < WRONG_LABEL_PADDING_DISTANCE)"
+ },
+ {
+ "comment": "The code calculates the mean of distances between valid points and assigns the result to \"features\". The function _selected_pixel() selects pixels from flattened arrays where reference labels are not -1. The function _nearest_neighbor_features_per_object_in_chunks() operates on flattened embeddings, labels, and object ids to compute nearest neighbor features per object in chunks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":83-112",
+ "content": " masked_dists = dists * valid_mask.float()\n pad_dist = paddle.max(masked_dists, axis=2, keepdim=True)[0].tile(\n (1, 1, masked_dists.shape[-1]))\n dists = paddle.where(valid_mask, dists, pad_dist)\n # take mean of distances\n features = paddle.mean(dists, axis=2, keepdim=True)\n return features, ys\n###\ndef _selected_pixel(ref_labels_flat, ref_emb_flat):\n index_list = paddle.arange(len(ref_labels_flat))\n index_list = index_list\n index_ = paddle.masked_select(index_list, ref_labels_flat != -1)\n index_ = long_(index_)\n ref_labels_flat = paddle.index_select(ref_labels_flat, index_, 0)\n ref_emb_flat = paddle.index_select(ref_emb_flat, index_, 0)\n return ref_labels_flat, ref_emb_flat\n###\ndef _nearest_neighbor_features_per_object_in_chunks(reference_embeddings_flat,\n query_embeddings_flat,\n reference_labels_flat,\n ref_obj_ids,"
+ },
+ {
+ "comment": "This function calculates the nearest neighbor features per object in chunks to save memory, using chunking for bounding memory usage. It takes embedding vectors for reference and query frames, their class labels, unique object IDs, number of nearest neighbors, and number of chunks as input. The function returns a tensor of nearest neighbor features for the query frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":113-133",
+ "content": " k_nearest_neighbors,\n n_chunks, **cfg):\n \"\"\"Calculates the nearest neighbor features per object in chunks to save mem.\n Uses chunking to bound the memory use.\n Args:\n reference_embeddings_flat: Tensor of shape [n, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings_flat: Tensor of shape [m, embedding_dim], the embedding\n vectors for the query frames.\n reference_labels_flat: Tensor of shape [n], the class labels of the\n reference frame.\n ref_obj_ids: int tensor of unique object ids in the reference labels.\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [m, n_objects, feature_dim].\n \"\"\"\n # reference_embeddings_flat = reference_embeddings_flat.cpu()"
+ },
+ {
+ "comment": "This code splits the query embeddings into multiple chunks, depending on the number of chunks specified. It then applies a function to each chunk and appends the results to the all_features list. If in test mode, it selects pixels from the reference and query embeddings. It also creates a wrong label mask for the reference labels and query embeddings.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":134-157",
+ "content": " # query_embeddings_flat = query_embeddings_flat.cpu()\n # reference_labels_flat = reference_labels_flat.cpu()\n # ref_obj_ids = ref_obj_ids.cpu()\n chunk_size = int_(\n np.ceil((float_(query_embeddings_flat.shape[0]) / n_chunks).numpy()))\n if cfg.get('test_mode'):\n reference_labels_flat, reference_embeddings_flat = _selected_pixel(\n reference_labels_flat, reference_embeddings_flat)\n wrong_label_mask = (reference_labels_flat != paddle.unsqueeze(\n ref_obj_ids, 1))\n all_features = []\n for n in range(n_chunks):\n if n == 0:\n ys = None\n if n_chunks == 1:\n query_embeddings_flat_chunk = query_embeddings_flat\n else:\n chunk_start = n * chunk_size\n chunk_end = (n + 1) * chunk_size\n query_embeddings_flat_chunk = query_embeddings_flat[\n chunk_start:chunk_end]\n features, ys = _nn_features_per_object_for_chunk(\n reference_embeddings_flat, query_embeddings_flat_chunk,"
+ },
+ {
+ "comment": "This code calculates the nearest neighbor features per object using reference embeddings, query embeddings, and reference labels. It takes into account k-nearest neighbors and can handle a specified number of chunks for subsampling. The function returns the nearest neighbor features in the form of a tensor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":158-180",
+ "content": " wrong_label_mask, k_nearest_neighbors, ys)\n all_features.append(features)\n if n_chunks == 1:\n nn_features = all_features[0]\n else:\n nn_features = paddle.concat(all_features, axis=0)\n return nn_features\ndef nearest_neighbor_features_per_object(reference_embeddings,\n query_embeddings,\n reference_labels,\n k_nearest_neighbors,\n gt_ids=None,\n n_chunks=100,\n **cfg):\n \"\"\"Calculates the distance to the nearest neighbor per object.\n For every pixel of query_embeddings calculate the distance to the\n nearest neighbor in the (possibly subsampled) reference_embeddings per object.\n Args:\n reference_embeddings: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the reference frame.\n query_embeddings: Tensor of shape [n_query_images, height, width,"
+ },
+ {
+ "comment": "This code calculates nearest neighbors for query frames based on the given embedding vectors. It takes input parameters like reference frame class labels, maximum number of candidates, and number of nearest neighbors to use. The function returns nearest neighbor features, unique sorted object ids present in the reference labels, and potentially gt_ids if provided.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":181-200",
+ "content": " embedding_dim], the embedding vectors for the query frames.\n reference_labels: Tensor of shape [height, width, 1], the class labels of\n the reference frame.\n max_neighbors_per_object: Integer, the maximum number of candidates\n for the nearest neighbor query per object after subsampling,\n or 0 for no subsampling.\n k_nearest_neighbors: Integer, the number of nearest neighbors to use.\n gt_ids: Int tensor of shape [n_objs] of the sorted unique ground truth\n ids in the first frame. If None, it will be derived from\n reference_labels.\n n_chunks: Integer, the number of chunks to use to save memory\n (set to 1 for no chunking).\n Returns:\n nn_features: A float32 tensor of nearest neighbor features of shape\n [n_query_images, height, width, n_objects, feature_dim].\n gt_ids: An int32 tensor of the unique sorted object ids present\n in the reference labels.\n \"\"\"\n # reference_embeddings = reference_embeddings.detach().cpu()\n # query_embeddings = query_embeddings.detach().cpu()"
+ },
+ {
+ "comment": "This code is reshaping tensors and calculating nearest neighbor features for each object in chunks. It first reshapes the embeddings, then applies a function to find the closest neighbors and returns a tensor of these features. This process is done in chunks for efficiency and memory management.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":201-223",
+ "content": " # reference_labels = reference_labels.detach().cpu()\n assert (reference_embeddings.shape[:2] == reference_labels.shape[:2])\n h, w, _ = query_embeddings.shape\n reference_labels_flat = reference_labels.reshape([-1])\n if gt_ids is None:\n ref_obj_ids = paddle.unique(reference_labels_flat)[-1]\n ref_obj_ids = np.arange(0, ref_obj_ids + 1)\n gt_ids = paddle.to_tensor(ref_obj_ids)\n gt_ids = int_(gt_ids)\n else:\n gt_ids = int_(paddle.arange(0, gt_ids + 1))\n embedding_dim = query_embeddings.shape[-1]\n query_embeddings_flat = query_embeddings.reshape([-1, embedding_dim])\n reference_embeddings_flat = reference_embeddings.reshape(\n [-1, embedding_dim])\n nn_features = _nearest_neighbor_features_per_object_in_chunks(\n reference_embeddings_flat, query_embeddings_flat,\n reference_labels_flat, gt_ids, k_nearest_neighbors, n_chunks, **cfg)\n nn_features_dim = nn_features.shape[-1]\n nn_features = nn_features.reshape(\n [1, h, w, gt_ids.shape[0], nn_features_dim])"
+ },
+ {
+ "comment": "This function calculates pairwise squared L2 distances using a local search window, with naive implementation using map_fn. It is used as a fallback when correlation_cost is not available. Inputs are tensors x and y of shape [height, width, feature\\_dim]. It returns a tensor of squared distance values shaped [height, width, (2 * max\\_distance + 1) ** 2], where max\\_distance is an integer representing the maximum distance in pixel coordinates per dimension. The function also applies average pooling with a 2x2 filter and pads the tensors x and y before calculating the distances.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":224-251",
+ "content": " return nn_features.cuda(), gt_ids\n########################################################################LOCAL_DIST_MAP\ndef local_pairwise_distances2(x, y, max_distance=9):\n \"\"\"Computes pairwise squared l2 distances using a local search window.\n Naive implementation using map_fn.\n Used as a slow fallback for when correlation_cost is not available.\n Args:\n x: Float32 tensor of shape [height, width, feature_dim].\n y: Float32 tensor of shape [height, width, feature_dim].\n max_distance: Integer, the maximum distance in pixel coordinates\n per dimension which is considered to be in the search window.\n Returns:\n Float32 distances tensor of shape\n [height, width, (2 * max_distance + 1) ** 2].\n \"\"\"\n ori_h, ori_w, _ = x.shape\n x = paddle.transpose(x, [2, 0, 1]).unsqueeze(0)\n x = F.avg_pool2d(x, (2, 2), (2, 2))\n y = paddle.transpose(y, [2, 0, 1]).unsqueeze(0)\n y = F.avg_pool2d(y, (2, 2), (2, 2))\n _, channels, height, width = x.shape\n padding_val = 1e20\n padded_y = F.pad(y,"
+ },
+ {
+ "comment": "This code calculates the nearest neighbor features for local matches in a video. It takes in parameters like previous frame embedding, query embedding, previous frame labels, and ground truth IDs. The function computes distances between frames using Sigmoid activation and bilinear interpolation. Max distance determines the maximum allowed distance for a match to be considered valid.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":252-277",
+ "content": " (max_distance, max_distance, max_distance, max_distance),\n mode='constant',\n value=padding_val)\n offset_y = F.unfold(padded_y, kernel_sizes=[height, width]).reshape(\n [1, channels, height, width, -1])\n x = x.reshape([1, channels, height, width, 1])\n minus = x - offset_y\n dists = paddle.sum(paddle.multiply(minus, minus),\n axis=1).reshape([1, height, width,\n -1]).transpose([0, 3, 1, 2])\n dists = (paddle.nn.functional.sigmoid(dists) - 0.5) * 2\n dists = F.interpolate(dists,\n size=[ori_h, ori_w],\n mode='bilinear',\n align_corners=True)\n dists = dists.squeeze(0).transpose([1, 2, 0])\n return dists\ndef local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding,\n query_embedding,\n prev_frame_labels,\n gt_ids,\n max_distance=12):\n \"\"\"Computes nearest neighbor features while only allowing local matches."
+ },
+ {
+ "comment": "This code calculates the nearest neighbor features by comparing embedding vectors of query frames with the last frame. It takes input tensors for embedding vectors, previous frame labels, and ground truth IDs along with a maximum distance limit. The function returns the nearest neighbor features in a specific shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":278-297",
+ "content": " Args:\n prev_frame_embedding: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the last frame.\n query_embedding: Tensor of shape [height, width, embedding_dim],\n the embedding vectors for the query frames.\n prev_frame_labels: Tensor of shape [height, width, 1], the class labels of\n the previous frame.\n gt_ids: Int Tensor of shape [n_objs] of the sorted unique ground truth\n ids in the first frame.\n max_distance: Integer, the maximum distance allowed for local matching.\n Returns:\n nn_features: A float32 np.array of nearest neighbor features of shape\n [1, height, width, n_objects, 1].\n \"\"\"\n # print(query_embedding.shape, prev_frame_embedding.shape)\n # print(query_embedding.place, prev_frame_embedding.place)\n # query_embedding = query_embedding.cpu()\n # prev_frame_embedding = prev_frame_embedding.cpu()\n # prev_frame_labels = prev_frame_labels.cpu()\n # print(prev_frame_labels.place, prev_frame_embedding.place, query_embedding.place)"
+ },
+ {
+ "comment": "Code snippet performs local pairwise distance calculation between query and previous frame embeddings. If MODEL_UNFOLD is enabled, it generates offset labels by unfolding padded labels with kernel sizes matching height and width of the previous frame embedding. It then creates offset masks by checking equality between offset labels and gt_ids. If MODEL_UNFOLD is not enabled, it directly creates masks by comparing previous frame labels and gt_ids. Finally, it pads the masks using nn.functional.pad with specified padding values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":299-324",
+ "content": " d = local_pairwise_distances2(query_embedding,\n prev_frame_embedding,\n max_distance=max_distance)\n height, width = prev_frame_embedding.shape[:2]\n if MODEL_UNFOLD:\n labels = float_(prev_frame_labels).transpose([2, 0, 1]).unsqueeze(0)\n padded_labels = F.pad(labels, (\n 2 * max_distance,\n 2 * max_distance,\n 2 * max_distance,\n 2 * max_distance,\n ))\n offset_labels = F.unfold(padded_labels,\n kernel_sizes=[height, width],\n strides=[2,\n 2]).reshape([height, width, -1, 1])\n offset_masks = paddle.equal(\n offset_labels,\n float_(gt_ids).unsqueeze(0).unsqueeze(0).unsqueeze(0))\n else:\n masks = paddle.equal(prev_frame_labels,\n gt_ids.unsqueeze(0).unsqueeze(0))\n padded_masks = nn.functional.pad(masks, ("
+ },
+ {
+ "comment": "The code is performing feature extraction and masking for a specific model. It first tiles input data, then applies offset masks to selected regions, and finally extracts minimum distances using the tiled and masked data. The result is a new set of distances which are then reshaped for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":325-357",
+ "content": " 0,\n 0,\n max_distance,\n max_distance,\n max_distance,\n max_distance,\n ))\n offset_masks = []\n for y_start in range(2 * max_distance + 1):\n y_end = y_start + height\n masks_slice = padded_masks[y_start:y_end]\n for x_start in range(2 * max_distance + 1):\n x_end = x_start + width\n offset_mask = masks_slice[:, x_start:x_end]\n offset_masks.append(offset_mask)\n offset_masks = paddle.stack(offset_masks, axis=2)\n d_tiled = d.unsqueeze(-1).tile((1, 1, 1, gt_ids.shape[0]))\n pad = paddle.ones_like(d_tiled)\n d_masked = paddle.where(offset_masks, d_tiled, pad)\n dists = paddle.min(d_masked, axis=2)\n dists = dists.reshape([1, height, width, gt_ids.shape[0], 1])\n return dists\n##############################################################\n#################\nclass _res_block(nn.Layer):\n def __init__(self, in_dim, out_dim, **cfg):\n super(_res_block, self).__init__()"
+ },
+ {
+ "comment": "This code defines a convolutional neural network (CNN) architecture for image processing tasks. The class `IntVOS` contains two 2D convolutions, batch normalization, and ReLU activations in its forward pass. The `IntSegHead` class initializes another CNN with different parameters, which seems to be a part of the overall model. Both classes extend `nn.Layer`, indicating they are PaddlePaddle's version of PyTorch layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":358-389",
+ "content": " self.conv1 = nn.Conv2D(in_dim,\n out_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.relu1 = nn.ReLU()\n self.bn1 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg['train_bn_mom'])\n self.conv2 = nn.Conv2D(out_dim,\n out_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.relu2 = nn.ReLU()\n self.bn2 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg['train_bn_mom'])\n def forward(self, x):\n res = x\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu2(x)\n x += res\n return x\n####################\nclass IntSegHead(nn.Layer):\n def __init__(self, in_dim, emb_dim, **cfg):\n super(IntSegHead, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,"
+ },
+ {
+ "comment": "This code defines a custom Convolutional Neural Network (CNN) layer for extracting features from input images. It consists of multiple convolutions, batch normalizations, and ReLU activations. The input image is first passed through several convolution layers with different configurations, followed by batch normalization and ReLU activation functions to improve model performance. Finally, the output is returned after passing it through a single convolution layer and another batch normalization and ReLU activation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":390-417",
+ "content": " emb_dim,\n kernel_size=7,\n stride=1,\n padding=3)\n self.bn1 = paddle.nn.BatchNorm2D(emb_dim, momentum=cfg['train_bn_mom'])\n self.relu1 = nn.ReLU(True)\n self.res1 = _res_block(emb_dim, emb_dim, **cfg)\n self.res2 = _res_block(emb_dim, emb_dim, **cfg)\n self.conv2 = nn.Conv2D(256,\n emb_dim,\n kernel_size=3,\n stride=1,\n padding=1)\n self.bn2 = paddle.nn.BatchNorm2D(emb_dim, momentum=cfg['train_bn_mom'])\n self.relu2 = nn.ReLU(True)\n self.conv3 = nn.Conv2D(emb_dim, 1, 1, 1)\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.res1(x)\n x = self.res2(x)\n x = self.conv2(x)\n x = self.bn2(x)\n x = self.relu2(x)\n x = self.conv3(x)\n return x"
+ },
+ {
+ "comment": "This code defines a custom layer _split_separable_conv2d, which consists of two convolutional layers followed by ReLU and batch normalization. The first convolution is performed with the same number of input and output channels, while the second has fewer output channels than input dimensions. This architecture helps to reduce parameters and computational cost in a deep learning model for image processing tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":420-441",
+ "content": "class _split_separable_conv2d(nn.Layer):\n def __init__(self, in_dim, out_dim, kernel_size=7, **cfg):\n super(_split_separable_conv2d, self).__init__()\n self.conv1 = nn.Conv2D(in_dim,\n in_dim,\n kernel_size=kernel_size,\n stride=1,\n padding=int((kernel_size - 1) / 2),\n groups=in_dim)\n self.relu1 = nn.ReLU(True)\n self.bn1 = paddle.nn.BatchNorm2D(in_dim, momentum=cfg['train_bn_mom'])\n self.conv2 = nn.Conv2D(in_dim, out_dim, kernel_size=1, stride=1)\n self.relu2 = nn.ReLU(True)\n self.bn2 = paddle.nn.BatchNorm2D(out_dim, momentum=cfg['train_bn_mom'])\n kaiming_normal_(self.conv1.weight, mode='fan_out', nonlinearity='relu')\n kaiming_normal_(self.conv2.weight, mode='fan_out', nonlinearity='relu')\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu1(x)\n x = self.conv2(x)"
+ },
+ {
+ "comment": "The code defines two classes: IntVOS and DynamicSegHead. IntVOS is a subclass of nn.Layer and utilizes the DynamicSegHead class as its segmentation head. DynamicSegHead is also a subclass of nn.Layer and consists of several layers (layer1, layer2, layer3, layer4) that apply separable convolutions to the input. Finally, there's a nn.Conv2D layer with Kaiming initialization for the output. This architecture can be used for segmentation tasks in computer vision applications.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":442-487",
+ "content": " x = self.bn2(x)\n x = self.relu2(x)\n return x\nclass DynamicSegHead(nn.Layer):\n def __init__(self, in_dim, embed_dim, **cfg):\n super(DynamicSegHead, self).__init__()\n self.layer1 = _split_separable_conv2d(in_dim, embed_dim, **cfg)\n self.layer2 = _split_separable_conv2d(embed_dim, embed_dim, **cfg)\n self.layer3 = _split_separable_conv2d(embed_dim, embed_dim, **cfg)\n self.layer4 = _split_separable_conv2d(embed_dim, embed_dim, **cfg)\n self.conv = nn.Conv2D(embed_dim, 1, 1, 1)\n kaiming_normal_(self.conv.weight, mode='fan_out', nonlinearity='relu')\n def forward(self, x):\n x = self.layer1(x)\n x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n x = self.conv(x)\n return x\nfrom ..registry import HEADS\n\"\"\"\n\u8986\u76d6\u539f\u7406\nclass c1:\n def __init__(self):\n self.a = 1\nclass c2(c1):\n def __init__(self):\n super(c2, self).__init__()\n self.a = 2\nc = c2()\nprint(c.a)\n\"\"\"\n@HEADS.register()\nclass IntVOS(nn.Layer):"
+ },
+ {
+ "comment": "This code defines a class called IntVOS. The constructor takes in a feature_extracter and **cfg parameters, initializes the instance variables, and adds layers to the feature_extracter if required. It also initializes the embedding convolution layer for semantic embedding extraction.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":488-505",
+ "content": " def __init__(self, feature_extracter, **cfg):\n super(IntVOS, self).__init__()\n self.feature_extracter = feature_extracter ##embedding extractor\n self.feature_extracter.cls_conv = nn.Sequential()\n self.feature_extracter.upsample4 = nn.Sequential()\n self.semantic_embedding = None\n self.seperate_conv = nn.Conv2D(cfg['model_aspp_outdim'],\n cfg['model_aspp_outdim'],\n kernel_size=3,\n stride=1,\n padding=1,\n groups=cfg['model_aspp_outdim'])\n self.bn1 = paddle.nn.BatchNorm2D(cfg['model_aspp_outdim'],\n momentum=cfg['train_bn_mom'])\n self.relu1 = nn.ReLU(True)\n self.embedding_conv = nn.Conv2D(cfg['model_aspp_outdim'],\n cfg['model_semantic_embedding_dim'], 1,\n 1)"
+ },
+ {
+ "comment": "The code initializes and configures the layers for semantic segmentation. It creates a ReLU activation function, a batch normalization layer with specified parameters, and a sequential neural network containing the separate convolution, first batch norm, first ReLU, embedding convolution, second batch norm, and second ReLU. The code also initializes the dynamic segmentation head and (optionally) an inter-segmentation head depending on the config's 'model_useintseg' flag.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":506-529",
+ "content": " self.relu2 = nn.ReLU(True)\n self.bn2 = paddle.nn.BatchNorm2D(cfg['model_semantic_embedding_dim'],\n momentum=cfg['train_bn_mom'])\n self.semantic_embedding = nn.Sequential(*[\n self.seperate_conv, self.bn1, self.relu1, self.embedding_conv,\n self.bn2, self.relu2\n ])\n for m in self.semantic_embedding:\n if isinstance(m, nn.Conv2D):\n kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')\n self.dynamic_seghead = DynamicSegHead(\n in_dim=cfg['model_semantic_embedding_dim'] + 3,\n embed_dim=cfg['model_head_embedding_dim'],\n **cfg) # propagation segm head\n if cfg['model_useintseg']:\n self.inter_seghead = IntSegHead(\n in_dim=cfg['model_semantic_embedding_dim'] + 3,\n emb_dim=cfg['model_head_embedding_dim'],\n **cfg)\n else:\n self.inter_seghead = DynamicSegHead(\n in_dim=cfg['model_semantic_embedding_dim'] + 2,"
+ },
+ {
+ "comment": "This code defines a class for a model head that takes input, initializes weights (loading pretrained if available), and calculates the loss during forward pass. It uses various input parameters such as x, ref_scribble_label, previous_frame_mask, etc. The forward function extracts features from the input, and is responsible for losses related to the model head.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":530-558",
+ "content": " embed_dim=cfg['model_head_embedding_dim'],\n **cfg) # interaction segm head\n self.pretrained = cfg.get('pretrained', None)\n self.cfg = cfg\n def init_weights(self):\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n self.set_state_dict(load(self.pretrained, self.state_dict()))\n print('loaded pretrained model')\n def loss(self, **kwargs):\n return self.loss_func(**kwargs)\n def forward(self,\n x=None,\n ref_scribble_label=None,\n previous_frame_mask=None,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,\n global_map_tmp_dic=None,\n local_map_dics=None,\n interaction_num=None,\n start_annotated_frame=None,\n frame_num=None):\n x = self.extract_feature(x)"
+ },
+ {
+ "comment": "This code is splitting input feature x into three parts (ref, previous, current frame embeddings), then calling the prop_seghead function to compute a dictionary of results. If global_map_tmp_dic is None, it returns only the dictionary; otherwise, it also updates global_map_tmp_dic and returns both.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":559-587",
+ "content": " # print('extract_feature:', x.mean().item())\n ref_frame_embedding, previous_frame_embedding, current_frame_embedding = paddle.split(\n x, num_or_sections=3, axis=0)\n if global_map_tmp_dic is None:\n dic = self.prop_seghead(\n ref_frame_embedding,\n previous_frame_embedding,\n current_frame_embedding,\n ref_scribble_label,\n previous_frame_mask,\n normalize_nearest_neighbor_distances,\n use_local_map,\n seq_names,\n gt_ids,\n k_nearest_neighbors,\n global_map_tmp_dic,\n local_map_dics,\n interaction_num,\n start_annotated_frame,\n frame_num,\n self.dynamic_seghead,\n )\n return dic\n else:\n dic, global_map_tmp_dic = self.prop_seghead(\n ref_frame_embedding,\n previous_frame_embedding,"
+ },
+ {
+ "comment": "This code defines a class with three methods: \"IntVOS\", \"extract_feature\", and \"prop_seghead\". The \"IntVOS\" function returns two dictionaries after performing some operations. The \"extract_feature\" method extracts features from input image using feature extracter and semantic embedding. The \"prop_seghead\" method takes various inputs, including frame embeddings, scribble label, and mask, and performs propagation segmentation head task with optional normalization and local map usage.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":588-621",
+ "content": " current_frame_embedding,\n ref_scribble_label,\n previous_frame_mask,\n normalize_nearest_neighbor_distances,\n use_local_map,\n seq_names,\n gt_ids,\n k_nearest_neighbors,\n global_map_tmp_dic,\n local_map_dics,\n interaction_num,\n start_annotated_frame,\n frame_num,\n self.dynamic_seghead,\n )\n return dic, global_map_tmp_dic\n def extract_feature(self, x):\n x = self.feature_extracter(x)\n x = self.semantic_embedding(x)\n return x\n def prop_seghead(\n self,\n ref_frame_embedding=None,\n previous_frame_embedding=None,\n current_frame_embedding=None,\n ref_scribble_label=None,\n previous_frame_mask=None,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,"
+ },
+ {
+ "comment": "This function takes in various parameters and returns feature_embedding, global_match_map, local_match_map, and previous_frame_mask. It initializes global_map_tmp_dic, dic_tmp, bs, c, h, w from current_frame_embedding, checks if it is in test mode, scales ref_scribble_label and previous_frame_mask using interpolation for matching dimensions, and then iterates through a range of bs, performing operations on seq_current_frame_embedding.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":622-645",
+ "content": " global_map_tmp_dic=None,\n local_map_dics=None,\n interaction_num=None,\n start_annotated_frame=None,\n frame_num=None,\n dynamic_seghead=None,\n ):\n \"\"\"return: feature_embedding,global_match_map,local_match_map,previous_frame_mask\"\"\"\n ###############\n cfg = self.cfg\n global_map_tmp_dic = global_map_tmp_dic\n dic_tmp = {}\n bs, c, h, w = current_frame_embedding.shape\n if cfg.get('test_mode'):\n scale_ref_scribble_label = float_(ref_scribble_label)\n else:\n scale_ref_scribble_label = paddle.nn.functional.interpolate(\n float_(ref_scribble_label), size=(h, w), mode='nearest')\n scale_ref_scribble_label = int_(scale_ref_scribble_label)\n scale_previous_frame_label = paddle.nn.functional.interpolate(\n float_(previous_frame_mask), size=(h, w), mode='nearest')\n scale_previous_frame_label = int_(scale_previous_frame_label)\n for n in range(bs):\n seq_current_frame_embedding = current_frame_embedding[n]"
+ },
+ {
+ "comment": "This code calculates nearest neighbor features for each object using reference and current frame embeddings, and scribble labels. It transposes the embeddings and label to match the global map format and uses k-nearest neighbors to find the corresponding features. If normalization is enabled, it applies a sigmoid function to normalize the distances.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":646-664",
+ "content": " seq_ref_frame_embedding = ref_frame_embedding[n]\n seq_prev_frame_embedding = previous_frame_embedding[n]\n seq_ref_frame_embedding = seq_ref_frame_embedding.transpose(\n [1, 2, 0])\n seq_current_frame_embedding = seq_current_frame_embedding.transpose(\n [1, 2, 0])\n seq_ref_scribble_label = scale_ref_scribble_label[n].transpose(\n [1, 2, 0])\n #########Global Map\n nn_features_n, ref_obj_ids = nearest_neighbor_features_per_object(\n reference_embeddings=seq_ref_frame_embedding,\n query_embeddings=seq_current_frame_embedding,\n reference_labels=seq_ref_scribble_label,\n k_nearest_neighbors=k_nearest_neighbors,\n gt_ids=gt_ids[n],\n n_chunks=10)\n if normalize_nearest_neighbor_distances:\n nn_features_n = (paddle.nn.functional.sigmoid(nn_features_n) -\n 0.5) * 2"
+ },
+ {
+ "comment": "This code section checks if a sequence name exists in the global map dictionary, and if not, creates an entry for it. It then compares the current frame's features to the corresponding value in the global map for that sequence. If the current frame's features are less than or equal to the stored value, they remain unchanged; otherwise, they get updated with the stored value. Finally, it updates the global map entry with the new frame's features.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":666-686",
+ "content": " # print(nn_features_n)\n ###\n if global_map_tmp_dic is not None: ###when testing, use global map memory\n if seq_names[n] not in global_map_tmp_dic:\n global_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).tile([1000, 1, 1, 1, 1])\n nn_features_n = paddle.where(\n nn_features_n <= global_map_tmp_dic[seq_names[n]][\n frame_num[n]].unsqueeze(0), nn_features_n,\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(\n 0))\n # print('detach 1')\n # print(nn_features_n.shape)\n # nn_features_n = nn_features_n.detach()\n global_map_tmp_dic[seq_names[n]][\n frame_num[n]] = nn_features_n.detach()[0]\n #########################Local dist map\n seq_prev_frame_embedding = seq_prev_frame_embedding.transpose("
+ },
+ {
+ "comment": "This code is finding the nearest neighbor features for the previous frame's embedding, based on whether local mapping is used or not. If local mapping is used, it calls a separate function `local_previous_frame_nearest_neighbor_features_per_object` to get the features and labels. Otherwise, it uses the `nearest_neighbor_features_per_object` function with specified parameters to find the nearest neighbors. The resulting features are stored in `prev_frame_nn_features_n`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":687-706",
+ "content": " [1, 2, 0])\n seq_previous_frame_label = scale_previous_frame_label[n].transpose(\n [1, 2, 0])\n if use_local_map:\n prev_frame_nn_features_n = local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding=seq_prev_frame_embedding,\n query_embedding=seq_current_frame_embedding,\n prev_frame_labels=seq_previous_frame_label,\n gt_ids=ref_obj_ids,\n max_distance=cfg['model_max_local_distance'])\n else:\n prev_frame_nn_features_n, _ = nearest_neighbor_features_per_object(\n reference_embeddings=seq_prev_frame_embedding,\n query_embeddings=seq_current_frame_embedding,\n reference_labels=seq_previous_frame_label,\n k_nearest_neighbors=k_nearest_neighbors,\n gt_ids=gt_ids[n],\n n_chunks=20)\n prev_frame_nn_features_n = ("
+ },
+ {
+ "comment": "This code is checking if the local map dictionaries are not None, indicating testing with local map memory. If a specific sequence name isn't in the local map distance dictionary or temporary map dictionary, it prints an error message and creates a new zero tensor to store the data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":707-723",
+ "content": " paddle.nn.functional.sigmoid(prev_frame_nn_features_n) -\n 0.5) * 2\n # print(prev_frame_nn_features_n.mean().item(), prev_frame_nn_features_n.shape, interaction_num) # o\n #############\n if local_map_dics is not None: ##When testing, use local map memory\n local_map_tmp_dic, local_map_dist_dic = local_map_dics\n if seq_names[n] not in local_map_dist_dic:\n print(seq_names[n], 'not in local_map_dist_dic')\n local_map_dist_dic[seq_names[n]] = paddle.zeros(1000, 9)\n if seq_names[n] not in local_map_tmp_dic:\n print(seq_names[n], 'not in local_map_tmp_dic')\n local_map_tmp_dic[seq_names[n]] = paddle.zeros_like(\n prev_frame_nn_features_n).unsqueeze(0).tile(\n [1000, 9, 1, 1, 1, 1])\n # print(local_map_dist_dic[seq_names[n]].shape)\n # print('detach 2')"
+ },
+ {
+ "comment": "This code segment appears to be part of a larger function that processes video frames and interactions. It stores the distance from the current frame to the first annotated frame in the local_map_dist_dic dictionary, as well as the corresponding previous frame features in the local_map_tmp_dic. The code also updates the value of prev_frame_nn_features_n based on certain conditions involving interaction numbers and distances between frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":724-740",
+ "content": " # prev_frame_nn_features_n = prev_frame_nn_features_n.detach()\n local_map_dist_dic[seq_names[n]][\n frame_num[n], interaction_num -\n 1] = 1.0 / (abs(frame_num[n] - start_annotated_frame)\n ) # bugs fixed.\n local_map_tmp_dic[seq_names[n]][\n frame_num[n],\n interaction_num - 1] = prev_frame_nn_features_n.squeeze(\n 0).detach() # bugs fixed.\n if interaction_num == 1:\n prev_frame_nn_features_n = local_map_tmp_dic[seq_names[n]][\n frame_num[n]][interaction_num - 1]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n else:\n if local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num - 1] > \\\n local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num - 2]:"
+ },
+ {
+ "comment": "This code appears to be part of a video modeling process. It seems to involve local map dictionaries and interaction numbers, comparing previous frames with current ones for float comparisons, unsqueezing and reshaping features and labels, and potentially using these operations in some video modeling or analysis task.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":741-762",
+ "content": " prev_frame_nn_features_n = local_map_tmp_dic[\n seq_names[n]][frame_num[n]][interaction_num - 1]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n else:\n prev_frame_nn_features_n = local_map_tmp_dic[\n seq_names[n]][frame_num[n]][interaction_num - 2]\n prev_frame_nn_features_n = prev_frame_nn_features_n.unsqueeze(\n 0)\n local_map_dics = (local_map_tmp_dic, local_map_dist_dic)\n to_cat_previous_frame = (\n float_(seq_previous_frame_label) == float_(ref_obj_ids)\n ) # float comparision?\n to_cat_current_frame_embedding = current_frame_embedding[\n n].unsqueeze(0).tile((ref_obj_ids.shape[0], 1, 1, 1))\n to_cat_nn_feature_n = nn_features_n.squeeze(0).transpose(\n [2, 3, 0, 1])\n to_cat_previous_frame = float_("
+ },
+ {
+ "comment": "This code is defining a function \"int_seghead\" that takes in various inputs and returns output dictionaries. It concatenates embeddings and features, passes them to the dynamic_seghead function, transposes the result, and stores it in a dictionary. If global_map_tmp_dic is not None, the function also returns other dictionaries.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":763-786",
+ "content": " to_cat_previous_frame.unsqueeze(-1).transpose([2, 3, 0, 1]))\n to_cat_prev_frame_nn_feature_n = prev_frame_nn_features_n.squeeze(\n 0).transpose([2, 3, 0, 1])\n to_cat = paddle.concat(\n (to_cat_current_frame_embedding, to_cat_nn_feature_n,\n to_cat_prev_frame_nn_feature_n, to_cat_previous_frame), 1)\n pred_ = dynamic_seghead(to_cat)\n pred_ = pred_.transpose([1, 0, 2, 3])\n dic_tmp[seq_names[n]] = pred_\n if global_map_tmp_dic is None:\n return dic_tmp\n else:\n if local_map_dics is None:\n return dic_tmp, global_map_tmp_dic\n else:\n return dic_tmp, global_map_tmp_dic, local_map_dics\n def int_seghead(self,\n ref_frame_embedding=None,\n ref_scribble_label=None,\n prev_round_label=None,\n normalize_nearest_neighbor_distances=True,\n global_map_tmp_dic=None,"
+ },
+ {
+ "comment": "This code snippet calculates the local distance map for each frame in the batch and possibly a previous round if it's not the first interaction. The function takes in various parameters such as ref_frame_embedding, prev_round_label, gt_ids, etc., and performs interpolation to resize the reference scribble label and previous round label. It then iterates over each frame in the batch, creating a gt_id array, and calculating the local distance map for the current frame's embedding. This process may involve interpolation and integer conversion of the resized labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":787-812",
+ "content": " local_map_dics=None,\n interaction_num=None,\n seq_names=None,\n gt_ids=None,\n k_nearest_neighbors=1,\n frame_num=None,\n first_inter=True):\n dic_tmp = {}\n bs, c, h, w = ref_frame_embedding.shape\n scale_ref_scribble_label = paddle.nn.functional.interpolate(\n float_(ref_scribble_label), size=(h, w), mode='nearest')\n scale_ref_scribble_label = int_(scale_ref_scribble_label)\n if not first_inter:\n scale_prev_round_label = paddle.nn.functional.interpolate(\n float_(prev_round_label), size=(h, w), mode='nearest')\n scale_prev_round_label = int_(scale_prev_round_label)\n n_chunks = 500\n for n in range(bs):\n gt_id = paddle.arange(0, gt_ids[n] + 1)\n gt_id = int_(gt_id)\n seq_ref_frame_embedding = ref_frame_embedding[n]\n ########################Local dist map"
+ },
+ {
+ "comment": "Updating the global map with the nearest neighbor features for each sequence, only if it's not already in the global_map_tmp_dic.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":813-831",
+ "content": " seq_ref_frame_embedding = paddle.transpose(seq_ref_frame_embedding,\n [1, 2, 0])\n seq_ref_scribble_label = paddle.transpose(\n scale_ref_scribble_label[n], [1, 2, 0])\n nn_features_n = local_previous_frame_nearest_neighbor_features_per_object(\n prev_frame_embedding=seq_ref_frame_embedding,\n query_embedding=seq_ref_frame_embedding,\n prev_frame_labels=seq_ref_scribble_label,\n gt_ids=gt_id,\n max_distance=self.cfg['model_max_local_distance'])\n #######\n ######################Global map update\n if seq_names[n] not in global_map_tmp_dic:\n global_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).tile([1000, 1, 1, 1, 1])\n nn_features_n_ = paddle.where(\n nn_features_n <=\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0),"
+ },
+ {
+ "comment": "This code segment appears to be updating the global and local map dictionaries in a video processing model. The global_map_tmp_dic is being updated with nn_features_n_.detach()[0] at the current frame. Additionally, if the sequence name exists in the local_map_dist_dic or local_map_tmp_dic it is being modified accordingly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":832-853",
+ "content": " nn_features_n,\n global_map_tmp_dic[seq_names[n]][frame_num[n]].unsqueeze(0))\n ###\n ###\n # print('detach 3')\n # nn_features_n_ = nn_features_n_.detach()\n global_map_tmp_dic[seq_names[n]][\n frame_num[n]] = nn_features_n_.detach()[0]\n ##################Local map update\n if local_map_dics is not None:\n local_map_tmp_dic, local_map_dist_dic = local_map_dics\n if seq_names[n] not in local_map_dist_dic:\n local_map_dist_dic[seq_names[n]] = paddle.zeros([1000, 9])\n if seq_names[n] not in local_map_tmp_dic:\n local_map_tmp_dic[seq_names[n]] = paddle.ones_like(\n nn_features_n).unsqueeze(0).tile([1000, 9, 1, 1, 1, 1])\n local_map_dist_dic[seq_names[n]][frame_num[n]][interaction_num\n - 1] = 0\n local_map_dics = (local_map_tmp_dic, local_map_dist_dic)"
+ },
+ {
+ "comment": "This code calculates the current frame embedding and nn_feature_n for each object instance in the scene. It then creates a scribble mask for each object and, if not the first iteration, also creates a previous round mask. The code uses transpose and unsqueeze functions for tensor manipulation and float comparisons to create binary masks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":855-877",
+ "content": " ##################\n to_cat_current_frame_embedding = ref_frame_embedding[n].unsqueeze(\n 0).tile((gt_id.shape[0], 1, 1, 1))\n to_cat_nn_feature_n = nn_features_n.squeeze(0).transpose(\n [2, 3, 0, 1])\n to_cat_scribble_mask_to_cat = (\n float_(seq_ref_scribble_label) == float_(gt_id)\n ) # float comparision?\n to_cat_scribble_mask_to_cat = float_(\n to_cat_scribble_mask_to_cat.unsqueeze(-1).transpose(\n [2, 3, 0, 1]))\n if not first_inter:\n seq_prev_round_label = scale_prev_round_label[n].transpose(\n [1, 2, 0])\n to_cat_prev_round_to_cat = (\n float_(seq_prev_round_label) == float_(gt_id)\n ) # float comparision?\n to_cat_prev_round_to_cat = float_(\n to_cat_prev_round_to_cat.unsqueeze(-1).transpose(\n [2, 3, 0, 1]))\n else:"
+ },
+ {
+ "comment": "In this code, a concatenation of current frame embedding, scribble mask, and previous round information is passed to inter_seghead for segmentation prediction. The predictions are then transposed before being added to dic_tmp for further processing. If local_map_dics is None, the function returns dic_tmp; otherwise, it returns both dic_tmp and local_map_dics.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/heads/IntVOS.py\":878-892",
+ "content": " to_cat_prev_round_to_cat = paddle.zeros_like(\n to_cat_scribble_mask_to_cat)\n to_cat_prev_round_to_cat[0] = 1.\n to_cat = paddle.concat(\n (to_cat_current_frame_embedding, to_cat_scribble_mask_to_cat,\n to_cat_prev_round_to_cat), 1)\n pred_ = self.inter_seghead(to_cat)\n pred_ = pred_.transpose([1, 0, 2, 3])\n dic_tmp[seq_names[n]] = pred_\n if local_map_dics is None:\n return dic_tmp\n else:\n return dic_tmp, local_map_dics"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4613d6ed-c878-4172-a444-515ae063a628.json b/docs/doc/4613d6ed-c878-4172-a444-515ae063a628.json
new file mode 100644
index 000000000..4b7f2452a
--- /dev/null
+++ b/docs/doc/4613d6ed-c878-4172-a444-515ae063a628.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code provides a list of useful action recognition datasets along with their respective links for further reference. These datasets are essential for training and evaluating action recognition models, each serving its specific purpose in the field of computer vision.",
+ "details": [
+ {
+ "comment": "This code provides a list of useful action recognition datasets along with their respective links for further reference. These datasets are essential for training and evaluating action recognition models, each serving its specific purpose in the field of computer vision.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/Action Recognition Datasets\":0-11",
+ "content": "Usefull Action Recognition Datasets.\n AVA, https://arxiv.org/abs/1705.08421\n Kinetics, https://arxiv.org/abs/1705.06950\n YouTube-8M, https://arxiv.org/abs/1609.08675\n ActivityNet, http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.html\n Moments in Time, https://arxiv.org/pdf/1801.03150.pdf\n Charades, https://arxiv.org/abs/1604.01753\n EPIC-Kitchens, https://arxiv.org/abs/1804.02748\n THUMOS, https://arxiv.org/abs/1604.06182\n UCF-101, http://crcv.ucf.edu/papers/UCF101_CRCV-TR-12-01.pdf\n HMDB51, http://serre-lab.clps.brown.edu/wp-content/uploads/2012/08/Kuehne_etal_iccv11.pdf"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/462dc8cb-b385-4035-b04c-9d58c98e7e51.json b/docs/doc/462dc8cb-b385-4035-b04c-9d58c98e7e51.json
new file mode 100644
index 000000000..63750305a
--- /dev/null
+++ b/docs/doc/462dc8cb-b385-4035-b04c-9d58c98e7e51.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The \"BoxList\" class manages bounding boxes, and the _is_valid_boxes function checks if data array of shape [N, 4] representing box coordinates adheres to the correct format. The function returns a boolean indicating whether all ymax are greater than or equal to ymin, all xmax are greater than or equal to xmin, and the data is not empty.",
+ "details": [
+ {
+ "comment": "The code defines a class called \"BoxList\" that represents a collection of bounding boxes as a numpy array. Each box is represented by 4 numbers: y_min, x_min, y_max, and x_max. It assumes all boxes in the list correspond to a single image.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_evaluation/np_box_list.py\":0-25",
+ "content": "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# =============================================================================\n\"\"\"Numpy BoxList classes and functions.\"\"\"\nimport numpy as np\nclass BoxList:\n \"\"\"Box collection.\n BoxList represents a list of bounding boxes as numpy array, where each\n bounding box is represented as a row of 4 numbers,\n [y_min, x_min, y_max, x_max]. It is assumed that all bounding boxes within\n a given list correspond to a single image."
+ },
+ {
+ "comment": "This code defines a class for box collections, where users can optionally add additional related fields such as objectness or classification scores. The `__init__` method checks if the input data is a numpy array, has valid dimensions and data type (float), and raises a ValueError if any of these conditions are not met. It then stores the data in a dictionary with key \"boxes\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_evaluation/np_box_list.py\":27-51",
+ "content": " Optionally, users can add additional related fields (such as\n objectness/classification scores).\n \"\"\"\n def __init__(self, data):\n \"\"\"Constructs box collection.\n Args:\n data: a numpy array of shape [N, 4] representing box coordinates\n Raises:\n ValueError: if bbox data is not a numpy array\n ValueError: if invalid dimensions for bbox data\n \"\"\"\n if not isinstance(data, np.ndarray):\n raise ValueError('data must be a numpy array.')\n if len(data.shape) != 2 or data.shape[1] != 4:\n raise ValueError('Invalid dimensions for box data.')\n if data.dtype != np.float32 and data.dtype != np.float64:\n raise ValueError(\n 'Invalid data type for box data: float is required.')\n if not self._is_valid_boxes(data):\n raise ValueError('Invalid box data. data must be a numpy array of '\n 'N*[y_min, x_min, y_max, x_max]')\n self.data = {'boxes': data}"
+ },
+ {
+ "comment": "This code defines a class with methods to handle box collections. It provides functionality for counting the number of boxes, retrieving non-box fields, checking if a specific field exists, and adding data to an existing or new field while handling errors related to field existence and data dimensions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_evaluation/np_box_list.py\":53-80",
+ "content": " def num_boxes(self):\n \"\"\"Return number of boxes held in collections.\"\"\"\n return self.data['boxes'].shape[0]\n def get_extra_fields(self):\n \"\"\"Return all non-box fields.\"\"\"\n return [k for k in self.data if k != 'boxes']\n def has_field(self, field):\n return field in self.data\n def add_field(self, field, field_data):\n \"\"\"Add data to a specified field.\n Args:\n field: a string parameter used to speficy a related field to be\n accessed.\n field_data: a numpy array of [N, ...] representing the data\n associated with the field.\n Raises:\n ValueError: if the field is already exist or the dimension of the\n field data does not matches the number of boxes.\n \"\"\"\n if self.has_field(field):\n raise ValueError('Field ' + field + 'already exists')\n if len(field_data.shape) < 1 or field_data.shape[0] != self.num_boxes(\n ):\n raise ValueError('Invalid dimensions for field data')"
+ },
+ {
+ "comment": "The code defines a class with methods to access box coordinates from a stored dataset. The \"get\" method returns a numpy array of shape [N, 4] representing box corners. The \"get_field\" method is used to access data related to a specific field in the box collection. If an invalid field is provided, it raises a ValueError. The \"get_coordinates\" method returns a list of 4 1-d numpy arrays containing y_min, x_min, y_max, and x_max values for each box.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_evaluation/np_box_list.py\":81-116",
+ "content": " self.data[field] = field_data\n def get(self):\n \"\"\"Convenience function for accesssing box coordinates.\n Returns:\n a numpy array of shape [N, 4] representing box corners\n \"\"\"\n return self.get_field('boxes')\n def get_field(self, field):\n \"\"\"Accesses data associated with the specified field in the box\n collection.\n Args:\n field: a string parameter used to speficy a related field to be\n accessed.\n Returns:\n a numpy 1-d array representing data of an associated field\n Raises:\n ValueError: if invalid field\n \"\"\"\n if not self.has_field(field):\n raise ValueError(f'field {field} does not exist')\n return self.data[field]\n def get_coordinates(self):\n \"\"\"Get corner coordinates of boxes.\n Returns:\n a list of 4 1-d numpy arrays [y_min, x_min, y_max, x_max]\n \"\"\"\n box_coordinates = self.get()\n y_min = box_coordinates[:, 0]"
+ },
+ {
+ "comment": "This code defines a function `_is_valid_boxes` which checks if the data array of shape [N, 4] representing box coordinates fulfills the format N*[ymin, xmin, ymax, xmax]. It returns a boolean indicating whether all ymax of boxes are equal or greater than ymin and all xmax of boxes are equal or greater than xmin. The function also checks if the data is not empty.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/ava_evaluation/np_box_list.py\":117-137",
+ "content": " x_min = box_coordinates[:, 1]\n y_max = box_coordinates[:, 2]\n x_max = box_coordinates[:, 3]\n return [y_min, x_min, y_max, x_max]\n def _is_valid_boxes(self, data):\n \"\"\"Check whether data fullfills the format of N*[ymin, xmin, ymax,\n xmin].\n Args:\n data: a numpy array of shape [N, 4] representing box coordinates\n Returns:\n a boolean indicating whether all ymax of boxes are equal or greater\n than ymin, and all xmax of boxes are equal or greater than xmin.\n \"\"\"\n if len(data):\n for v in data:\n if v[0] > v[2] or v[1] > v[3]:\n return False\n return True"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/46462441-92b5-4cd2-9738-198b110c5ae8.json b/docs/doc/46462441-92b5-4cd2-9738-198b110c5ae8.json
new file mode 100644
index 000000000..57cd25ae5
--- /dev/null
+++ b/docs/doc/46462441-92b5-4cd2-9738-198b110c5ae8.json
@@ -0,0 +1,50 @@
+{
+ "summary": "The document explains the Oxford-RobotCar data preparation process for day-night depth estimation and provides related file download links. It outlines dataset preprocessing steps for ADDS-DepthNet training, including filtering, renaming, and image processing. The code showcases a directory structure with consistent training/verification sequences for day and night images.",
+ "details": [
+ {
+ "comment": "This is a brief introduction to the Oxford-RobotCar-for-ADDS data preparation document. It provides information on downloading and preprocessing the dataset for autonomous driving tasks, specifically day-night depth estimation. The original dataset can be found at the link provided in the text, and any use of this modified version should cite the referenced paper.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":0-23",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../zh-CN/dataset/Oxford_RobotCar.md) | English\n# Oxford-RobotCar-for-ADDS data preparation\n- [Introduction](#Introduction)\n- [Data Set Download](#Download)\n- [Preprocessing](#Preprocessing)\n- [1. Image De-distortion](#1-Image-de-distortion)\n- [2. Dynamic frame filter](#2-Dynamic-frame-filter)\n- [3. Image Rename](#3-Image-Rename)\n- [4. Preparation for Day-Pseudo Night Image Pair](#4-Day-Pseudo-Night-Image-Pair-Preparation)\n## Introduction\n[Oxford RobotCar Dataset](https://robotcar-dataset.robots.ox.ac.uk/) is a large-scale autonomous driving data set that contains a large amount of data in different autonomous driving scenarios.\nWhat is used here is to filter a part of the data used for day-night depth estimation from the original Oxford RobotCar data set, namely Oxford-RobotCar-for-ADDS.\nIf you want to use Oxford-RobotCar-for-ADDS, please cite the following papers:\n```latex\n@article{maddern20171,\n title={1 year, 1000 km: The oxford robotcar dataset},\n author={Maddern, Will and Pascoe, Geoffrey and Linegar, Chris and Newman, Paul},"
+ },
+ {
+ "comment": "This code represents the citation for two research papers in the format of BibTeX. The first paper is titled \"Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation\" and was published in 2021 at the IEEE/CVF International Conference on Computer Vision. The second paper is an Oxford RobotCar dataset study published in The International Journal of Robotics Research in 2017. Both papers are cited within a broader document, likely discussing the use or application of these datasets for computer vision tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":24-45",
+ "content": " journal={The International Journal of Robotics Research},\n volume={36},\n number={1},\n pages={3--15},\n year={2017},\n publisher={SAGE Publications Sage UK: London, England}\n}\n```\n```latex\n@inproceedings{liu2021self,\n title={Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation},\n author={Liu, Lina and Song, Xibin and Wang, Mengmeng and Liu, Yong and Zhang, Liangjun},\n booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\n pages={12737--12746},\n year={2021}\n}\n```\n## Download\n1. Download the left eye image of Bumblebee XB3 in the sequence [2014-12-09](https://robotcar-dataset.robots.ox.ac.uk/datasets/2014-12-09-13-21-02/) as For the training set of the daytime scene, the downloaded images are decompressed in the same folder.\n2. Download the left eye image of Bumblebee XB3 in the sequence [2014-12-16](https://robotcar-dataset.robots.ox.ac.uk/datasets/2014-12-16-18-44-24/) as The training set of the night scene, the downloaded images are unzipped in the same folder."
+ },
+ {
+ "comment": "The code provides download links for validation and original raw data sets in Oxford RobotCar dataset, used for image and depth truth values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":46-63",
+ "content": "3. The images and depth truth values \u200b\u200bof the validation set are filtered from the original data set and downloaded from the link we gave. (The data download links are below)\n ```shell\n https://videotag.bj.bcebos.com/Data/ADDS/1209_all_files.txt\n https://videotag.bj.bcebos.com/Data/ADDS/1216_all_files.txt\n https://videotag.bj.bcebos.com/Data/ADDS/day_train_all.7z.001\n https://videotag.bj.bcebos.com/Data/ADDS/day_train_all.7z.002\n https://videotag.bj.bcebos.com/Data/ADDS/day_train_all_fake_night.7z.001\n https://videotag.bj.bcebos.com/Data/ADDS/day_train_all_fake_night.7z.002\n https://videotag.bj.bcebos.com/Data/ADDS/day_val_451.7z\n https://videotag.bj.bcebos.com/Data/ADDS/day_val_451_gt.7z\n https://videotag.bj.bcebos.com/Data/ADDS/night_val_411.7z\n https://videotag.bj.bcebos.com/Data/ADDS/night_val_411_gt.7z\n ```\n the original raw data download links:\n ```shell\n # data in day\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.001\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.002"
+ },
+ {
+ "comment": "Links to parts of a 7z compressed file containing training data for day and night scenes from the Oxford RobotCar dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":64-79",
+ "content": " https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.003\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.004\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.005\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.006\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.007\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.008\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.009\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.010\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.011\n https://videotag.bj.bcebos.com/Data/original-ADDS/day_train_all.7z.012\n # data in night\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.001\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.002\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.003\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.004"
+ },
+ {
+ "comment": "This code provides a list of URLs for various file segments (005 to 015) related to the \"night_train_all.7z\" file. These files are likely part of the Oxford RobotCar dataset and are used in preprocessing steps, such as image de-distortion, which pairs sequences from specific dates. The official toolbox mentioned is necessary for this process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":80-97",
+ "content": " https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.005\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.006\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.007\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.008\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.009\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.010\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.011\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.012\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.013\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.014\n https://videotag.bj.bcebos.com/Data/original-ADDS/night_train_all.7z.015\n ```\n## Preprocessing\n### 1-Image-de-distortion\nUse the official toolbox [robotcar-dataset-sdk](https://github.com/ori-mrg/robotcar-dataset-sdk/tree/master/python) to pair the sequence 2014-12-09 and 2014-12- The image of 16 is de-distorted."
+ },
+ {
+ "comment": "This code segment discusses two main components: 1) dynamic frame filtering for self-supervised training, and 2) renaming of original image timestamps to create continuous number sequences. The dataset contains daytime and nighttime training images, as well as daytime verification images with corresponding depth truth values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":100-113",
+ "content": "### 2-Dynamic-frame-filter\nSince we use the self-supervised method, we need to filter out dynamic frames for training. The filtering principle is that the inter-frame pose change is greater than 0.1m and it is considered a dynamic frame. After filtering, the sequence of the training set is obtained.\n### 3-Image-Rename\nRename the original image timestamp to a continuous number sequence. For daytime scene correspondence, see [1209_all_files.txt](https://videotag.bj.bcebos.com/Data/ADDS/1209_all_files.txt), for night scene correspondence, see [1216_all_files.txt](https://videotag.bj.bcebos.com/Data/ADDS/1216_all_files.txt). The renamed data format is as follows:\n```\n\u251c\u2500\u2500 oxford_processing\n \u251c\u2500\u2500 day_train_all #Day training image folder (day_train_all.7z.001 ~ day_train_all.7z.012)\n \u251c\u2500\u2500 night_train_all #Night training image folder (night_train_all.7z.001 ~ day_train_all.7z.015)\n \u251c\u2500\u2500 day_val_451 #Daytime verification image folder (day_val_451.7z)\n \u251c\u2500\u2500 day_val_451_gt #Daytime verification depth truth value folder (day_val_451_gt.7z)"
+ },
+ {
+ "comment": "This code provides the location of image folders and annotation files for a robot car dataset, as well as the sequence used for training and verification. It also mentions the usage of CycleGAN to generate day-pseudo-night image pairs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":114-136",
+ "content": " \u251c\u2500\u2500 night_val_411 #night verification image folder (night_val_411.7z)\n \u2514\u2500\u2500 night_val_411_gt #Night verification depth truth value folder (night_val_411_gt.7z)\n```\nannotation files download links are below:\n```shell\nhttps://videotag.bj.bcebos.com/Data/ADDS/train_files.txt\nhttps://videotag.bj.bcebos.com/Data/ADDS/val_day_files.txt\nhttps://videotag.bj.bcebos.com/Data/ADDS/val_night_files.txt\n```\nThe sequence used for training and verification is as follows:\n```\nsplits/oxford_day/train_files.txt # training sequence during the day\nsplits/oxford_night/train_files.txt # training sequence at night\nsplits/oxford_day_451/val_files.txt # verification sequence during the day\nsplits/oxford_night_411/val_files.txt # night verification sequence\n```\n### 4-Day-Pseudo-Night-Image-Pair-Preparation\nIn order to use our framework to extract the common information of day and night images, we use [CycleGAN](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) to generate day-pseudo-night image pairs, where pseudo-n"
+ },
+ {
+ "comment": "This code describes the file structure and data format of the Oxford-RobotCar dataset for ADDS-DepthNet training and verification. It includes daytime and nighttime images, as well as their ground truth depth values, organized into separate folders for training and validation purposes. The data has been preprocessed and scaled, with corresponding pseudo-night images generated using CycleGAN and histogram equalization applied to night images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":136-149",
+ "content": "ight The night images corresponding to the daytime generated for CycleGAN, all images are scaled to 192x640, the night images are enhanced with histogram equalization, 75 epochs are trained, and the Oxford-RobotCar-for-ADDS is finally obtained. The generated day-pseudo-night image pair The data format is as follows, which can be directly used for training and verification of ADDS-DepthNet:\n```\n\u251c\u2500\u2500 oxford_processing_forADDS\n \u251c\u2500\u2500 day_train_all #Day training image folder (day_train_all.7z.001 ~ day_train_all.7z.002)\n \u251c\u2500\u2500 night_train_all #Night training image folder (night_train_all.7z.001 ~ day_train_all.7z.002)\n \u251c\u2500\u2500 day_val_451 #Daytime verification image folder (day_val_451.7z)\n \u251c\u2500\u2500 day_val_451_gt #Daytime verification depth truth value folder (day_val_451_gt.7z)\n \u251c\u2500\u2500 night_val_411 #night verification image folder (night_val_411.7z)\n \u2514\u2500\u2500 night_val_411_gt #Night verification depth truth value folder (night_val_411_gt.7z)\ndata\n\u2514\u2500\u2500 oxford\n \u251c\u2500\u2500 splits\n \u251c\u2500\u2500 train_files.txt\n \u251c\u2500\u2500 val_day_files.txt"
+ },
+ {
+ "comment": "The code represents a directory structure containing day and night training and verification image folders, along with their respective depth truth value folders. The sequences used for both training and verification are consistent.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/Oxford_RobotCar.md\":150-161",
+ "content": " \u2514\u2500\u2500 val_night_files.txt\n \u2514\u2500\u2500 oxford_processing_forADDS\n \u251c\u2500\u2500 day_train_all/ #Day training image folder (from day_train_all.7z.001 ~ day_train_all.7z.002)\n \u251c\u2500\u2500 night_train_all/ #Night training image folder (from night_train_all.7z.001 ~ day_train_all.7z.002)\n \u251c\u2500\u2500 day_val_451/ #Daytime verification image folder (from day_val_451.7z)\n \u251c\u2500\u2500 day_val_451_gt/ #Daytime verification depth truth value folder (from day_val_451_gt.7z)\n \u251c\u2500\u2500 night_val_411/ #night verification image folder (from night_val_411.7z)\n \u2514\u2500\u2500 night_val_411_gt/ #Night verification depth truth value folder (from night_val_411_gt.7z)\n```\nThe sequences used for training and verification are consistent with the foregoing."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/468ec0a1-622d-48fe-993d-08d86f88d526.json b/docs/doc/468ec0a1-622d-48fe-993d-08d86f88d526.json
new file mode 100644
index 000000000..c0d9268e6
--- /dev/null
+++ b/docs/doc/468ec0a1-622d-48fe-993d-08d86f88d526.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code defines a config parser and provides two functions for parsing, merging, setting, and printing configuration in different sections. The code handles section validity and updates values when merging.",
+ "details": [
+ {
+ "comment": "This code defines a config parser and provides two functions: `parse_config` and `merge_configs`. The `parse_config` function reads a configuration file and returns an `AttrDict` object containing the parsed configurations for different sections ('train', 'valid', 'test', 'infer'). The `merge_configs` function takes an existing configuration object, a section name, and an optional dictionary of arguments to merge into the configuration. It checks if the section is valid before attempting to merge the new arguments. If a value is None, it is ignored during the merging process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/config.py\":0-51",
+ "content": "\"\"\"\nconfig parser\n\"\"\"\ntry:\n from configparser import ConfigParser\nexcept BaseException:\n from ConfigParser import ConfigParser\nfrom utils import AttrDict\nimport logging\nlogger = logging.getLogger(__name__)\nCONFIG_SECS = [\n 'train',\n 'valid',\n 'test',\n 'infer',\n]\ndef parse_config(cfg_file):\n \"\"\"parse_config\n \"\"\"\n parser = ConfigParser()\n cfg = AttrDict()\n parser.read(cfg_file)\n for sec in parser.sections():\n sec_dict = AttrDict()\n for k, v in parser.items(sec):\n try:\n v = eval(v)\n except BaseException:\n pass\n setattr(sec_dict, k, v)\n setattr(cfg, sec.upper(), sec_dict)\n return cfg\ndef merge_configs(cfg, sec, args_dict):\n \"\"\"merge_configs\n \"\"\"\n assert sec in CONFIG_SECS, \"invalid config section {}\".format(sec)\n sec_dict = getattr(cfg, sec.upper())\n for k, v in args_dict.items():\n if v is None:\n continue\n # try:\n # if hasattr(sec_dict, k):\n # setattr(sec_dict, k, v)"
+ },
+ {
+ "comment": "This code defines two functions, `set_config` and `print_configs`. The `set_config` function takes a dictionary (cfg) as input, iterates through its keys and values, and sets the value for each key in the configuration dictionary (sec_dict). If the key already exists in sec_dict, it updates its value. Finally, the function returns the updated configuration dictionary. The `print_configs` function prints out the configuration in a formatted way using the logger module. It iterates through the sections and their corresponding values in the configuration dictionary and logs them to the console with proper indentation and section names.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/config.py\":52-70",
+ "content": " # except BaseException:\n # pass\n if k in sec_dict:\n setattr(sec_dict, k, v)\n return cfg\ndef print_configs(cfg, mode):\n \"\"\"print_configs\n \"\"\"\n logger.info(\"---------------- {:>5} Arguments ----------------\".format(mode))\n for sec, sec_items in cfg.items():\n if isinstance(sec_items, dict) is True:\n logger.info(\"{}:\".format(sec))\n for k, v in sec_items.items():\n logger.info(\" {}:{}\".format(k, v))\n else:\n logger.info(\"{}:{}\".format(sec, sec_items))\n logger.info(\"-------------------------------------------------\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/46b06b32-b631-43dd-9c96-079811df2c97.json b/docs/doc/46b06b32-b631-43dd-9c96-079811df2c97.json
new file mode 100644
index 000000000..464b4b050
--- /dev/null
+++ b/docs/doc/46b06b32-b631-43dd-9c96-079811df2c97.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code initializes and defines various backbone models for video analysis tasks in PaddleVideo, including ResNet, Vision Transformer, AGCN, and popular models such as ResNetTSN_MRI, ResNetTSM_MRI, and SwinTransformer3D. These models form the foundation for object detection, segmentation, motion estimation, and various computer vision applications in PaddlePaddle framework.",
+ "details": [
+ {
+ "comment": "This code is an initialization file for backbone models in PaddleVideo. It imports various model classes from submodules, including BertForMultiModalPreTraining, ADDS_DepthNet, AGCN, ASRF, BMN, CFBI, MoViNet, MSTCN, ResNet, ResNetSlowFast, ResNetSlowFast_MRI, and ResNetTSM, ResNetTSM_MRI. These models can be used for video analysis tasks in the PaddlePaddle framework.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/__init__.py\":0-26",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .actbert import BertForMultiModalPreTraining\nfrom .adds import ADDS_DepthNet\nfrom .agcn import AGCN\nfrom .asrf import ASRF\nfrom .bmn import BMN\nfrom .cfbi import CFBI\nfrom .movinet import MoViNet\nfrom .ms_tcn import MSTCN\nfrom .resnet import ResNet\nfrom .resnet_slowfast import ResNetSlowFast\nfrom .resnet_slowfast_MRI import ResNetSlowFast_MRI\nfrom .resnet_tsm import ResNetTSM\nfrom .resnet_tsm_MRI import ResNetTSM_MRI"
+ },
+ {
+ "comment": "The code imports various backbone models for video analysis from different modules within the PaddleVideo library, including ResNet, Vision Transformer, STGCN, AGCN, and more. The models are used for tasks like object detection, segmentation, and motion estimation in video processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/__init__.py\":27-54",
+ "content": "from .resnet_tsn_MRI import ResNetTSN_MRI\nfrom .resnet_tweaks_tsm import ResNetTweaksTSM\nfrom .resnet_tweaks_tsn import ResNetTweaksTSN\nfrom .stgcn import STGCN\nfrom .swin_transformer import SwinTransformer3D\nfrom .transnetv2 import TransNetV2\nfrom .vit import VisionTransformer\nfrom .vit_tweaks import VisionTransformer_tweaks\nfrom .ms_tcn import MSTCN\nfrom .asrf import ASRF\nfrom .resnet_tsn_MRI import ResNetTSN_MRI\nfrom .resnet_tsm_MRI import ResNetTSM_MRI\nfrom .resnet_slowfast_MRI import ResNetSlowFast_MRI\nfrom .cfbi import CFBI\nfrom .ctrgcn import CTRGCN\nfrom .agcn2s import AGCN2s\nfrom .movinet import MoViNet\nfrom .resnet3d_slowonly import ResNet3dSlowOnly\nfrom .toshift_vit import TokenShiftVisionTransformer\nfrom .pptsm_mv2 import PPTSM_MobileNetV2\nfrom .pptsm_mv3 import PPTSM_MobileNetV3\nfrom .pptsm_v2 import PPTSM_v2\nfrom .yowo import YOWO\n__all__ = [\n 'ResNet', 'ResNetTSM', 'ResNetTweaksTSM', 'ResNetSlowFast', 'BMN',\n 'ResNetTweaksTSN', 'VisionTransformer', 'STGCN', 'AGCN', 'TransNetV2',\n 'ADDS_DepthNet', 'VisionTransformer_tweaks', 'BertForMultiModalPreTraining',"
+ },
+ {
+ "comment": "This code defines a list of available backbones for video processing tasks, including popular models such as ResNetTSN_MRI, ResNetTSM_MRI, and SwinTransformer3D. These backbones serve as the foundation for various computer vision applications in PaddleVideo.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/__init__.py\":55-59",
+ "content": " 'ResNetTSN_MRI', 'ResNetTSM_MRI', 'ResNetSlowFast_MRI', 'CFBI', 'MSTCN',\n 'ASRF', 'MoViNet', 'SwinTransformer3D', 'CTRGCN',\n 'TokenShiftVisionTransformer', 'AGCN2s', 'PPTSM_MobileNetV2',\n 'PPTSM_MobileNetV3', 'PPTSM_v2', 'ResNet3dSlowOnly', 'YOWO'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/46c12cb0-b8fb-4249-8642-3ffc2ee2b978.json b/docs/doc/46c12cb0-b8fb-4249-8642-3ffc2ee2b978.json
new file mode 100644
index 000000000..b3331ee87
--- /dev/null
+++ b/docs/doc/46c12cb0-b8fb-4249-8642-3ffc2ee2b978.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The code initializes an LSTM-based model for basketball action detection using PaddlePaddle's inference API, with preprocessing and GPU memory optimization functions. It loads a pre-trained model, predicts actions in videos, and saves results in JSON format without ASCII conversion.",
+ "details": [
+ {
+ "comment": "This code is for an LSTM-based inferencing model in the BasketballAction application. It includes functions for preprocessing, processing results, and using PaddlePaddle's inference API. The class InferModel initializes the model based on a configuration file that contains information such as model and parameter files, GPU memory, device ID, and thread settings for different tasks like NMS and classification scoring. It also includes a label mapping file for classification purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/lstm_infer.py\":0-35",
+ "content": "\"\"\"\nppTSM InferModel\n\"\"\"\nimport sys\nimport numpy as np\nimport json\nimport pickle\nimport time\nsys.path.append('../')\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config\nfrom utils.process_result import get_action_result\nimport reader\nfrom paddle.inference import Config\nfrom paddle.inference import create_predictor\nclass InferModel(object):\n \"\"\"lstm infer\"\"\"\n def __init__(self, cfg, name='ACTION'): \n name = name.upper()\n self.name = name\n model_file = cfg[name]['model_file']\n params_file = cfg[name]['params_file']\n gpu_mem = cfg[name]['gpu_mem']\n device_id = cfg[name]['device_id']\n self.topk = cfg[name]['topk']\n self.frame_offset = cfg[name]['nms_offset']\n self.nms_thread = cfg[name]['nms_thread']\n self.cls_thread = cfg[name]['classify_score_thread']\n self.iou_thread = cfg[name]['iou_score_thread']\n self.label_map_file = cfg['COMMON']['label_dic']"
+ },
+ {
+ "comment": "This code initializes an LSTM-based predictor model for action detection. It sets FPS, NMS ID, and configures the model to enable GPU usage and memory optimization. The code then creates a zero copy feed fetch operator and assigns input and output tensors for the infer method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/lstm_infer.py\":36-60",
+ "content": " self.fps = cfg['COMMON']['fps']\n self.nms_id = 5\n # model init\n config = Config(model_file, params_file)\n config.enable_use_gpu(gpu_mem, device_id)\n config.switch_ir_optim(True) # default true\n config.enable_memory_optim()\n # use zero copy\n config.switch_use_feed_fetch_ops(False)\n self.predictor = create_predictor(config)\n input_names = self.predictor.get_input_names()\n self.input1_tensor = self.predictor.get_input_handle(input_names[0])\n #self.input2_tensor = self.predictor.get_input_handle(input_names[1])\n output_names = self.predictor.get_output_names()\n self.output1_tensor = self.predictor.get_output_handle(output_names[0])\n self.output2_tensor = self.predictor.get_output_handle(output_names[1])\n def infer(self, input1_arr, input1_lod, input2_arr=None, input2_lod=None):\n \"\"\"infer\"\"\"\n self.input1_tensor.copy_from_cpu(input1_arr)\n self.input1_tensor.set_lod(input1_lod)"
+ },
+ {
+ "comment": "This code appears to be part of a Python class that uses LSTM models for action detection in basketball videos. It preprocesses input data, runs the predictor, and returns output1 and output2 as results. The pre_process function takes an input, creates lod (lengths of dimensions) and arranges sub-items in a specific order to prepare it for the model. The predict function uses a reader to iterate through data, performing action detection on each video frame and returning the results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/lstm_infer.py\":61-89",
+ "content": " if not input2_arr is None:\n self.input2_tensor.copy_from_cpu(input2_arr)\n self.input2_tensor.set_lod(input2_lod)\n self.predictor.run()\n output1 = self.output1_tensor.copy_to_cpu()\n output2 = self.output2_tensor.copy_to_cpu()\n # print(output.shape)\n return output1, output2\n def pre_process(self, input):\n \"\"\"pre process\"\"\"\n input_arr = []\n input_lod = [0]\n start_lod = 0\n end_lod = 0\n for sub_item in input:\n end_lod = start_lod + len(sub_item)\n input_lod.append(end_lod)\n input_arr.extend(sub_item)\n start_lod = end_lod\n input_arr = np.array(input_arr)\n return input_arr, [input_lod]\n def predict(self, infer_config, material):\n \"\"\"predict\"\"\"\n infer_reader = reader.get_reader(self.name, 'infer', infer_config, material=material)\n results = []\n for infer_iter, data in enumerate(infer_reader()):\n video_id = [[items[-2], items[-1]] for items in data]"
+ },
+ {
+ "comment": "This code is a function that performs action detection on video frames using an LSTM model. It preprocesses the input data, infers predictions from the model, selects the top-k detections for each frame, and then combines these results to generate an action detection result. The results are returned after post-processing with additional functions. A main function is also provided that can be used to run inference on a video with specific configuration settings.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/lstm_infer.py\":90-111",
+ "content": " input1 = [items[0] for items in data]\n input1_arr, input1_lod = self.pre_process(input1)\n output1, output2 = self.infer(input1_arr, input1_lod)\n predictions_id = output1 \n predictions_iou = output2\n for i in range(len(predictions_id)):\n topk_inds = predictions_id[i].argsort()[0 - self.topk:]\n topk_inds = topk_inds[::-1]\n preds_id = predictions_id[i][topk_inds]\n preds_iou = predictions_iou[i][0]\n results.append((video_id[i], preds_id.tolist(), topk_inds.tolist(), preds_iou.tolist()))\n predict_result = get_action_result(results, self.label_map_file, self.fps, \n self.cls_thread, self.iou_thread, \n self.nms_id, self.nms_thread, self.frame_offset)\n return predict_result\nif __name__ == \"__main__\":\n cfg_file = '/home/work/inference/configs/configs.yaml' \n cfg = parse_config(cfg_file)"
+ },
+ {
+ "comment": "The code loads and initializes a pre-trained LSTM model for action detection. It then retrieves the video features and proposal information from JSON files. Finally, it uses the loaded model to predict actions based on the given material (features and proposals) and saves the results in a json file named 'results.json'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/lstm_infer.py\":112-140",
+ "content": " model = InferModel(cfg)\n # proposal total\n prop_dict = {}\n for dataset in ['EuroCup2016', 'WorldCup2018']:\n prop_json = '/home/work/datasets/{}/feature_bmn/prop.json'.format(dataset)\n json_data = json.load(open(prop_json, 'r'))\n for item in json_data:\n basename = prop_json.replace('feature_bmn/prop.json', 'mp4')\n basename = basename + '/' + item['video_name'] + '.mp4'\n prop_dict[basename] = item['bmn_results']\n imgs_path = '/home/work/datasets/WorldCup2018/frames/6e577252c4004961ac7caa738a52c238'\n # feature\n feature_path = imgs_path.replace(\"frames\", \"features\") + '.pkl'\n video_features = pickle.load(open(feature_path, 'rb'))\n # proposal\n basename = imgs_path.replace('frames', 'mp4') + '.mp4'\n bmn_results = prop_dict[basename]\n material = {'feature': video_features, 'proposal': bmn_results}\n t0 = time.time()\n outputs = model.predict(cfg, material)\n t1 = time.time()\n results = {'actions': outputs}\n with open('results.json', 'w', encoding='utf-8') as f:"
+ },
+ {
+ "comment": "The code dumps the results in JSON format with indentation and without converting special characters to ASCII. Then, it writes this data to a file and prints the time taken in minutes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/lstm_infer.py\":141-144",
+ "content": " data = json.dumps(results, indent=4, ensure_ascii=False)\n f.write(data) \n print('cost time = {} min'.format((t1 - t0) / 60.0))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/46ef5c44-324f-4720-96fd-a36a2db1e04e.json b/docs/doc/46ef5c44-324f-4720-96fd-a36a2db1e04e.json
new file mode 100644
index 000000000..14416ed24
--- /dev/null
+++ b/docs/doc/46ef5c44-324f-4720-96fd-a36a2db1e04e.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The \"InferModel\" class is for audio inference, initializing the model and creating a predictor object. It takes input, performs inference, returns output, and measures time taken. The code loads an audio file, sets path, performs prediction, prints shape, first output, and time.",
+ "details": [
+ {
+ "comment": "This code defines a class named \"InferModel\" for audio inference. It initializes the model by reading configuration files, enabling GPU usage, and creating a predictor object. The input name and handle are stored for later use during inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/audio_infer.py\":0-36",
+ "content": "\"\"\"\nppTSM InferModel\n\"\"\"\nimport sys\nimport numpy as np\nimport time\nsys.path.append('../')\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config\nimport reader\nfrom paddle.inference import Config\nfrom paddle.inference import create_predictor\nclass InferModel(object):\n \"\"\"audio infer\"\"\"\n def __init__(self, cfg, name='AUDIO'): \n name = name.upper()\n self.name = name\n model_file = cfg[name]['model_file']\n params_file = cfg[name]['params_file']\n gpu_mem = cfg[name]['gpu_mem']\n device_id = cfg[name]['device_id']\n # model init\n config = Config(model_file, params_file)\n config.enable_use_gpu(gpu_mem, device_id)\n config.switch_ir_optim(True) # default true\n config.enable_memory_optim()\n # use zero copy\n config.switch_use_feed_fetch_ops(False)\n self.predictor = create_predictor(config)\n input_names = self.predictor.get_input_names()\n self.input_tensor = self.predictor.get_input_handle(input_names[0])"
+ },
+ {
+ "comment": "The code defines a model that takes audio input, performs inference using the predictor, and returns output. The predict method reads data from infer_config and for each iteration, it prepares inputs, runs inference, collects feature lists and pcm lists, then combines them into feature_values and pcm_values before returning.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/audio_infer.py\":38-68",
+ "content": " output_names = self.predictor.get_output_names()\n self.output_tensor = self.predictor.get_output_handle(output_names[0])\n def infer(self, input):\n \"\"\"infer\"\"\"\n self.input_tensor.copy_from_cpu(input)\n self.predictor.run()\n output = self.output_tensor.copy_to_cpu()\n return output\n def predict(self, infer_config):\n \"\"\"predict\"\"\"\n infer_reader = reader.get_reader(self.name, 'infer', infer_config)\n feature_list = []\n pcm_list = []\n for infer_iter, data in enumerate(infer_reader()):\n inputs = np.array(data, dtype = 'float32')\n output = self.infer(inputs)\n feature_list.append(np.squeeze(output))\n pcm_list.append(inputs)\n feature_values = np.vstack(feature_list)\n pcm_values = np.vstack(pcm_list)\n return feature_values, pcm_values\nif __name__ == \"__main__\":\n cfg_file = '/home/work/inference/configs/configs.yaml' \n cfg = parse_config(cfg_file)\n model = InferModel(cfg)"
+ },
+ {
+ "comment": "This code loads an audio file, sets the path for it in the configuration file, performs prediction on the model, prints the shape and first output of the prediction, and calculates and prints the time taken in minutes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/audio_infer.py\":70-79",
+ "content": " pcm_path = '/home/work/datasets/WorldCup2018/pcm/6e577252c4004961ac7caa738a52c238.pcm'\n t0 = time.time()\n cfg['AUDIO']['pcm_file'] = pcm_path\n outputs = model.predict(cfg)\n # outputs = model.infer(np.random.rand(32, 8, 3, 224, 224).astype(np.float32))\n t1 = time.time()\n print(outputs.shape)\n print(outputs[0])\n print('cost time = {} min'.format((t1 - t0) / 60.0))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/478e000f-a0fc-403b-88e2-d2301a2afd4b.json b/docs/doc/478e000f-a0fc-403b-88e2-d2301a2afd4b.json
new file mode 100644
index 000000000..2761be271
--- /dev/null
+++ b/docs/doc/478e000f-a0fc-403b-88e2-d2301a2afd4b.json
@@ -0,0 +1,145 @@
+{
+ "summary": "The code provides data conversion, tensor handling, input compatibility, gradient norm clipping, and gain calculations for PyTorch and PaddlePaddle. It also offers tensor initialization using various methods with backward compatibility.",
+ "details": [
+ {
+ "comment": "This code file contains utility functions for converting data from PyTorch to Paddle, filling tensors with values, zeroing out tensor values, and changing the tensor's dtype. It also includes a function that attempts to import PyTorch and converts corresponding data types accordingly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":0-48",
+ "content": "import math\nimport warnings\nimport numpy\nimport numpy as np\nfrom numpy import inf\nfrom paddle import Tensor, concat, reshape, nn\nimport paddle\nfrom typing import Union, Iterable\n_tensor_or_tensors = Union[paddle.Tensor, Iterable[paddle.Tensor]]\nimport paddle\nimport PIL\nimport numbers\nimport numpy as np\nfrom PIL import Image\nfrom paddle.vision.transforms import BaseTransform\nfrom paddle.vision.transforms import functional as F\ndef torch2paddle(data):\n try:\n import torch\n if isinstance(data, dict):\n np_data = {}\n for k, v in data.items():\n np_data[k] = paddle.to_tensor(v.detach().numpy())\n return np_data\n else:\n return paddle.to_tensor(data.detach().numpy())\n except:\n pass\ndef fill_(tensor: Tensor, value):\n return tensor.set_value(paddle.full_like(tensor, value))\ndef zero_(tensor: Tensor):\n return tensor.set_value(paddle.zeros_like(tensor))\ndef float_(tensor: Tensor):\n return paddle.to_tensor(tensor, dtype='float32')\ndef long_(tensor: Tensor):"
+ },
+ {
+ "comment": "The code provides three tensor conversion functions: `int64`, `int32`, and `byte` that convert tensors into specific data types. It also includes a class `ToPILImage` for converting images to PIL Image format with the option to specify mode and keys. The function `_apply_image` checks if the input is a tensor or numpy array, raises an error if not, and then proceeds to convert 2D or 3D images into PIL Image format by adding channel dimension if necessary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":49-82",
+ "content": " return paddle.to_tensor(tensor, dtype='int64')\ndef int_(tensor: Tensor):\n return paddle.to_tensor(tensor, dtype='int32')\ndef byte_(tensor: Tensor):\n return paddle.to_tensor(tensor, dtype='bool')\nclass ToPILImage(BaseTransform):\n def __init__(self, mode=None, keys=None):\n super(ToPILImage, self).__init__(keys)\n def _apply_image(self, pic):\n \"\"\"\n Args:\n pic (Tensor|np.ndarray): Image to be converted to PIL Image.\n Returns:\n PIL: Converted image.\n \"\"\"\n if not (isinstance(pic, paddle.Tensor) or isinstance(pic, np.ndarray)):\n raise TypeError('pic should be Tensor or ndarray. Got {}.'.format(\n type(pic)))\n elif isinstance(pic, paddle.Tensor):\n if pic.ndimension() not in {2, 3}:\n raise ValueError(\n 'pic should be 2/3 dimensional. Got {} dimensions.'.format(\n pic.ndimension()))\n elif pic.ndimension() == 2:\n # if 2D image, add channel dimension (CHW)"
+ },
+ {
+ "comment": "Code checks if the input 'pic' is a Paddle or numpy array, adjusts dimensions if necessary, and converts data types accordingly. If 'pic' is not compatible with the code, it raises an error. This code ensures that the input image is in the correct format for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":83-111",
+ "content": " pic = pic.unsqueeze(0)\n elif isinstance(pic, np.ndarray):\n if pic.ndim not in {2, 3}:\n raise ValueError(\n 'pic should be 2/3 dimensional. Got {} dimensions.'.format(\n pic.ndim))\n elif pic.ndim == 2:\n # if 2D image, add channel dimension (HWC)\n pic = np.expand_dims(pic, 2)\n npimg = pic\n if isinstance(pic, paddle.Tensor) and \"float\" in str(\n pic.numpy().dtype) and self.mode != 'F':\n pic = pic.mul(255).byte()\n if isinstance(pic, paddle.Tensor):\n npimg = np.transpose(pic.numpy(), (1, 2, 0))\n if not isinstance(npimg, np.ndarray):\n raise TypeError(\n 'Input pic must be a paddle.Tensor or NumPy ndarray, ' +\n 'not {}'.format(type(npimg)))\n if npimg.shape[2] == 1:\n expected_mode = None\n npimg = npimg[:, :, 0]\n if npimg.dtype == np.uint8:\n expected_mode = 'L'"
+ },
+ {
+ "comment": "This code checks the data type of npimg and sets the expected mode accordingly. It then compares the input's mode to the expected mode, raising a ValueError if they don't match. For 2D inputs with 2 channels, only 'LA' mode is supported; it sets self.mode to 'LA' if necessary. For 4-channel inputs, the code supports modes like 'RGBA', 'CMYK', and 'RGBX'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":112-135",
+ "content": " elif npimg.dtype == np.int16:\n expected_mode = 'I;16'\n elif npimg.dtype == np.int32:\n expected_mode = 'I'\n elif npimg.dtype == np.float32:\n expected_mode = 'F'\n if self.mode is not None and self.mode != expected_mode:\n raise ValueError(\n \"Incorrect self.mode ({}) supplied for input type {}. Should be {}\"\n .format(self.mode, np.dtype, expected_mode))\n self.mode = expected_mode\n elif npimg.shape[2] == 2:\n permitted_2_channel_modes = ['LA']\n if self.mode is not None and self.mode not in permitted_2_channel_modes:\n raise ValueError(\n \"Only self.modes {} are supported for 2D inputs\".format(\n permitted_2_channel_modes))\n if self.mode is None and npimg.dtype == np.uint8:\n self.mode = 'LA'\n elif npimg.shape[2] == 4:\n permitted_4_channel_modes = ['RGBA', 'CMYK', 'RGBX']"
+ },
+ {
+ "comment": "This code checks the input image mode and data type, ensuring it matches with the supported modes for 3D or 4D inputs. If the mode is not recognized, a ValueError is raised. If no mode is provided and the data type is np.uint8, it assigns the appropriate default mode (RGB or RGBA). Finally, if there is no mode specified and the input data type is unsupported, a TypeError is raised. The code is part of a class called Identity which seems to be an identity operator for neural networks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":136-160",
+ "content": " if self.mode is not None and self.mode not in permitted_4_channel_modes:\n raise ValueError(\n \"Only self.modes {} are supported for 4D inputs\".format(\n permitted_4_channel_modes))\n if self.mode is None and npimg.dtype == np.uint8:\n self.mode = 'RGBA'\n else:\n permitted_3_channel_modes = ['RGB', 'YCbCr', 'HSV']\n if self.mode is not None and self.mode not in permitted_3_channel_modes:\n raise ValueError(\n \"Only self.modes {} are supported for 3D inputs\".format(\n permitted_3_channel_modes))\n if self.mode is None and npimg.dtype == np.uint8:\n self.mode = 'RGB'\n if self.mode is None:\n raise TypeError('Input type {} is not supported'.format(\n npimg.dtype))\n return Image.fromarray(npimg, mode=self.mode)\nclass Identity(nn.Layer):\n r\"\"\"A placeholder identity operator that is argument-insensitive."
+ },
+ {
+ "comment": "This code defines a class \"Identity\" that performs identity forwarding and a function \"convert\" to convert data between Paddle and Torch formats. It takes a dictionary of data, converts it into either Paddle or Torch format based on the specified type, and returns a new dictionary with the converted data. If the data is a numpy ndarray, it can also be casted to a specific dtype.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":162-197",
+ "content": " Args:\n args: any argument (unused)\n kwargs: any keyword argument (unused)\n \"\"\"\n def __init__(self, *args, **kwargs):\n super(Identity, self).__init__()\n def forward(self, input):\n return input\ndef convert(data: dict, to, dtype=None):\n assert isinstance(data, dict)\n input = {}\n for k, v in data.items():\n if 'paddle' == to:\n if isinstance(v, np.ndarray):\n if dtype is not None:\n input[k] = paddle.to_tensor(v.astype(dtype))\n else:\n input[k] = paddle.to_tensor(v)\n else:\n input[k] = v\n elif 'torch' == to:\n try:\n import torch\n if isinstance(v, np.ndarray):\n if dtype is not None:\n input[k] = torch.tensor(v.astype(dtype))\n else:\n input[k] = torch.tensor(v)\n else:\n input[k] = v\n except:\n pass"
+ },
+ {
+ "comment": "This code defines a function that clips the gradient norm of an iterable of parameters. It takes in an iterable of Tensors (parameters) and a maximum norm value, computes the norm over all gradients, and modifies them in-place if necessary. The norm type can be specified as well, with options including 'inf' for infinity norm. If nonfinite norms are present, an error will occur unless error_if_nonfinite is set to False.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":198-222",
+ "content": " else:\n if isinstance(v, np.ndarray):\n input[k] = v.astype(to)\n else:\n input[k] = v\n return input\ndef clip_grad_norm_(parameters: _tensor_or_tensors,\n max_norm: float,\n norm_type: float = 2.0,\n error_if_nonfinite: bool = False) -> paddle.Tensor:\n r\"\"\"Clips gradient norm of an iterable of parameters.\n The norm is computed over all gradients together, as if they were\n concatenated into a single vector. Gradients are modified in-place.\n Args:\n parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a\n single Tensor that will have gradients normalized\n max_norm (float or int): max norm of the gradients\n norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for\n infinity norm.\n error_if_nonfinite (bool): if True, an error is thrown if the total\n norm of the gradients from :attr:``parameters`` is ``nan``,"
+ },
+ {
+ "comment": "This function calculates the total norm of parameters viewed as a single vector. It takes parameters and optional arguments max_norm and norm_type for the maximum allowed norm value and type of norm (inf or other), respectively. If no parameters are provided, it returns 0. If max_norm is infinity, it finds the maximum absolute value among parameters. Otherwise, it calculates the norm of gradients using the provided norm_type. If error_if_nonfinite is True and the total norm is NaN or Inf, a RuntimeError is raised.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":223-249",
+ "content": " ``inf``, or ``-inf``. Default: False (will switch to True in the future)\n Returns:\n Total norm of the parameters (viewed as a single vector).\n \"\"\"\n import time\n if isinstance(parameters, paddle.Tensor):\n parameters = [parameters]\n parameters = [p for p in parameters if p.grad is not None]\n detached_grads = [p.grad.detach() for p in parameters]\n max_norm = float(max_norm)\n norm_type = float(norm_type)\n if len(parameters) == 0:\n return paddle.to_tensor(0.)\n if norm_type == inf:\n norms = [p.abs().max() for p in parameters]\n total_norm = norms[0] if len(norms) == 1 else paddle.max(\n paddle.stack(norms))\n else:\n total_norm = paddle.norm(\n paddle.stack([paddle.norm(g, norm_type) for g in detached_grads]),\n norm_type)\n if error_if_nonfinite and paddle.logical_or(total_norm.isnan(),\n total_norm.isinf()):\n raise RuntimeError(\n f'The total norm of order {norm_type} for gradients from '"
+ },
+ {
+ "comment": "This code snippet is a part of the PaddleVideo framework's Ma-Net application. It checks if 'parameters' are non-finite and clips them if not. It also defines a function to find the maximum value in a tensor, similar to numpy's max() function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":250-273",
+ "content": " '`parameters` is non-finite, so it cannot be clipped. To disable '\n 'this error and scale the gradients by the non-finite norm anyway, '\n 'set `error_if_nonfinite=False`')\n clip_coef = max_norm / (total_norm + 1e-6)\n # Note: multiplying by the clamped coef is redundant when the coef is clamped to 1, but doing so\n # avoids a `if clip_coef < 1:` conditional which can require a CPU <=> device synchronization\n # when the gradients do not reside in CPU memory.\n clip_coef_clamped = paddle.clip(clip_coef, max=1.0)\n for i, p in enumerate(parameters):\n p.grad.set_value(detached_grads[i] * clip_coef_clamped) # fixed\n return total_norm\ndef max(a: paddle.Tensor, axis=0, keepdim=True):\n \"\"\"ndarray=numpy.array([[1, 2, 3, 4],\n [4, 3, 2, 1],\n [5, 6, 7, 8],\n [8, 7, 6, 5]])\n np.where(ndarray == np.max(ndarray))\n (array([2, 3]), array([3, 0]))\n ndarray[np.where(ndarray == np.max(ndarray))]\n array([8, 8])\n \"\"\"\n max_ = a.max(axis).unsqueeze(-1)"
+ },
+ {
+ "comment": "This code calculates the maximum value in a tensor and returns the corresponding index for each dimension. It also provides functions to gather data along different dimensions using gather or index_sample operations, depending on the shape of the input tensor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":274-306",
+ "content": " index = paddle.argmax(a, axis=axis, keepdim=keepdim)\n max_ = max_.numpy()\n index = index.numpy()\n # index = paddle.argmax(a, axis=axis, keepdim=keepdim)[-1].flatten()\n return max_, index\ndef gather(tmp: paddle.Tensor, ind: paddle.Tensor):\n shape = tmp.shape\n tmp = paddle.to_tensor(tmp)\n ind = paddle.to_tensor(ind)\n if len(shape) == 2:\n b = shape[0]\n return concat([\n reshape(paddle.gather(tmp[i, :], ind[i, :]), [1, -1])\n for i in range(b)\n ],\n axis=0)\n elif len(shape) == 3:\n out = []\n for i in range(tmp.shape[0]):\n _ = paddle.index_sample(tmp[i], ind[i])\n out.append(_)\n return paddle.to_tensor(out)\n elif len(shape) == 4:\n b, c, d = shape[:3]\n return concat([\n reshape(\n concat([\n reshape(\n concat([\n reshape(\n paddle.gather(tmp[i, j, k, :], ind[i, j, k, :]),"
+ },
+ {
+ "comment": "This code defines three functions (_no_grad_uniform_, _no_grad_normal_, and _no_grad_trunc_normal_) to initialize the weights of a tensor using different distributions while ensuring the computations are performed without gradient calculation. The main purpose is to prevent unnecessary memory usage and computation time for backpropagation in cases where gradients are not required.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":307-337",
+ "content": " [1, -1]) for k in range(d)\n ],\n axis=0), [1, d, -1]) for j in range(c)\n ],\n axis=0), [1, c, d, -1]) for i in range(b)\n ],\n axis=0)\n else:\n pass\n# These no_grad_* functions are necessary as wrappers around the parts of these\n# functions that use `with torch.no_grad()`. The JIT doesn't support context\n# managers, so these need to be implemented as builtins. Using these wrappers\n# lets us keep those builtins small and re-usable.\ndef _no_grad_uniform_(tensor, a, b):\n with paddle.no_grad():\n tensor.set_value(paddle.uniform(tensor.shape, min=a, max=b))\n return tensor\ndef _no_grad_normal_(tensor, mean, std):\n with paddle.no_grad():\n tensor.set_value(paddle.normal(shape=tensor.shape, mean=mean, std=std))\n return tensor\ndef _no_grad_trunc_normal_(tensor, mean, std, a, b):\n from scipy import special\n # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf"
+ },
+ {
+ "comment": "This function initializes the weights of a neural network using truncated normal distribution. It first computes the standard normal cumulative distribution function and checks if the mean is more than 2 std away from [a, b]. If so, it issues a warning. Then it generates uniform values in [l, u] and transforms them to truncated standard normal distribution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":338-363",
+ "content": " def norm_cdf(x):\n # Computes standard normal cumulative distribution function\n return (1. + math.erf(x / math.sqrt(2.))) / 2.\n if (mean < a - 2 * std) or (mean > b + 2 * std):\n warnings.warn(\n \"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. \"\n \"The distribution of values may be incorrect.\",\n stacklevel=2)\n with paddle.no_grad():\n # Values are generated by using a truncated uniform distribution and\n # then using the inverse CDF for the normal distribution.\n # Get upper and lower cdf values\n l = norm_cdf((a - mean) / std)\n u = norm_cdf((b - mean) / std)\n # Uniformly fill tensor with values from [l, u], then translate to\n # [2l-1, 2u-1].\n tensor.set_value(\n paddle.uniform(tensor.shape, min=2 * l - 1, max=2 * u - 1))\n # tensor.uniform_(2 * l - 1, 2 * u - 1)\n # Use inverse cdf transform for normal distribution to get truncated\n # standard normal\n tensor.set_value(special.erfinv(tensor))"
+ },
+ {
+ "comment": "The provided code contains functions for transforming, filling, and zeroing tensors. It also includes a function that calculates the recommended gain value for different nonlinearity functions. The gain values are 1 for Linear/Identity, Conv{1,2,3}D, and Sigmoid; and 5/3 for Tanh, while for ReLU it is sqrt(2).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":365-397",
+ "content": " # Transform to proper mean, std\n tensor.set_value(tensor.multiply(paddle.to_tensor(std * math.sqrt(2.))))\n tensor.add_(mean)\n # Clamp to ensure it's in the proper range\n tensor.clip_(min=a, max=b)\n return tensor\ndef _no_grad_fill_(tensor, val):\n with paddle.no_grad():\n tensor.set_value(paddle.full_like(tensor, fill_value=val))\n return tensor\ndef _no_grad_zero_(tensor):\n with paddle.no_grad():\n tensor.set_value(paddle.zeros_like(tensor))\n return tensor\ndef calculate_gain(nonlinearity, param=None):\n r\"\"\"Return the recommended gain value for the given nonlinearity function.\n The values are as follows:\n ================= ====================================================\n nonlinearity gain\n ================= ====================================================\n Linear / Identity :math:`1`\n Conv{1,2,3}D :math:`1`\n Sigmoid :math:`1`\n Tanh :math:`\\frac{5}{3}`\n ReLU :math:`\\sqrt{2}`"
+ },
+ {
+ "comment": "This function calculates the gain value for different non-linear functions used in neural networks, such as Leaky ReLU, SELU, and others. It returns appropriate gain values depending on the specified nonlinearity parameter, considering any optional parameters as well.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":398-424",
+ "content": " Leaky Relu :math:`\\sqrt{\\frac{2}{1 + \\text{negative\\_slope}^2}}`\n SELU :math:`\\frac{3}{4}`\n ================= ====================================================\n Args:\n nonlinearity: the non-linear function (`nn.functional` name)\n param: optional parameter for the non-linear function\n Examples:\n >>> gain = nn.init.calculate_gain('leaky_relu', 0.2) # leaky_relu with negative_slope=0.2\n \"\"\"\n linear_fns = [\n 'linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d',\n 'conv_transpose2d', 'conv_transpose3d'\n ]\n if nonlinearity in linear_fns or nonlinearity == 'sigmoid':\n return 1\n elif nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if param is None:\n negative_slope = 0.01\n elif not isinstance(param, bool) and isinstance(\n param, int) or isinstance(param, float):\n # True/False are instances of int, hence check above"
+ },
+ {
+ "comment": "The code defines two functions, `uniform_` and `normal_`, used for initializing tensors with uniform or normal distribution respectively. The `_no_grad_uniform_` function is used internally by `uniform_`. The `else` and `if-elif` structures are used to handle nonlinearity cases in the `init_` function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":425-453",
+ "content": " negative_slope = param\n else:\n raise ValueError(\n \"negative_slope {} not a valid number\".format(param))\n return math.sqrt(2.0 / (1 + negative_slope**2))\n elif nonlinearity == 'selu':\n return 3.0 / 4 # Value found empirically (https://github.com/pytorch/pytorch/pull/50664)\n else:\n raise ValueError(\"Unsupported nonlinearity {}\".format(nonlinearity))\ndef uniform_(tensor: Tensor, a: float = 0., b: float = 1.) -> Tensor:\n r\"\"\"Fills the input Tensor with values drawn from the uniform\n distribution :math:`\\mathcal{U}(a, b)`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n a: the lower bound of the uniform distribution\n b: the upper bound of the uniform distribution\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.uniform_(w)\n \"\"\"\n return _no_grad_uniform_(tensor, a, b)\ndef normal_(tensor: Tensor, mean: float = 0., std: float = 1.) -> Tensor:\n r\"\"\"Fills the input Tensor with values drawn from the normal"
+ },
+ {
+ "comment": "This code initializes or fills a tensor with values drawn from a normal distribution, truncating values outside the specified range [a, b]. The function `_no_grad_normal_` initializes a tensor with values from a normal distribution with given mean and standard deviation. The `trunc_normal_` function initializes a tensor with values from a truncated normal distribution within the specified bounds [a, b].",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":454-482",
+ "content": " distribution :math:`\\mathcal{N}(\\text{mean}, \\text{std}^2)`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n mean: the mean of the normal distribution\n std: the standard deviation of the normal distribution\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.normal_(w)\n \"\"\"\n return _no_grad_normal_(tensor, mean, std)\ndef trunc_normal_(tensor: Tensor,\n mean: float = 0.,\n std: float = 1.,\n a: float = -2.,\n b: float = 2.) -> Tensor:\n r\"\"\"Fills the input Tensor with values drawn from a truncated\n normal distribution. The values are effectively drawn from the\n normal distribution :math:`\\mathcal{N}(\\text{mean}, \\text{std}^2)`\n with values outside :math:`[a, b]` redrawn until they are within\n the bounds. The method used for generating the random values works\n best when :math:`a \\leq \\text{mean} \\leq b`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n mean: the mean of the normal distribution"
+ },
+ {
+ "comment": "This code defines several functions for initializing Tensor objects in PyTorch. These functions include `trunc_normal_`, `constant_`, `ones_`, and `zeros_`. The `trunc_normal_` function initializes a tensor with values drawn from a truncated normal distribution, while the other three functions fill the tensor with constant values (specified by the user), ones, or zeros respectively. These functions can be used to set the initial values of a tensor before training a neural network model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":483-525",
+ "content": " std: the standard deviation of the normal distribution\n a: the minimum cutoff value\n b: the maximum cutoff value\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.trunc_normal_(w)\n \"\"\"\n return _no_grad_trunc_normal_(tensor, mean, std, a, b)\ndef constant_(tensor: Tensor, val: float) -> Tensor:\n r\"\"\"Fills the input Tensor with the value :math:`\\text{val}`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n val: the value to fill the tensor with\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.constant_(w, 0.3)\n \"\"\"\n return _no_grad_fill_(tensor, val)\ndef ones_(tensor: Tensor) -> Tensor:\n r\"\"\"Fills the input Tensor with the scalar value `1`.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.ones_(w)\n \"\"\"\n return _no_grad_fill_(tensor, 1.)\ndef zeros_(tensor: Tensor) -> Tensor:\n r\"\"\"Fills the input Tensor with the scalar value `0`.\n Args:\n tensor: an n-dimensional `torch.Tensor`"
+ },
+ {
+ "comment": "The code above contains three functions: _no_grad_zero_, eye_, and dirac_. The _no_grad_zero_ function returns a tensor with all elements set to zero while preserving the identity of inputs in Linear layers. The eye_ function fills a 2-dimensional input tensor with an identity matrix, preserving as many inputs as possible in Linear layers. Lastly, the dirac_ function fills a 3, 4, or 5-dimensional input tensor with Dirac delta functions while preserving the identity of inputs in Convolutional layers, considering groups if specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":527-561",
+ "content": " Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.zeros_(w)\n \"\"\"\n return _no_grad_zero_(tensor)\ndef eye_(tensor):\n r\"\"\"Fills the 2-dimensional input `Tensor` with the identity\n matrix. Preserves the identity of the inputs in `Linear` layers, where as\n many inputs are preserved as possible.\n Args:\n tensor: a 2-dimensional `torch.Tensor`\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.eye_(w)\n \"\"\"\n if tensor.ndimension() != 2:\n raise ValueError(\"Only tensors with 2 dimensions are supported\")\n with paddle.no_grad():\n tensor.set_value(paddle.eye(*tensor.shape))\n return tensor\ndef dirac_(tensor, groups=1):\n r\"\"\"Fills the {3, 4, 5}-dimensional input `Tensor` with the Dirac\n delta function. Preserves the identity of the inputs in `Convolutional`\n layers, where as many input channels are preserved as possible. In case\n of groups>1, each group of channels preserves identity\n Args:\n tensor: a {3, 4, 5}-dimensional `torch.Tensor`"
+ },
+ {
+ "comment": "The code is a function that initializes the convolutional layer weights using Dirac delta distribution for 3, 4, or 5-dimensional tensors. It first checks if the tensor dimensions are supported and then raises an error if not. Then it calculates the number of output channels per group and minimum dimension. The code then zeroes out the tensor and initializes the weights with Dirac delta distribution for specified groups and dimensions, performing a temporal convolution in 3-dimensions or spatial convolution in 4-dimensions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":562-591",
+ "content": " groups (optional): number of groups in the conv layer (default: 1)\n Examples:\n >>> w = torch.empty(3, 16, 5, 5)\n >>> nn.init.dirac_(w)\n >>> w = torch.empty(3, 24, 5, 5)\n >>> nn.init.dirac_(w, 3)\n \"\"\"\n dimensions = tensor.ndimension()\n if dimensions not in [3, 4, 5]:\n raise ValueError(\n \"Only tensors with 3, 4, or 5 dimensions are supported\")\n sizes = tensor.shape\n if sizes[0] % groups != 0:\n raise ValueError('dim 0 must be divisible by groups')\n out_chans_per_grp = sizes[0] // groups\n min_dim = min(out_chans_per_grp, sizes[1])\n with paddle.no_grad():\n tensor.zero_()\n for g in range(groups):\n for d in range(min_dim):\n if dimensions == 3: # Temporal convolution\n tensor[g * out_chans_per_grp + d, d,\n tensor.shape[2] // 2] = 1\n elif dimensions == 4: # Spatial convolution\n tensor[g * out_chans_per_grp + d, d, tensor.shape[2] // 2,"
+ },
+ {
+ "comment": "This code defines several utility functions related to tensors in PaddlePaddle. The `_calculate_fan_in_and_fan_out` function calculates the fan-in and fan-out of a tensor, while the `LongTensor` function converts an input to a long tensor (dtype: int64). The `IntTensor` function does the same but with an int32 dtype. Lastly, `xavier_uniform_` initializes a tensor's parameters using Xavier Uniform initialization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":592-626",
+ "content": " tensor.shape[3] // 2] = 1\n else: # Volumetric convolution\n tensor[g * out_chans_per_grp + d, d, tensor.shape[2] // 2,\n tensor.shape[3] // 2, tensor.shape[4] // 2] = 1\n return tensor\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = tensor.dim()\n if dimensions < 2:\n raise ValueError(\n \"Fan in and fan out can not be computed for tensor with fewer than 2 dimensions\"\n )\n num_input_fmaps = tensor.shape[1] # .size(1)\n num_output_fmaps = tensor.shape[0] # .size(0)\n receptive_field_size = 1\n if tensor.dim() > 2:\n for s in tensor.shape[2:]:\n receptive_field_size *= s # fixed\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef LongTensor(x):\n return paddle.to_tensor(x, dtype='int64')\ndef IntTensor(x):\n return paddle.to_tensor(x, dtype='int32')\ndef xavier_uniform_(tensor: Tensor, gain: float = 1.) -> Tensor:"
+ },
+ {
+ "comment": "This code initializes the input tensor with values following the Glorot initialization method. It uses a uniform distribution and calculates the scaling factor 'a' based on gain, fan_in, and fan_out dimensions of the tensor. The resulting tensor is sampled from a uniform distribution between -a and a. The function xavier_normal_ is a variation that also fills the input tensor but with values from a normal distribution instead of uniform.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":627-654",
+ "content": " r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Understanding the difficulty of training deep feedforward\n neural networks` - Glorot, X. & Bengio, Y. (2010), using a uniform\n distribution. The resulting tensor will have values sampled from\n :math:`\\mathcal{U}(-a, a)` where\n .. math::\n a = \\text{gain} \\times \\sqrt{\\frac{6}{\\text{fan\\_in} + \\text{fan\\_out}}}\n Also known as Glorot initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n gain: an optional scaling factor\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))\n \"\"\"\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n std = gain * math.sqrt(2.0 / float(fan_in + fan_out))\n a = math.sqrt(3.0) * std # Calculate uniform bounds from standard deviation\n return _no_grad_uniform_(tensor, -a, a)\ndef xavier_normal_(tensor: Tensor, gain: float = 1.) -> Tensor:\n r\"\"\"Fills the input `Tensor` with values according to the method"
+ },
+ {
+ "comment": "This function initializes a tensor with Xavier/Glorot normal distribution, using a normal distribution and a scaling factor 'gain'. The resulting tensor values are sampled from the normal distribution N(0, std^2), where std = gain * sqrt(2 / (fan_in + fan_out)). It also includes a function _calculate_correct_fan that checks for valid modes 'fan_in' or 'fan_out'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":655-686",
+ "content": " described in `Understanding the difficulty of training deep feedforward\n neural networks` - Glorot, X. & Bengio, Y. (2010), using a normal\n distribution. The resulting tensor will have values sampled from\n :math:`\\mathcal{N}(0, \\text{std}^2)` where\n .. math::\n \\text{std} = \\text{gain} \\times \\sqrt{\\frac{2}{\\text{fan\\_in} + \\text{fan\\_out}}}\n Also known as Glorot initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n gain: an optional scaling factor\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.xavier_normal_(w)\n \"\"\"\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)\n std = gain * math.sqrt(2.0 / float(fan_in + fan_out))\n return _no_grad_normal_(tensor, 0., std)\ndef _calculate_correct_fan(tensor, mode):\n mode = mode.lower()\n valid_modes = ['fan_in', 'fan_out']\n if mode not in valid_modes:\n raise ValueError(\"Mode {} not supported, please use one of {}\".format(\n mode, valid_modes))\n fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)"
+ },
+ {
+ "comment": "This function fills a tensor with values from a uniform distribution according to the method described in He et al.'s (2015) paper, using either fan_in or fan_out mode. It also takes an optional argument for the negative slope of the rectifier used after this layer when 'leaky_relu' is specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":687-708",
+ "content": " return fan_in if mode == 'fan_in' else fan_out\ndef kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu'):\n r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Delving deep into rectifiers: Surpassing human-level\n performance on ImageNet classification` - He, K. et al. (2015), using a\n uniform distribution. The resulting tensor will have values sampled from\n :math:`\\mathcal{U}(-\\text{bound}, \\text{bound})` where\n .. math::\n \\text{bound} = \\text{gain} \\times \\sqrt{\\frac{3}{\\text{fan\\_mode}}}\n Also known as He initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n a: the negative slope of the rectifier used after this layer (only\n used with ``'leaky_relu'``)\n mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``\n preserves the magnitude of the variance of the weights in the\n forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the"
+ },
+ {
+ "comment": "This code is a PyTorch implementation of the Kaiming Uniform initialization method, used for initializing weights in neural networks. The function takes in a tensor and sets its values according to a uniform distribution with bounds calculated based on the tensor shape and nonlinearity (default is 'leaky_relu'). It also calculates the fan and gain based on the mode and nonlinearity to determine the standard deviation for the uniform distribution. The function then uses Paddle's `uniform` method to set the values in the tensor with the calculated bounds.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":709-731",
+ "content": " backwards pass.\n nonlinearity: the non-linear function (`nn.functional` name),\n recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')\n \"\"\"\n fan = _calculate_correct_fan(tensor, mode)\n gain = calculate_gain(nonlinearity, a)\n std = gain / math.sqrt(fan)\n bound = math.sqrt(\n 3.0) * std # Calculate uniform bounds from standard deviation\n with paddle.no_grad():\n tensor.set_value(paddle.uniform(tensor.shape, min=-bound, max=bound))\n return tensor\ndef kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu'):\n r\"\"\"Fills the input `Tensor` with values according to the method\n described in `Delving deep into rectifiers: Surpassing human-level\n performance on ImageNet classification` - He, K. et al. (2015), using a\n normal distribution. The resulting tensor will have values sampled from"
+ },
+ {
+ "comment": "Function initializes weights using Kaiming normal distribution. It takes in an n-dimensional tensor, a slope value for rectifier (optional), mode as 'fan_in' or 'fan_out', and nonlinearity function. It preserves weight variance in forward pass with 'fan_in' and backward pass with 'fan_out'. Recommended to use with 'relu' or 'leaky_relu'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":732-757",
+ "content": " :math:`\\mathcal{N}(0, \\text{std}^2)` where\n .. math::\n \\text{std} = \\frac{\\text{gain}}{\\sqrt{\\text{fan\\_mode}}}\n Also known as He initialization.\n Args:\n tensor: an n-dimensional `torch.Tensor`\n a: the negative slope of the rectifier used after this layer (only\n used with ``'leaky_relu'``)\n mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``\n preserves the magnitude of the variance of the weights in the\n forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the\n backwards pass.\n nonlinearity: the non-linear function (`nn.functional` name),\n recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).\n Examples:\n >>> w = torch.empty(3, 5)\n >>> kaiming_normal_(w, mode='fan_out', nonlinearity='relu')\n \"\"\"\n fan = _calculate_correct_fan(tensor, mode)\n gain = calculate_gain(nonlinearity, a)\n std = gain / math.sqrt(fan)\n with paddle.no_grad():"
+ },
+ {
+ "comment": "The code initializes a tensor with a (semi) orthogonal matrix based on the input tensor. The tensor must have at least 2 dimensions, and the trailing dimensions are flattened. If rows are less than columns, transpose the tensor. It computes the QR factorization of the tensor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":758-788",
+ "content": " tensor.set_value(paddle.normal(shape=tensor.shape, mean=0, std=std))\n return tensor\ndef orthogonal_(tensor, gain=1):\n r\"\"\"Fills the input `Tensor` with a (semi) orthogonal matrix, as\n described in `Exact solutions to the nonlinear dynamics of learning in deep\n linear neural networks` - Saxe, A. et al. (2013). The input tensor must have\n at least 2 dimensions, and for tensors with more than 2 dimensions the\n trailing dimensions are flattened.\n Args:\n tensor: an n-dimensional `torch.Tensor`, where :math:`n \\geq 2`\n gain: optional scaling factor\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.orthogonal_(w)\n \"\"\"\n if tensor.ndimension() < 2:\n raise ValueError(\"Only tensors with 2 or more dimensions are supported\")\n rows = tensor.shape[0] # .size(0)\n cols = tensor.numel() // rows\n flattened = tensor.new(rows, cols).normal_(0, 1)\n if rows < cols:\n flattened.t_()\n # Compute the qr factorization\n q, r = paddle.to_tensor(np.linalg.qr(flattened.numpy()))"
+ },
+ {
+ "comment": "Line 789: q, r = torch.qr(flattened) - Performs QR decomposition on flattened tensor\nLine 790-792: Make Q uniform according to https://arxiv.org/pdf/math-ph/0609050.pdf - Applies diag function on r, multiplies q by ph, and transposes q if rows < cols\nLine 794-798: with paddle.no_grad(): tensor.view_as(q).copy_(q) - Uses no_grad context manager to prevent gradients from being recorded during the operation\nLine 799: tensor.mul_(gain) - Multiplies tensor by a gain factor\nLine 802: return tensor - Returns the modified tensor after applying QR decomposition and scaling",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":789-821",
+ "content": " # q, r = torch.qr(flattened)\n # Make Q uniform according to https://arxiv.org/pdf/math-ph/0609050.pdf\n d = paddle.diag(r, 0)\n ph = d.sign()\n q *= ph\n if rows < cols:\n q.t_()\n with paddle.no_grad():\n tensor.view_as(q).copy_(q)\n tensor.mul_(gain)\n return tensor\ndef sparse_(tensor, sparsity, std=0.01):\n r\"\"\"Fills the 2D input `Tensor` as a sparse matrix, where the\n non-zero elements will be drawn from the normal distribution\n :math:`\\mathcal{N}(0, 0.01)`, as described in `Deep learning via\n Hessian-free optimization` - Martens, J. (2010).\n Args:\n tensor: an n-dimensional `torch.Tensor`\n sparsity: The fraction of elements in each column to be set to zero\n std: the standard deviation of the normal distribution used to generate\n the non-zero values\n Examples:\n >>> w = torch.empty(3, 5)\n >>> nn.init.sparse_(w, sparsity=0.1)\n \"\"\"\n if tensor.ndimension() != 2:\n raise ValueError(\"Only tensors with 2 dimensions are supported\")"
+ },
+ {
+ "comment": "This code defines a function that initializes the values of a tensor using the Kaiming normal distribution. It also includes a deprecated method for backward compatibility, warning users to use the new method instead.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/utils/api.py\":823-856",
+ "content": " rows, cols = tensor.shape\n num_zeros = int(math.ceil(sparsity * rows))\n with paddle.no_grad():\n tensor.normal_(0, std)\n for col_idx in range(cols):\n row_indices = paddle.randperm(rows)\n zero_indices = row_indices[:num_zeros]\n tensor[zero_indices, col_idx] = 0\n return tensor\n# for backward compatibility\ndef _make_deprecate(meth):\n new_name = meth.__name__\n old_name = new_name[:-1]\n def deprecated_init(*args, **kwargs):\n warnings.warn(\n \"nn.init.{} is now deprecated in favor of nn.init.{}.\".format(\n old_name, new_name),\n stacklevel=2)\n return meth(*args, **kwargs)\n deprecated_init.__doc__ = r\"\"\"\n {old_name}(...)\n .. warning::\n This method is now deprecated in favor of :func:`torch.nn.init.{new_name}`.\n See :func:`~torch.nn.init.{new_name}` for details.\"\"\".format(\n old_name=old_name, new_name=new_name)\n deprecated_init.__name__ = old_name\n return deprecated_init"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/47abd731-e6ef-4b69-9c7a-4f6bb0d86c93.json b/docs/doc/47abd731-e6ef-4b69-9c7a-4f6bb0d86c93.json
new file mode 100644
index 000000000..bd112fc11
--- /dev/null
+++ b/docs/doc/47abd731-e6ef-4b69-9c7a-4f6bb0d86c93.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code creates a class to calculate top-1 and possibly top-5 accuracy metrics in image classification tasks, tracking and averaging them during iteration, with support for multi-GPU scenarios using all-reduce operations.",
+ "details": [
+ {
+ "comment": "This code defines the class CenterCropMetric_MRI, a metric for a video processing framework. It initializes variables and tracks top1 accuracy during iteration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/center_crop_metric_MRI.py\":0-32",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass CenterCropMetric_MRI(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1, if_slowfast=0):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.top1 = []\n self.if_slowfast = if_slowfast\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter"
+ },
+ {
+ "comment": "This code snippet defines a class for calculating top-1 and possibly top-5 accuracy metrics in an image classification task. It collects the metrics for each batch during testing, then averages them at the end of all iterations. The code handles multi-GPU scenarios by performing all-reduce operations on the metric values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/center_crop_metric_MRI.py\":33-59",
+ "content": " \"\"\"\n labels = data[1]\n if self.if_slowfast:\n labels = data[2]\n top1 = paddle.metric.accuracy(input=outputs, label=labels, k=1)\n #top5 = paddle.metric.accuracy(input=outputs, label=labels, k=5)\n #NOTE(shipping): deal with multi cards validate\n if self.world_size > 1:\n top1 = paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM) / self.world_size\n # top5 = paddle.distributed.all_reduce(\n # top5, op=paddle.distributed.ReduceOp.SUM) / self.world_size\n self.top1.append(top1.numpy())\n #self.top5.append(top5.numpy())\n # preds ensemble\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n logger.info('[TEST] finished, avg_acc1= {}'.format("
+ },
+ {
+ "comment": "Calculates mean of top-1 accuracy across all samples in the batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/center_crop_metric_MRI.py\":60-60",
+ "content": " np.mean(np.array(self.top1))))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/487818af-e889-45da-85cd-ee8335367a08.json b/docs/doc/487818af-e889-45da-85cd-ee8335367a08.json
new file mode 100644
index 000000000..b3495b1c4
--- /dev/null
+++ b/docs/doc/487818af-e889-45da-85cd-ee8335367a08.json
@@ -0,0 +1,85 @@
+{
+ "summary": "The code introduces an AttentionLstmErnie class for ERNIE-based scenario classification, implementing an LSTM-based attention model for video tagging using text and audio data. It employs dropout, batch normalization, and Neural Machine Translation approach.",
+ "details": [
+ {
+ "comment": "This code defines the AttentionLstmErnie class, which extends the functionality of scenario-classify by incorporating text information. It uses ERNIE to extract text features and operates in either 'train' or 'infer' mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":0-33",
+ "content": "#!/usr/bin/env python\n# coding=utf-8\n\"\"\"\nattention lstm add ernie model\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport paddle\nimport paddle.static as static\nfrom .ernie import ErnieConfig, ErnieModel\nclass AttentionLstmErnie(object):\n \"\"\"\n Base on scenario-classify (image + audio), add text information\n use ERNIE to extract text feature\n \"\"\"\n def __init__(self, name, cfg, mode='train'):\n self.cfg = cfg\n self.name = name"
+ },
+ {
+ "comment": "This code initializes a model by setting attributes from a configuration file and calling the `get_config` function. The `get_config` function retrieves the model's configurations, including feature numbers, dimensions, data types, and more. It also gets mode-specific settings like batch size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":34-58",
+ "content": " self.mode = mode\n self.py_reader = None\n self.get_config()\n def get_config(self):\n \"\"\"get_config\n \"\"\"\n # get model configs\n self.feature_num = self.cfg.MODEL.feature_num\n self.feature_names = self.cfg.MODEL.feature_names\n self.feature_dims = self.cfg.MODEL.feature_dims\n self.feature_dtypes = self.cfg.MODEL.feature_dtypes\n self.feature_lod_level = self.cfg.MODEL.feature_lod_level\n self.num_classes = self.cfg.MODEL.num_classes\n self.embedding_size = self.cfg.MODEL.embedding_size\n self.lstm_size_img = self.cfg.MODEL.lstm_size_img\n self.lstm_size_audio = self.cfg.MODEL.lstm_size_audio\n self.ernie_freeze = self.cfg.MODEL.ernie_freeze\n self.lstm_pool_mode = self.cfg.MODEL.lstm_pool_mode\n self.drop_rate = self.cfg.MODEL.drop_rate\n self.loss_type = self.cfg.TRAIN.loss_type\n self.ernie_pretrain_dict_path = self.cfg.TRAIN.ernie_pretrain_dict_path\n # get mode configs\n self.batch_size = self.get_config_from_sec(self.mode, 'batch_size', 1)"
+ },
+ {
+ "comment": "This code is a part of the AttentionLSTMERNIE model. It initializes the number of GPUs, learning rate, weight decay, and other parameters for training mode. The function get_config_from_sec retrieves values from a configuration file using section and item names. The build_input function constructs input data by iterating over feature names, dimensions, data types, and lod levels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":59-84",
+ "content": " self.num_gpus = self.get_config_from_sec(self.mode, 'num_gpus', 1)\n if self.mode == 'train':\n self.learning_rate = self.get_config_from_sec(\n 'train', 'learning_rate', 1e-3)\n self.weight_decay = self.get_config_from_sec(\n 'train', 'weight_decay', 8e-4)\n self.num_samples = self.get_config_from_sec(\n 'train', 'num_samples', 5000000)\n self.decay_epochs = self.get_config_from_sec(\n 'train', 'decay_epochs', [5])\n self.decay_gamma = self.get_config_from_sec(\n 'train', 'decay_gamma', 0.1)\n def get_config_from_sec(self, sec, item, default=None):\n \"\"\"get_config_from_sec\"\"\"\n if sec.upper() not in self.cfg:\n return default\n return self.cfg[sec.upper()].get(item, default)\n def build_input(self, use_pyreader):\n \"\"\"\n build input\n \"\"\"\n self.feature_input = []\n for name, dim, dtype, lod_level in zip(self.feature_names,"
+ },
+ {
+ "comment": "Code initializes the reader for data feeding into the model, sets the label input, and defines a function \"ernie_encoder\" that extracts text features using the Ernie model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":85-107",
+ "content": " self.feature_dims,\n self.feature_dtypes,\n self.feature_lod_level):\n self.feature_input.append(\n static.data(shape=dim,\n lod_level=lod_level,\n dtype=dtype,\n name=name))\n self.label_input = static.data(shape=[self.num_classes],\n dtype='float32',\n name='label')\n self.py_reader = paddle.fluid.io.PyReader(feed_list=self.feature_input +\n [self.label_input],\n capacity=1024,\n iterable=True)\n def ernie_encoder(self):\n \"\"\"\n text feature extractor\n \"\"\"\n ernie_config = ErnieConfig(\n os.path.join(self.ernie_pretrain_dict_path, 'ernie_config.json'))"
+ },
+ {
+ "comment": "This code initializes an ErnieModel with features extracted from input data. If self.ernie_freeze is True, it freezes the ERNIE model's parameters to prevent further training. It then retrieves the sequence output and applies a dropout if in train mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":108-130",
+ "content": " if self.mode != 'train':\n ernie_config['attention_probs_dropout_prob'] = 0.0\n ernie_config['hidden_dropout_prob'] = 0.0\n src_ids = self.feature_input[2][:, 0]\n sent_ids = self.feature_input[2][:, 1]\n position_ids = self.feature_input[2][:, 2]\n task_ids = self.feature_input[2][:, 3]\n input_mask = self.feature_input[2][:, 4].astype('float32')\n ernie = ErnieModel(src_ids=src_ids,\n position_ids=position_ids,\n sentence_ids=sent_ids,\n task_ids=task_ids,\n input_mask=input_mask,\n config=ernie_config)\n enc_out = ernie.get_sequence_output()\n # to Freeze ERNIE param\n if self.ernie_freeze is True:\n enc_out.stop_gradient = True\n # ernie cnn\n enc_out_cnn = ernie.get_sequence_textcnn_output(enc_out, input_mask)\n enc_out_cnn_drop = paddle.nn.functional.dropout(enc_out_cnn, p=self.drop_rate, training=(self.mode=='train'))"
+ },
+ {
+ "comment": "This code defines a function called \"build_model\" that creates and returns the model for video tagging. The model takes image, audio, and text features as input to generate attention-based LSTM features from the image. It applies fully connected layers (fc) on the image features and then passes them through dynamic LSTMs to obtain the forward and backward LSTM outputs. These outputs are used in further processing for video tagging.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":131-153",
+ "content": " return enc_out_cnn_drop\n def build_model(self):\n \"\"\"build_model\n \"\"\"\n # ---------------- transfer from old paddle ---------------\n # get image,audio,text feature\n video_input_tensor = self.feature_input[0]\n audio_input_tensor = self.feature_input[1]\n self.ernie_feature = self.ernie_encoder()\n # ------image------\n lstm_forward_fc = static.nn.fc(x=video_input_tensor,\n size=self.lstm_size_img * 4,\n activation=None,\n bias_attr=False)\n lstm_forward, _ = paddle.fluid.layers.dynamic_lstm(input=lstm_forward_fc,\n size=self.lstm_size_img *\n 4,\n is_reverse=False,\n use_peepholes=True)\n lsmt_backward_fc = static.nn.fc(x=video_input_tensor,"
+ },
+ {
+ "comment": "This code defines a dynamic LSTM layer for image features, concatenates it with the backward pass, and applies dropout if in training mode. If 'text_guide' pooling mode is selected, it computes attention weights between text features and LSTM output using seq2seq attention. Otherwise, it uses an FC layer to reduce the dimensions of the LSTM output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":154-171",
+ "content": " size=self.lstm_size_img * 4,\n activation=None,\n bias_attr=None)\n lstm_backward, _ = paddle.fluid.layers.dynamic_lstm(input=lsmt_backward_fc,\n size=self.lstm_size_img *\n 4,\n is_reverse=True,\n use_peepholes=True)\n lstm_forward_img = paddle.concat(\n x=[lstm_forward, lstm_backward], axis=1)\n lstm_dropout = paddle.nn.functional.dropout(lstm_forward_img, p=self.drop_rate, training=(self.mode=='train'))\n if self.lstm_pool_mode == 'text_guide':\n lstm_weight = self.attention_weight_by_feature_seq2seq_attention(\n self.ernie_feature, lstm_dropout, self.lstm_size_img * 2)\n else:\n lstm_weight = static.nn.fc(x=lstm_dropout,"
+ },
+ {
+ "comment": "This code snippet is defining a LSTM model for processing both visual and audio inputs. It initializes a LSTM layer with dropout, applies element-wise multiplication with weights, performs sequence pooling on the output, and defines FC layers followed by LSTM for processing audio input. Regularization is applied to the audio LSTM layer using an L2 decay regularizer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":172-193",
+ "content": " size=1,\n activation='sequence_softmax',\n bias_attr=None)\n scaled = paddle.multiply(x=lstm_dropout,\n y=lstm_weight)\n self.lstm_pool = paddle.static.nn.sequence_pool(input=scaled,\n pool_type='sum')\n # ------audio------\n lstm_forward_fc_audio = static.nn.fc(\n x=audio_input_tensor,\n size=self.lstm_size_audio * 4,\n activation=None,\n bias_attr=paddle.ParamAttr(\n regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)))\n lstm_forward_audio, _ = paddle.fluid.layers.dynamic_lstm(\n input=lstm_forward_fc_audio,\n size=self.lstm_size_audio * 4,\n is_reverse=False,\n use_peepholes=True)\n lsmt_backward_fc_audio = static.nn.fc(x=audio_input_tensor,"
+ },
+ {
+ "comment": "This code is creating a dynamic LSTM for audio input, reversing it, concatenating the forward and backward outputs, applying dropout if in training mode, and then performing attention weight calculation based on the pooling mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":194-213",
+ "content": " size=self.lstm_size_audio * 4,\n activation=None,\n bias_attr=False)\n lstm_backward_audio, _ = paddle.fluid.layers.dynamic_lstm(\n input=lsmt_backward_fc_audio,\n size=self.lstm_size_audio * 4,\n is_reverse=True,\n use_peepholes=True)\n lstm_forward_audio = paddle.concat(\n x=[lstm_forward_audio, lstm_backward_audio], axis=1)\n lstm_dropout_audio = paddle.nn.functional.dropout(lstm_forward_audio, p=self.drop_rate, training=(self.mode=='train'))\n if self.lstm_pool_mode == 'text_guide':\n lstm_weight_audio = self.attention_weight_by_feature_seq2seq_attention(\n self.ernie_feature, lstm_dropout_audio,\n self.lstm_size_audio * 2)\n else:\n lstm_weight_audio = static.nn.fc(x=lstm_dropout_audio,\n size=1,"
+ },
+ {
+ "comment": "This code implements a LSTM-based attention model that combines audio and text data for video tagging. It consists of three main parts: LSTM layers, attention mechanism, and fully connected (FC) layer. The LSTM layers process the text and audio inputs separately and then concatenate them with the ERNIE feature. The attention mechanism is applied to calculate the weight for each LSTM output sequence. The FC layer has a softmax activation function if loss type is set to 'softmax', otherwise, it uses no activation when loss type is 'sigmoid'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":214-234",
+ "content": " activation='sequence_softmax',\n bias_attr=None)\n scaled_audio = paddle.multiply(x=lstm_dropout_audio,\n y=lstm_weight_audio)\n self.lstm_pool_audio = paddle.static.nn.sequence_pool(input=scaled_audio,\n pool_type='sum')\n lstm_concat = paddle.concat(\n x=[self.lstm_pool, self.lstm_pool_audio, self.ernie_feature],\n axis=1,\n name='final_concat')\n # lstm_concat = self.add_bn(lstm_concat)\n if self.loss_type == 'softmax':\n self.fc = static.nn.fc(x=lstm_concat,\n size=self.num_classes,\n activation='softmax')\n elif self.loss_type == 'sigmoid':\n self.fc = static.nn.fc(x=lstm_concat,\n size=self.num_classes,\n activation=None)"
+ },
+ {
+ "comment": "This code calculates attention weights for a feature sequence using a Neural Machine Translation approach. It expands the text feature across the sequence, concatenates it with the original sequence feature, and passes it through an FC layer with 'tanh' activation to calculate energy values. The calculated energy is then used to determine attention weights by feature.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":235-259",
+ "content": " self.logit = self.fc\n self.fc = paddle.nn.functional.sigmoid(self.fc)\n self.network_outputs = [self.fc]\n def attention_weight_by_feature_seq2seq_attention(\n self,\n text_feature,\n sequence_feature,\n sequence_feature_dim,\n name_prefix=\"seq2seq_attention\"):\n \"\"\"\n caculate weight by feature\n Neural Machine Translation by Jointly Learning to Align and Translate\n \"\"\"\n text_feature_expand = paddle.static.nn.sequence_expand(text_feature,\n sequence_feature,\n ref_level=0)\n sequence_text_concat = paddle.concat(\n x=[sequence_feature, text_feature_expand],\n axis=-1,\n name='video_text_concat')\n energy = static.nn.fc(x=sequence_text_concat,\n size=sequence_feature_dim,\n activation='tanh',"
+ },
+ {
+ "comment": "This function adds dropout and batch normalization to the LSTM concatenation. It projects the input to 8192 dimensions using an FC layer, applies batch normalization, and then applies a relu activation if in training mode. If not in training mode (is_test), it skips the batch normalization step. Finally, it applies dropout to the result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":260-284",
+ "content": " name=name_prefix + \"_tanh_fc\")\n weight_vector = static.nn.fc(x=energy,\n size=1,\n activation='sequence_softmax',\n bias_attr=None,\n name=name_prefix + \"_softmax_fc\")\n return weight_vector\n def add_bn(self, lstm_concat):\n \"\"\"\n v2.5 add drop out and batch norm\n \"\"\"\n input_fc_proj = static.nn.fc(\n x=lstm_concat,\n size=8192,\n activation=None,\n bias_attr=paddle.ParamAttr(\n regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)))\n input_fc_proj_bn = paddle.static.nn.batch_norm(\n input=input_fc_proj,\n act=\"relu\",\n is_test=(not self.mode == 'train'))\n input_fc_proj_dropout = paddle.nn.functional.dropout(\n input_fc_proj_bn,"
+ },
+ {
+ "comment": "This code defines an attention LSTM model using Ernie. It applies dropout, fully connected layers with batch normalization and dropout again. The optimizer function sets a learning rate that decays over specified epochs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":285-311",
+ "content": " p=self.drop_rate,\n training=(self.mode=='train'))\n input_fc_hidden = static.nn.fc(\n x=input_fc_proj_dropout,\n size=4096,\n activation=None,\n bias_attr=paddle.ParamAttr(\n regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)))\n input_fc_hidden_bn = paddle.static.nn.batch_norm(\n input=input_fc_hidden,\n act=\"relu\",\n is_test=(not self.mode == 'train'))\n input_fc_hidden_dropout = paddle.nn.functional.dropout(\n input_fc_hidden_bn,\n p=self.drop_rate,\n training=(self.mode=='train'))\n return input_fc_hidden_dropout\n def optimizer(self):\n \"\"\"\n optimizer\n \"\"\"\n assert self.mode == 'train', \"optimizer only can be get in train mode\"\n values = [\n self.learning_rate * (self.decay_gamma ** i)\n for i in range(len(self.decay_epochs) + 1)"
+ },
+ {
+ "comment": "This code defines a class that contains two methods: one for initializing an optimizer with piecewise decay learning rate and another for calculating the softlabel cross-entropy loss. The optimizer uses RMSProp algorithm and decays the learning rate based on defined epochs. The loss is calculated using the soft label version of cross entropy, suitable for certain types of neural networks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":312-333",
+ "content": " ]\n iter_per_epoch = self.num_samples / self.batch_size\n boundaries = [e * iter_per_epoch for e in self.decay_epochs]\n return paddle.optimizer.RMSProp(\n learning_rate=paddle.optimizer.lr.PiecewiseDecay(values=values,\n boundaries=boundaries),\n centered=True,\n weight_decay=paddle.regularizer.L2Decay(\n coeff=self.weight_decay))\n def softlabel_cross_entropy_loss(self):\n \"\"\"\n softlabel_cross_entropy_loss\n \"\"\"\n assert self.mode != 'infer', \"invalid loss calculationg in infer mode\"\n '''\n cost = paddle.nn.functional.cross_entropy(input=self.network_outputs[0], \\\n label=self.label_input)\n '''\n cost = paddle.nn.functional.cross_entropy(input=self.network_outputs[0], \\\n label=self.label_input,\n soft_label=True)"
+ },
+ {
+ "comment": "The code defines a loss function that takes in a loss type ('sigmoid' or others) and returns the calculated loss value. It includes functions for computing the sum of losses (sum_cost) and scaling it based on the number of GPUs (self.num_gpus). For the 'sigmoid' loss type, it uses binary cross-entropy with a reduction to be none, and calculates the mean loss over all batch elements. The scale operation is used to adjust the loss value for distributed training across multiple GPUs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":335-364",
+ "content": " cost = paddle.sum(x=cost, axis=-1)\n sum_cost = paddle.sum(x=cost)\n self.loss_ = paddle.scale(sum_cost,\n scale=self.num_gpus,\n bias_after_scale=False)\n return self.loss_\n def sigmoid_cross_entropy_loss(self):\n \"\"\"\n sigmoid_cross_entropy_loss\n \"\"\"\n assert self.mode != 'infer', \"invalid loss calculationg in infer mode\"\n cost = paddle.nn.functional.binary_cross_entropy(input=self.logit,\\\n label=self.label_input, reduction=None)\n cost = paddle.sum(x=cost, axis=-1)\n sum_cost = paddle.sum(x=cost)\n self.loss_ = paddle.scale(sum_cost,\n scale=self.num_gpus,\n bias_after_scale=False)\n return self.loss_\n def loss(self):\n \"\"\"\n loss\n \"\"\"\n if self.loss_type == 'sigmoid':\n return self.sigmoid_cross_entropy_loss()"
+ },
+ {
+ "comment": "This code defines several methods for a model class. The `softlabel_cross_entropy_loss()` method returns the soft label cross-entropy loss. The `outputs()` method returns the network outputs. The `feeds()` method returns the feature and label inputs based on the current mode. The `pyreader()` method returns the PyReader object. Finally, the `epoch_num()` method returns the number of training epochs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/models/attention_lstm_ernie.py\":365-399",
+ "content": " else:\n return self.softlabel_cross_entropy_loss()\n def outputs(self):\n \"\"\"\n get outputs\n \"\"\"\n return self.network_outputs\n def feeds(self):\n \"\"\"\n get feeds\n \"\"\"\n return self.feature_input if self.mode == 'infer' else self.feature_input + [\n self.label_input\n ]\n def pyreader(self):\n \"\"\"pyreader\"\"\"\n return self.py_reader\n def epoch_num(self):\n \"\"\"get train epoch num\"\"\"\n return self.cfg.TRAIN.epoch\n def load_test_weights_file(self, exe, weights, prog, place):\n \"\"\"\n load_test_weights_file\n \"\"\"\n load_vars = [x for x in prog.list_vars() \\\n if isinstance(x, paddle.framework.Parameter)]\n static.load_vars(exe,\n dirname=weights,\n vars=load_vars,\n filename=\"param\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/491e594e-2388-4319-80c0-77b4ae736363.json b/docs/doc/491e594e-2388-4319-80c0-77b4ae736363.json
new file mode 100644
index 000000000..98e025fbb
--- /dev/null
+++ b/docs/doc/491e594e-2388-4319-80c0-77b4ae736363.json
@@ -0,0 +1,45 @@
+{
+ "summary": "This code initializes a video/painting app with UI elements, connects events to functions for smooth operation, and displays \"Hi, This is EIVideo\" on stop.",
+ "details": [
+ {
+ "comment": "This code initializes the main window of a video application, sets its size and geometry, creates an empty list for capturing frames, and starts a timer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":0-31",
+ "content": "# -*- coding: utf-8 -*-\n# Form implementation generated from reading ui file 'GUI.ui'\n#\n# Created by: PyQt5 UI code generator 5.15.2\n#\n# WARNING: Any manual changes made to this file will be lost when pyuic5 is\n# run again. Do not edit this file unless you know what you are doing.\nfrom PyQt5 import QtCore, QtWidgets\nfrom PyQt5.QtGui import *\nfrom PyQt5.QtWidgets import *\nfrom PyQt5.QtCore import *\nfrom QEIVideo.widget.PaintBoard import PaintBoard\nclass Ui_MainWindow(object):\n def setupUi(self, MainWindow):\n MainWindow.setObjectName(\"EIVideo\")\n MainWindow.resize(1101, 751)\n self.centralwidget = QtWidgets.QWidget(MainWindow)\n self.centralwidget.setObjectName(\"centralwidget\")\n self.frame = QtWidgets.QFrame(self.centralwidget)\n self.frame.setGeometry(QtCore.QRect(20, 20, 1271, 771))\n self.frame.setFrameShadow(QtWidgets.QFrame.Raised)\n self.frame.setObjectName(\"frame\")\n self.cap = []\n self.all_frames = []\n self.fps = None\n self.timer = QTimer(self.frame)"
+ },
+ {
+ "comment": "This code is initializing UI elements in a window, setting up progress slider for video position tracking, adding a picture label and paint board for drawing, and enabling interaction with eraser checkbox and clear button.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":32-55",
+ "content": " self.time_label = QLabel('--/--', self.frame)\n self.progress_slider = QSlider(self.frame)\n self.progress_slider.setEnabled(True)\n self.progress_slider.setOrientation(Qt.Horizontal)\n self.progress_slider.setFixedWidth(710)\n self.progress_slider.setFixedHeight(20)\n self.progress_slider.setSingleStep(1) # \u8bbe\u7f6e\u53d8\u5316\u6b65\u957f\n self.progress_slider.setValue(0)\n self.progress_slider.sliderReleased.connect(self.update_video_position_func) # \u62d6\u62fd\u8fdb\u5ea6\u6761\n self.picturelabel = QtWidgets.QLabel(self.frame)\n self.picturelabel.setGeometry(30, 30, 810, 458)\n self.picturelabel.setText(\"\")\n self.picturelabel.setObjectName(\"picturelabel\")\n self.paintBoard = PaintBoard(self.frame)\n self.paintBoard.setGeometry(30, 30, 810, 458)\n self.cbtn_Eraser = QCheckBox(\"\u6a61\u76ae\u64e6\")\n self.cbtn_Eraser.setParent(self.frame)\n self.cbtn_Eraser.move(950, 40)\n self.cbtn_Eraser.clicked.connect(self.on_cbtn_eraser_clicked)\n self.btn_Clear = QPushButton(\"\u6e05\u7a7a\u753b\u677f\")"
+ },
+ {
+ "comment": "The code sets the parent object of btn_Clear to self.frame, moves the btn_Clear and label_penColor widgets to specific positions, connects a button click event to clear the paintBoard, fills a comboBox_penColor with color options, places it at a particular location, links its currentIndexChanged signal to on_pen_color_change function, and provides guidance for using EIVideo.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":56-76",
+ "content": " self.btn_Clear.setParent(self.frame) # \u8bbe\u7f6e\u7236\u5bf9\u8c61\u4e3a\u672c\u754c\u9762\n self.btn_Clear.move(950, 60)\n self.btn_Clear.clicked.connect(self.paintBoard.clear)\n self.label_penColor = QLabel(self.frame)\n self.label_penColor.setText(\"\u753b\u7b14\u989c\u8272\")\n self.label_penColor.move(990, 100)\n # \u83b7\u53d6\u989c\u8272\u5217\u8868(\u5b57\u7b26\u4e32\u7c7b\u578b)\n self.colorList = QColor.colorNames()\n self.comboBox_penColor = QComboBox(self.frame)\n self.fill_color_list(self.comboBox_penColor) # \u7528\u5404\u79cd\u989c\u8272\u586b\u5145\u4e0b\u62c9\u5217\u8868\n self.comboBox_penColor.move(1080, 80)\n self.comboBox_penColor.currentIndexChanged.connect(\n self.on_pen_color_change) # \u5173\u8054\u4e0b\u62c9\u5217\u8868\u7684\u5f53\u524d\u7d22\u5f15\u53d8\u66f4\u4fe1\u53f7\u4e0e\u51fd\u6570on_PenColorChange\n self.helplabel = QLabel()\n self.helplabel.setText(\"Hi,Welcome to use EIVideo\\n\"\n \"This is a guide for EIVideo,\\n\"\n \"please check\\n\"\n \"1. Choose 'Add' for a video\\n\"\n \"2. Click 'Play' to start playing\\n\"\n \"3. At this point, all functions \\n\""
+ },
+ {
+ "comment": "This code sets up a user interface layout for a painting application. It includes a help label, buttons to clear and switch the eraser, a color picker, and a 'GO' push button. The layout is organized in a vertical box and horizontal box arrangement.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":77-100",
+ "content": " \"are unlocked\\n\"\n \"4. Paint and enjoy it!\\n\")\n self.widget2 = QtWidgets.QWidget(self.frame)\n self.widget2.setGeometry(860, 60, 200, 300)\n self.widget2.setObjectName(\"widget2\")\n self.rightLayout = QtWidgets.QVBoxLayout(self.widget2)\n self.rightLayout.setContentsMargins(0, 0, 0, 0)\n self.rightLayout.setObjectName(\"rightLayout\")\n self.rightLayout.addWidget(self.helplabel)\n self.rightLayout.addSpacing(50)\n self.rightLayout.addWidget(self.cbtn_Eraser)\n self.rightLayout.addWidget(self.btn_Clear)\n self.colorLayout = QtWidgets.QHBoxLayout(self.widget2)\n self.colorLayout.setContentsMargins(0, 0, 0, 0)\n self.colorLayout.setObjectName('colorLayout')\n self.colorLayout.addWidget(self.label_penColor)\n self.colorLayout.addWidget(self.comboBox_penColor)\n self.rightLayout.addLayout(self.colorLayout)\n # pushButton_6 -> GO\n self.pushButton_6 = QtWidgets.QPushButton(self.frame)"
+ },
+ {
+ "comment": "This code is creating a UI for a video player. It sets the position and function of a play button, as well as defining layouts for other UI elements such as time display. The play button's click event is connected to a method called \"btn_func\" which takes the play button as an argument.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":101-121",
+ "content": " self.pushButton_6.setGeometry(870, 600, 150, 90)\n self.pushButton_6.setObjectName(\"pushButton_6\")\n self.pushButton_6.clicked.connect(self.infer)\n self.widget1 = QtWidgets.QWidget(self.frame)\n self.widget1.move(60, 520)\n self.widget1.setObjectName(\"widget1\")\n self.barLayout = QtWidgets.QVBoxLayout(self.widget1)\n self.barLayout.setContentsMargins(0, 0, 0, 0)\n self.barLayout.setObjectName(\"barLayout\")\n self.horizontalLayout = QtWidgets.QHBoxLayout(self.widget1)\n self.horizontalLayout.setContentsMargins(0, 0, 0, 0)\n self.horizontalLayout.setObjectName(\"horizontalLayout\")\n self.timeLayout = QtWidgets.QHBoxLayout(self.widget1)\n self.timeLayout.setContentsMargins(0, 0, 0, 0)\n self.timeLayout.setObjectName(\"horizontalLayout\")\n self.playbtn = QtWidgets.QPushButton(self.widget1)\n self.playbtn.setObjectName(\"playbtn\")\n self.playbtn.clicked.connect(lambda: self.btn_func(self.playbtn))\n self.horizontalLayout.addWidget(self.playbtn)"
+ },
+ {
+ "comment": "This code initializes two push buttons, connects their click events to a function, adds them to a horizontal layout, and adds the layout to a splitter. It also adds time-related widgets to another layout, and adds both layouts to the splitter.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":122-141",
+ "content": " self.pushButton_2 = QtWidgets.QPushButton(self.widget1)\n self.pushButton_2.setObjectName(\"pushButton_2\")\n self.pushButton_2.clicked.connect(lambda: self.btn_func(self.pushButton_2))\n self.horizontalLayout.addWidget(self.pushButton_2)\n self.pushButton_4 = QtWidgets.QPushButton(self.widget1)\n self.pushButton_4.setObjectName(\"pushButton_4\")\n self.pushButton_4.clicked.connect(lambda: self.btn_func(self.pushButton_4))\n self.horizontalLayout.addWidget(self.pushButton_4)\n self.timeLayout.addWidget(self.progress_slider)\n self.timeLayout.addWidget(self.time_label)\n self.barLayout.addSpacing(20)\n self.barLayout.addLayout(self.timeLayout)\n self.barLayout.addSpacing(30)\n self.barLayout.addLayout(self.horizontalLayout)\n self.splitter = QtWidgets.QSplitter(self.frame)\n self.splitter.setGeometry(QtCore.QRect(71, 670, 750, 20))\n self.splitter.setOrientation(QtCore.Qt.Horizontal)\n self.splitter.setObjectName(\"splitter\")"
+ },
+ {
+ "comment": "This code is creating a user interface for a main window of an application. It includes a label, progress bar, menu bar, and status bar. The window has a title and two buttons: \"GO\" and \"Play\". The `retranslateUi` function is used to set the window title and button labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":142-163",
+ "content": " self.label = QtWidgets.QLabel(self.splitter)\n self.label.setObjectName(\"label\")\n self.progressBar = QtWidgets.QProgressBar(self.splitter)\n self.progressBar.setProperty(\"value\", 0)\n self.progressBar.setObjectName(\"progressBar\")\n MainWindow.setCentralWidget(self.centralwidget)\n self.menubar = QtWidgets.QMenuBar(MainWindow)\n self.menubar.setGeometry(QtCore.QRect(0, 0, 1327, 23))\n self.menubar.setObjectName(\"menubar\")\n MainWindow.setMenuBar(self.menubar)\n self.statusbar = QtWidgets.QStatusBar(MainWindow)\n self.statusbar.setObjectName(\"statusbar\")\n MainWindow.setStatusBar(self.statusbar)\n self.retranslateUi(MainWindow)\n QtCore.QMetaObject.connectSlotsByName(MainWindow)\n def retranslateUi(self, MainWindow):\n _translate = QtCore.QCoreApplication.translate\n MainWindow.setWindowTitle(_translate(\"MainWindow\", \"MainWindow\"))\n self.pushButton_6.setText(_translate(\"MainWindow\", \"GO\"))\n self.playbtn.setText(_translate(\"MainWindow\", \"Play\"))"
+ },
+ {
+ "comment": "This code updates the text on three GUI elements in the \"MainWindow\" class. The first button is labeled \"Stop,\" the second button is labeled \"Add,\" and a label displays \"Hi, This is EIVideo.\"",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/gui/ui_main_window.py\":164-166",
+ "content": " self.pushButton_2.setText(_translate(\"MainWindow\", \"Stop\"))\n self.pushButton_4.setText(_translate(\"MainWindow\", \"Add\"))\n self.label.setText(_translate(\"MainWindow\", \"Hi, This is EIVideo\"))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/496e7372-817b-427a-803a-7bf4a1e61f3e.json b/docs/doc/496e7372-817b-427a-803a-7bf4a1e61f3e.json
new file mode 100644
index 000000000..2a29e6907
--- /dev/null
+++ b/docs/doc/496e7372-817b-427a-803a-7bf4a1e61f3e.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code uses Paddle Serving for postprocessing and PaddleVideo framework for video processing, initializing client, preprocessing input, sending data to server, receiving prediction, and printing output.",
+ "details": [
+ {
+ "comment": "This code snippet is importing necessary libraries and defining a function for postprocessing prediction outputs from a Paddle Serving client. The function takes raw predictions in the form of a numpy array and returns the postprocessed prediction as a dictionary containing any desired data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/serving_client.py\":0-31",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport argparse\nfrom typing import Any, Dict\nimport numpy as np\nfrom paddle_serving_client import Client\nfrom preprocess_ops import get_preprocess_func, np_softmax\ndef postprocess(fetch_map: Dict[str, np.ndarray]) -> Dict[str, Any]:\n \"\"\"postprocess\n Args:\n fetch_map (Dict[str, np.ndarray]): raw prediction\n Returns:\n Dict[str, Any]: postprocessed prediction\n \"\"\"\n score_list = fetch_map[\"outputs\"] # [b,num_classes]"
+ },
+ {
+ "comment": "The code defines a function that calculates the class id and probability based on scores, converts them to strings, and returns a dictionary with these values. It also includes a function for parsing arguments such as model name, serving client config file path, and URL to access the CPP serving.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/serving_client.py\":32-61",
+ "content": " fetch_dict = {\"class_id\": [], \"prob\": []}\n for score in score_list:\n score = np_softmax(score, axis=0)\n score = score.tolist()\n max_score = max(score)\n fetch_dict[\"class_id\"].append(score.index(max_score))\n fetch_dict[\"prob\"].append(max_score)\n fetch_dict[\"class_id\"] = str(fetch_dict[\"class_id\"])\n fetch_dict[\"prob\"] = str(fetch_dict[\"prob\"])\n return fetch_dict\ndef parse_args():\n # general params\n parser = argparse.ArgumentParser(\"PaddleVideo CPP Serving model script\")\n parser.add_argument(\"-n\",\n \"--name\",\n type=str,\n default=\"PPTSM\",\n help=\"model's name, such as PPTSM, PPTSN...\")\n parser.add_argument(\n \"-c\",\n \"--config\",\n type=str,\n help=\"serving client config file(serving_client_conf.prototxt) path\")\n parser.add_argument(\"--url\",\n type=str,\n default=\"127.0.0.1:9993\",\n help=\"url to access cpp serving\")"
+ },
+ {
+ "comment": "This code is a Python function that parses command line arguments, initializes a client object for video processing, preprocesses input video file, sends data to server, receives prediction, post-processes results and prints output. It uses the PaddleVideo framework with specific model configuration file and preprocessing function based on input name.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_serving/serving_client.py\":62-94",
+ "content": " parser.add_argument(\"--logid\", type=int, default=\"10000\", help=\"log id\")\n parser.add_argument(\"--input_file\",\n type=str,\n default=\"../../data/example.avi\",\n help=\"input video file\")\n return parser.parse_args()\nif __name__ == \"__main__\":\n # parse args\n args = parse_args()\n url = args.url\n logid = args.logid\n input_file_path = args.input_file\n model_name = args.name\n # get preprocess by model name\n preprocess = get_preprocess_func(model_name)\n # initialize client object & connect\n client = Client()\n client.load_client_config(args.config)\n client.connect([url])\n # preprocess\n feed, fetch = preprocess(input_file_path)\n # send data & get prediction from server\n fetch_map = client.predict(feed=feed, fetch=fetch)\n # postprocess & output\n result = postprocess(fetch_map)\n print(result)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/49a26df6-daa0-45ce-b445-d01a9f4aa985.json b/docs/doc/49a26df6-daa0-45ce-b445-d01a9f4aa985.json
new file mode 100644
index 000000000..d7e7dbbab
--- /dev/null
+++ b/docs/doc/49a26df6-daa0-45ce-b445-d01a9f4aa985.json
@@ -0,0 +1,50 @@
+{
+ "summary": "This code shows deploying PaddleServing for deep learning model prediction via HTTP using PP-TSM models and Docker on Linux. Issues with proxy, no response; check log file for errors at \"./deploy/python_serving/PipelineServingLogs/pipeline.log\". Refer to Serving's GitHub for more deployment types like RPC prediction service.",
+ "details": [
+ {
+ "comment": "This code provides an overview of deploying a model service using PaddleServing for deep learning predictions. It uses HTTP prediction service deployment as an example and suggests installing Serving through Docker on Linux platforms, while Windows is currently not supported.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":0-15",
+ "content": "English | [\u7b80\u4f53\u4e2d\u6587](./readme.md)\n# Model service deployment\n## Introduction\n[Paddle Serving](https://github.com/PaddlePaddle/Serving) aims to help deep learning developers easily deploy online prediction services, support one-click deployment of industrial-grade service capabilities, high concurrency between client and server Efficient communication and support for developing clients in multiple programming languages.\nThis section takes the HTTP prediction service deployment as an example to introduce how to use PaddleServing to deploy the model service in PaddleVideo. Currently, only Linux platform deployment is supported, and Windows platform is not currently supported.\n## Serving installation\nThe Serving official website recommends using docker to install and deploy the Serving environment. First, you need to pull the docker environment and create a Serving-based docker.\n```bash\n# start GPU docker\ndocker pull paddlepaddle/serving:0.7.0-cuda10.2-cudnn7-devel\nnvidia-docker run -p 9292:9292 --name test -dit paddlepaddle/serving:0.7.0-cuda10.2-cudnn7-devel bash"
+ },
+ {
+ "comment": "Install PaddleServing server and client packages for CPU and GPU environments, depending on the deployment type.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":16-40",
+ "content": "nvidia-docker exec -it test bash\n# start CPU docker\ndocker pull paddlepaddle/serving:0.7.0-devel\ndocker run -p 9292:9292 --name test -dit paddlepaddle/serving:0.7.0-devel bash\ndocker exec -it test bash\n```\nAfter entering docker, you need to install Serving-related python packages.\n```bash\npython3.7 -m pip install paddle-serving-client==0.7.0\npython3.7 -m pip install paddle-serving-app==0.7.0\npython3.7 -m pip install faiss-cpu==1.7.1post2\n#If it is a CPU deployment environment:\npython3.7 -m pip install paddle-serving-server==0.7.0 #CPU\npython3.7 -m pip install paddlepaddle==2.2.0 # CPU\n#If it is a GPU deployment environment\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post102 # GPU with CUDA10.2 + TensorRT6\npython3.7 -m pip install paddlepaddle-gpu==2.2.0 # GPU with CUDA10.2\n#Other GPU environments need to confirm the environment and then choose which one to execute\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post101 # GPU with CUDA10.1 + TensorRT6\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post112 # GPU with CUDA11.2 + TensorRT8"
+ },
+ {
+ "comment": "This code snippet provides instructions on how to deploy a behavior recognition service using PaddleServing. It explains that the model must be converted into a Serving model and provides an example of PP-TSM model conversion process. The user is guided to enter the PaddleVideo directory, download the trained PP-TSM model, convert it into an inference model, and finally, provide an option to download a pre-converted inference model if desired.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":41-62",
+ "content": "```\n* If the installation speed is too slow, you can change the source through `-i https://pypi.tuna.tsinghua.edu.cn/simple` to speed up the installation process\n* For more environment and corresponding installation packages, see: https://github.com/PaddlePaddle/Serving/blob/v0.9.0/doc/Install_Linux_Env_CN.md\n## Behavior recognition service deployment\n### Model conversion\nWhen using PaddleServing for service deployment, you need to convert the saved inference model into a Serving model. The following uses the PP-TSM model as an example to introduce how to deploy the behavior recognition service.\n- Download the trained PP-TSM model and convert it into an inference model:\n ```bash\n # Enter PaddleVideo directory\n cd PaddleVideo\n wget -P data/ https://videotag.bj.bcebos.com/PaddleVideo-release2.1/PPTSM/ppTSM_k400_uniform.pdparams\n python3.7 tools/export_model.py \\\n -c configs/recognition/pptsm/pptsm_k400_frames_uniform.yaml \\\n -p data/ppTSM_k400_uniform.pdparams \\\n -o inference/ppTSM\n ```\n- We also provide the converted inference model, download and unzip by the following command"
+ },
+ {
+ "comment": "This code downloads a pre-trained model and converts it into a format suitable for server deployment using paddle_serving_client. The converted model is saved in the specified directory with the corresponding program and parameter files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":63-82",
+ "content": " ```bash\n mkdir ./inference\n wget -nc -P ./inference https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM.zip --no-check-certificate\n pushd ./inference\n unzip ppTSM.zip\n popd\n ```\n- Use paddle_serving_client to convert the converted inference model into a model format that is easy for server deployment:\n ```bash\n python3.7 -m paddle_serving_client.convert \\\n --dirname inference/ppTSM \\\n --model_filename ppTSM.pdmodel \\\n --params_filename ppTSM.pdiparams \\\n --serving_server ./deploy/python_serving/ppTSM_serving_server/ \\\n --serving_client ./deploy/python_serving/ppTSM_serving_client/\n ```\n | parameter | type | default value | description |\n | ----------------- | ---- | ------------------ | ------- -------------------------------------------------- --- |\n | `dirname` | str | - | The storage path of the model file to be converted. The program structure file and parameter file are saved in this directory. |\n | `model_filename` | str | None | The name of the file storing the model In"
+ },
+ {
+ "comment": "This code defines the required parameters for converting a PaddleVideo PP-TSM inference model. Upon successful conversion, it creates `ppTSM_serving_server` and `ppTSM_serving_client` folders with necessary files for the converted model's serving.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":82-93",
+ "content": "ference Program structure that needs to be converted. If set to None, use `__model__` as the default filename |\n | `params_filename` | str | None | File name where all parameters of the model to be converted are stored. It needs to be specified if and only if all model parameters are stored in a single binary file. If the model parameters are stored in separate files, set it to None |\n | `serving_server` | str | `\"serving_server\"` | The storage path of the converted model files and configuration files. Default is serving_server |\n | `serving_client` | str | `\"serving_client\"` | The converted client configuration file storage path. Default is serving_client |\nAfter the PP-TSM inference model is converted, there will be additional folders of `ppTSM_serving_server` and `ppTSM_serving_client` in the current folder, with the following formats:\n ```bash\n PaddleVideo/deploy/python_serving\n \u251c\u2500\u2500 ppTSM_serving_server\n \u251c\u2500\u2500 ppTSM.pdiparams\n \u251c\u2500\u2500 ppTSM.pdmodel\n \u251c\u2500\u2500 serving_server_conf.prototxt"
+ },
+ {
+ "comment": "This code snippet is modifying the model configuration files `serving_server_conf.prototxt` and `serving_client_conf.stream.prototxt`. It changes the `alias_name` under `fetch_var` to \"outputs\" in both files for compatibility with different models during deployment. This allows the inference and deployment of various models without modifying the code, simply by updating the configuration file's alias names.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":94-118",
+ "content": " \u2514\u2500\u2500 serving_server_conf.stream.prototxt\n \u251c\u2500\u2500 ppTSM_serving_client\n \u251c\u2500\u2500 serving_client_conf.prototxt\n \u2514\u2500\u2500 serving_client_conf.stream.prototxt\n ```\nAfter getting the model files, you need to modify the files `serving_server_conf.prototxt` under `ppTSM_serving_server` and `ppTSM_serving_client` respectively, and change `alias_name` under `fetch_var` in both files to `outputs`\n**Remarks**: In order to be compatible with the deployment of different models, Serving provides the function of input and output renaming. In this way, when different models are inferred and deployed, they only need to modify the `alias_name` of the configuration file, and the inference deployment can be completed without modifying the code.\nThe modified `serving_server_conf.prototxt` looks like this:\n```yaml\nfeed_var {\n name: \"data_batch_0\"\n alias_name: \"data_batch_0\"\n is_lod_tensor: false\n feed_type: 1\n shape: 8\n shape: 3\n shape: 224\n shape: 224\n}\nfetch_var {\n name: \"linear_2.tmp_1\"\n alias_name: \"outputs\"\n is_lod_tensor: false"
+ },
+ {
+ "comment": "This code snippet is for starting the PaddleVideo pipeline service in Python using the recognition_web_service.py script. The `-n` flag specifies the name of the model, and the `-c` flag points to the configuration file for the pipeline service.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":119-144",
+ "content": " fetch_type: 1\n shape: 400\n}\n```\n### Service deployment and requests\nThe `python_serving` directory contains the code for starting the pipeline service, C++ serving service (TODO) and sending prediction requests, including:\n```bash\n__init__.py\nconfigs/xxx.yaml # start the configuration file of the pipeline service\npipeline_http_client.py # python script for sending pipeline prediction request via http\npipeline_rpc_client.py # python script for sending pipeline prediction request in rpc mode\nrecognition_web_service.py # python script that starts the pipeline server\nutils.py # common functions used in inference, such as parse_file_paths, numpy_to_base64, video_to_numpy\n```\n#### Python Serving\n- Go to the working directory:\n```bash\ncd deploy/python_serving\n```\n- Start the service:\n```bash\n# Start in the current command line window and stay in front\npython3.7 recognition_web_service.py -n PPTSM -c configs/PP-TSM.yaml\n# Start in the background, the logs printed during the process will be redirected and saved to log.txt"
+ },
+ {
+ "comment": "This code is running a web service for model prediction and two client scripts to send prediction requests via HTTP and RPC, printing the results in the command line. The result shows an example output with probabilities and labels for a given input video file. If no result is returned or there's an output decoding error, it might be related to the proxy setting when starting the service.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":145-174",
+ "content": "python3.7 recognition_web_service.py -n PPTSM -c configs/PP-TSM.yaml &>log.txt &\n```\n- send request:\n```bash\n# Send a prediction request in http and receive the result\npython3.7 pipeline_http_client.py -i ../../data/example.avi\n# Send a prediction request in rpc and receive the result\npython3.7 pipeline_rpc_client.py -i ../../data/example.avi\n```\nAfter a successful run, the results of the model prediction will be printed in the cmd window, and the results are as follows:\n```bash\n# http method print result\n{'err_no': 0, 'err_msg': '', 'key': ['label', 'prob'], 'value': [\"['archery']\", '[0.9907388687133789]'], 'tensors ': []}\n# The result of printing in rpc mode\nPipelineClient::predict pack_data time:1645631086.764019\nPipelineClient::predict before time:1645631086.8485317\nkey: \"label\"\nkey: \"prob\"\nvalue: \"[\\'archery\\']\"\nvalue: \"[0.9907388687133789]\"\n```\n## FAQ\n**Q1**: No result is returned after the request is sent or an output decoding error is prompted\n**A1**: Do not set the proxy when starting the service an"
+ },
+ {
+ "comment": "Closing the proxy before starting the service and sending request using \"unset https_proxy; unset http_proxy\". No response after server started, check log file for error message at \"./deploy/python_serving/PipelineServingLogs/pipeline.log\". For more deployment types like RPC prediction service, refer to Serving's GitHub official website.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme_en.md\":174-184",
+ "content": "d sending the request. You can close the proxy before starting the service and sending the request. The command to close the proxy is:\n```\nunset https_proxy\nunset http_proxy\n```\n**Q2**: There is no response after the server is started, and it has been stopped at `start proxy service`\n**A2**: It is likely that a problem was encountered during the startup process. You can view the detailed error message in the `./deploy/python_serving/PipelineServingLogs/pipeline.log` log file\nFor more service deployment types, such as `RPC prediction service`, you can refer to Serving's [github official website](https://github.com/PaddlePaddle/Serving/tree/v0.7.0/examples)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4a0caae3-5a32-4b12-8eaf-ea69856a0f76.json b/docs/doc/4a0caae3-5a32-4b12-8eaf-ea69856a0f76.json
new file mode 100644
index 000000000..9b137a708
--- /dev/null
+++ b/docs/doc/4a0caae3-5a32-4b12-8eaf-ea69856a0f76.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code snippet is providing two links to tutorials, one in English and the other in Simplified Chinese (\u7b80\u4f53\u4e2d\u6587). The English tutorial can be accessed at \"../../zh-CN/tutorials/accelerate.md\" and the Chinese one at the current location \"PaddleVideo/english_documents/tutorials/accelerate.md\".",
+ "details": [
+ {
+ "comment": "This code snippet is providing two links to tutorials, one in English and the other in Simplified Chinese (\u7b80\u4f53\u4e2d\u6587). The English tutorial can be accessed at \"../../zh-CN/tutorials/accelerate.md\" and the Chinese one at the current location \"PaddleVideo/english_documents/tutorials/accelerate.md\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/accelerate.md\":0-0",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../zh-CN/tutorials/accelerate.md) | English"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4a6e8677-5817-4bc6-b864-c058c6d70d4c.json b/docs/doc/4a6e8677-5817-4bc6-b864-c058c6d70d4c.json
new file mode 100644
index 000000000..e00e964fe
--- /dev/null
+++ b/docs/doc/4a6e8677-5817-4bc6-b864-c058c6d70d4c.json
@@ -0,0 +1,50 @@
+{
+ "summary": "The code imports necessary libraries, sets up a DALI reader, defines TSN_Dali_loader class, initializes parallel video preprocessing, handles potential import errors, and returns output and label using PaddleOps for normalization.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries, sets up logger, and attempts to import DALI pipeline and related functions for creating a generic iterator for PaddlePaddle. If any of these imports fail, it falls back by setting the respective variable as an object.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":0-31",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nimport math\nimport paddle\nfrom paddle.distributed import ParallelEnv\nimport paddle.distributed as dist\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ntry:\n from nvidia.dali.pipeline import Pipeline\n import nvidia.dali.ops as ops\n import nvidia.dali.types as types\n import tempfile\n from nvidia.dali.plugin.paddle import DALIGenericIterator\nexcept:\n Pipeline = object"
+ },
+ {
+ "comment": "The code defines a class `TSN_Dali_loader` that initializes attributes related to batch size, file path, number of segments, segment length, input and target image sizes. It also sets variables for distributed training, data normalization, and builds a DALI reader for training data using shuffled full lines from the file path.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":34-64",
+ "content": "def get_input_data(data):\n return paddle.to_tensor(data[0]['image']), paddle.to_tensor(\n data[0]['label'])\nclass TSN_Dali_loader(object):\n def __init__(self, cfg):\n self.batch_size = cfg.batch_size\n self.file_path = cfg.file_path\n self.num_seg = cfg.num_seg\n self.seglen = cfg.seglen\n self.short_size = cfg.short_size\n self.target_size = cfg.target_size\n # set num_shards and shard_id when distributed training is implemented\n self.num_shards = dist.get_world_size()\n self.shard_id = ParallelEnv().local_rank\n self.dali_mean = cfg.mean * (self.num_seg * self.seglen)\n self.dali_std = cfg.std * (self.num_seg * self.seglen)\n def build_dali_reader(self):\n \"\"\"\n build dali training reader\n \"\"\"\n def reader_():\n with open(self.file_path) as flist:\n full_lines = [line for line in flist]\n if (not hasattr(reader_, 'seed')):\n reader_.seed = 0\n random.Random(reader_.seed).shuffle(full_lines)"
+ },
+ {
+ "comment": "This code snippet initializes a reader and distributes the data evenly across multiple shards. It calculates the number of lines to be assigned to each shard based on the total number of lines and the number of shards. It then ensures that the full_lines list is an even multiple of the total_lines by appending additional items if necessary. The snippet asserts that the length of full_lines equals total_lines, assigns lines to trainers based on their shard ID, and logs information about the distribution of data among shards.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":65-87",
+ "content": " logger.info(f\"reader shuffle seed: {reader_.seed}.\")\n if reader_.seed is not None:\n reader_.seed += 1\n per_node_lines = int(\n math.ceil(len(full_lines) * 1.0 / self.num_shards))\n total_lines = per_node_lines * self.num_shards\n # aligned full_lines so that it can evenly divisible\n full_lines += full_lines[:(total_lines - len(full_lines))]\n assert len(full_lines) == total_lines\n # trainer get own sample\n lines = full_lines[self.shard_id:total_lines:self.num_shards]\n assert len(lines) == per_node_lines\n logger.info(\n f\"shard_id: {self.shard_id}, trainer_count: {self.num_shards}\"\n )\n logger.info(\n f\"read videos from {self.shard_id * per_node_lines}, \"\n f\"length: {per_node_lines}, \"\n f\"lines length: {len(lines)}, \""
+ },
+ {
+ "comment": "This code initializes a PaddlePaddle VideoPipe instance, loading and preprocessing video files in parallel for training. It sets the batch size, number of threads, device ID, file list, sequence length, number of segments, segment length, resize shorter scale, crop target size, whether it's in training mode, and the number of shards and shard ID.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":88-110",
+ "content": " f\"total: {len(full_lines)}\")\n video_files = ''.join([item for item in lines])\n tf = tempfile.NamedTemporaryFile()\n tf.write(str.encode(video_files))\n tf.flush()\n video_files = tf.name\n device_id = ParallelEnv().local_rank\n logger.info(f'---------- device_id: {device_id} -----------')\n pipe = VideoPipe(batch_size=self.batch_size,\n num_threads=1,\n device_id=device_id,\n file_list=video_files,\n sequence_length=self.num_seg * self.seglen,\n num_seg=self.num_seg,\n seg_length=self.seglen,\n resize_shorter_scale=self.short_size,\n crop_target_size=self.target_size,\n is_training=True,\n num_shards=self.num_shards,\n shard_id=self.shard_id,"
+ },
+ {
+ "comment": "This code initializes a DALI (Data Augmentation Library for Images) generic iterator to load video data from a file list, and returns it. It uses a VideoPipe class to define the pipeline configuration, including parameters such as batch size, number of threads, device ID, sequence length, and more.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":111-141",
+ "content": " dali_mean=self.dali_mean,\n dali_std=self.dali_std)\n logger.info(\n 'initializing dataset, it will take several minutes if it is too large .... '\n )\n video_loader = DALIGenericIterator([pipe], ['image', 'label'],\n len(lines),\n dynamic_shape=True,\n auto_reset=True)\n return video_loader\n dali_reader = reader_()\n return dali_reader\nclass VideoPipe(Pipeline):\n def __init__(self,\n batch_size,\n num_threads,\n device_id,\n file_list,\n sequence_length,\n num_seg,\n seg_length,\n resize_shorter_scale,\n crop_target_size,\n is_training=False,\n initial_prefetch_size=20,\n num_shards=1,"
+ },
+ {
+ "comment": "This code initializes a VideoPipe object with the given parameters, including file list, sequence length, and number of segments. It uses ops.VideoReader to read video data from the file list in the specified format. Due to the limitations of resize function, it transposes and reshapes the data before performing resizing operation on the 2-D image.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":142-159",
+ "content": " shard_id=0,\n dali_mean=0.,\n dali_std=1.0):\n super(VideoPipe, self).__init__(batch_size, num_threads, device_id)\n self.input = ops.VideoReader(device=\"gpu\",\n file_list=file_list,\n sequence_length=sequence_length,\n num_seg=num_seg,\n seg_length=seg_length,\n is_training=is_training,\n num_shards=num_shards,\n shard_id=shard_id,\n random_shuffle=is_training,\n initial_fill=initial_prefetch_size)\n # the sequece data read by ops.VideoReader is of shape [F, H, W, C]\n # Because the ops.Resize does not support sequence data,\n # it will be transposed into [H, W, F, C],\n # then reshaped to [H, W, FC], and then resized like a 2-D image."
+ },
+ {
+ "comment": "The code creates a DALI loader for image processing, with transpose, reshape, resize operations, and implements crop and mirror normalization. It also includes uniform distribution generators for position and mirror. The normalization will be implemented using PaddleOps due to the difficulty of dimension broadcasting in DALI.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":160-175",
+ "content": " self.transpose = ops.Transpose(device=\"gpu\", perm=[1, 2, 0, 3])\n self.reshape = ops.Reshape(device=\"gpu\",\n rel_shape=[1.0, 1.0, -1],\n layout='HWC')\n self.resize = ops.Resize(device=\"gpu\",\n resize_shorter=resize_shorter_scale)\n # crops and mirror are applied by ops.CropMirrorNormalize.\n # Normalization will be implemented in paddle due to the difficulty of dimension broadcast,\n # It is not sure whether dimension broadcast can be implemented correctly by dali, just take the Paddle Op instead.\n self.pos_rng_x = ops.Uniform(range=(0.0, 1.0))\n self.pos_rng_y = ops.Uniform(range=(0.0, 1.0))\n self.mirror_generator = ops.Uniform(range=(0.0, 1.0))\n self.cast_mirror = ops.Cast(dtype=types.DALIDataType.INT32)\n self.crop_mirror_norm = ops.CropMirrorNormalize(\n device=\"gpu\",\n crop=[crop_target_size, crop_target_size],"
+ },
+ {
+ "comment": "The code defines a DALI loader and its associated operations for image processing. It includes mean and std for normalization, reshaping, casting to int64, transpose, resize, normalization by dividing by 255, generating positional information, cropping with mirror flag, and finally reshaping the output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":176-201",
+ "content": " mean=dali_mean,\n std=dali_std)\n self.reshape_back = ops.Reshape(\n device=\"gpu\",\n shape=[num_seg, seg_length * 3, crop_target_size, crop_target_size],\n layout='FCHW')\n self.cast_label = ops.Cast(device=\"gpu\", dtype=types.DALIDataType.INT64)\n def define_graph(self):\n output, label = self.input(name=\"Reader\")\n output = self.transpose(output)\n output = self.reshape(output)\n output = self.resize(output)\n output = output / 255.\n pos_x = self.pos_rng_x()\n pos_y = self.pos_rng_y()\n mirror_flag = self.mirror_generator()\n mirror_flag = (mirror_flag > 0.5)\n mirror_flag = self.cast_mirror(mirror_flag)\n output = self.crop_mirror_norm(output,\n crop_pos_x=pos_x,\n crop_pos_y=pos_y,\n mirror=mirror_flag)\n output = self.reshape_back(output)\n label = self.cast_label(label)"
+ },
+ {
+ "comment": "The code defines a method that returns an output and label, and another method for determining the length of the loader.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dali_loader.py\":202-205",
+ "content": " return output, label\n def __len__(self):\n return self.epoch_size()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4a9f6fd5-205b-4d02-99f9-ba84b7a3b91f.json b/docs/doc/4a9f6fd5-205b-4d02-99f9-ba84b7a3b91f.json
new file mode 100644
index 000000000..23c6eb5a3
--- /dev/null
+++ b/docs/doc/4a9f6fd5-205b-4d02-99f9-ba84b7a3b91f.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code calculates Pearson and Spearman correlation coefficients (PLCC & SROCC) for a given output and label pair using numpy arrays and scipy's stats functions.",
+ "details": [
+ {
+ "comment": "This code defines the QuqlityMetric class for measuring video quality. It imports necessary libraries, registers it with METRIC, and initializes attributes including data_size, batch_size, and log_interval. The output and label lists are used to store data during processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/quality_metric.py\":0-34",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n\"\"\"\nimport numpy as np\nimport paddle\nfrom paddle.hapi.model import _all_gather\nfrom scipy import stats\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass QuqlityMetric(BaseMetric):\n \"\"\"CenterCropQualityMetric\"\"\"\n def __init__(self, data_size, batch_size, log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.output = []\n self.label = []"
+ },
+ {
+ "comment": "This code defines a class for calculating Pearson and Spearman correlation coefficients. The `update` method updates the metrics for each batch during training, while the `accumulate` method calculates the final Pearson (PLCC) and Spearman (SROCC) correlation coefficients after all iterations are finished.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/quality_metric.py\":35-61",
+ "content": " self.y_pred = np.zeros(data_size)\n self.y_test = np.zeros(data_size)\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n labels = data[1]\n predict_output = paddle.tolist(outputs)\n predict_label = paddle.tolist(labels)\n predict_output_len = len(predict_output)\n for i in range(predict_output_len):\n self.output.append(predict_output[i][0])\n self.label.append(predict_label[i][0])\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n test_output_np = np.array(self.output)\n test_label_np = np.array(self.label)\n PLCC = stats.pearsonr(test_output_np, test_label_np)[0]\n SROCC = stats.spearmanr(test_output_np, test_label_np)[0]"
+ },
+ {
+ "comment": "This code snippet calculates the Pearson and Spearman correlation coefficients (PLCC and SROCC) for a given output and label pair. It uses numpy arrays to convert the input into numeric data types, then calculates the correlation values using scipy's stats.pearsonr and stats.spearmanr functions respectively. Finally, it returns the calculated PLCC and SROCC values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/quality_metric.py\":63-71",
+ "content": " logger.info('[TEST] finished, PLCC= {}, SROCC= {} '.format(PLCC, SROCC))\n def accumulate_train(self, output, label):\n \"\"\"accumulate_train\"\"\"\n output_np = np.array(output)\n label_np = np.array(label)\n PLCC = stats.pearsonr(output_np, label_np)[0]\n SROCC = stats.spearmanr(output_np, label_np)[0]\n return PLCC, SROCC"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4b0429f3-5899-40ff-ba98-adb02f3d59f4.json b/docs/doc/4b0429f3-5899-40ff-ba98-adb02f3d59f4.json
new file mode 100644
index 000000000..9620ade1e
--- /dev/null
+++ b/docs/doc/4b0429f3-5899-40ff-ba98-adb02f3d59f4.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code includes functions for parsing, setting parameters, and performing status checks. The last function logs success or failure with command details to the specified run_log file using tee -a.",
+ "details": [
+ {
+ "comment": "This code defines several functions for parsing and setting parameters, as well as performing status checks. The functions extract keys and values from strings using specific delimiters, set parameters based on their presence and content, and check the exit code of previous commands.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/common_func.sh\":0-57",
+ "content": "#!/bin/bash\nfunction func_parser_key(){\n strs=$1\n IFS=\":\"\n array=(${strs})\n tmp=${array[0]}\n echo ${tmp}\n}\nfunction func_parser_value(){\n strs=$1\n IFS=\":\"\n array=(${strs})\n tmp=${array[1]}\n echo ${tmp}\n}\nfunction func_set_params(){\n key=$1\n value=$2\n if [ ${key}x = \"null\"x ];then\n echo \" \"\n elif [[ ${value} = \"null\" ]] || [[ ${value} = \" \" ]] || [ ${#value} -le 0 ];then\n echo \" \"\n else \n echo \"${key}=${value}\"\n fi\n}\nfunction func_parser_params(){\n strs=$1\n IFS=\":\"\n array=(${strs})\n key=${array[0]}\n tmp=${array[1]}\n IFS=\"|\"\n res=\"\"\n for _params in ${tmp[*]}; do\n IFS=\"=\"\n array=(${_params})\n mode=${array[0]}\n value=${array[1]}\n if [[ ${mode} = ${MODE} ]]; then\n IFS=\"|\"\n #echo $(func_set_params \"${mode}\" \"${value}\")\n echo $value\n break\n fi\n IFS=\"|\"\n done\n echo ${res}\n}\nfunction status_check(){\n last_status=$1 # the exit code\n run_command=$2\n run_log=$3"
+ },
+ {
+ "comment": "This function checks the last status and logs whether the run was successful or failed with specific command details. It appends the log to the specified run_log file using tee -a command.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/common_func.sh\":58-65",
+ "content": " model_name=$4\n log_path=$5\n if [ $last_status -eq 0 ]; then\n echo -e \"\\033[33m Run successfully with command - ${model_name} - ${run_command} - ${log_path} \\033[0m\" | tee -a ${run_log}\n else\n echo -e \"\\033[33m Run failed with command - ${model_name} - ${run_command} - ${log_path} \\033[0m\" | tee -a ${run_log}\n fi\n}"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4b99f1e3-4868-4798-b9f8-fe5ed1e1cfed.json b/docs/doc/4b99f1e3-4868-4798-b9f8-fe5ed1e1cfed.json
new file mode 100644
index 000000000..d41531013
--- /dev/null
+++ b/docs/doc/4b99f1e3-4868-4798-b9f8-fe5ed1e1cfed.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is licensing information and imports the RandomSampler class from a submodule, then defines the __all__ variable to include only the RandomSampler class.",
+ "details": [
+ {
+ "comment": "This code is licensing information and imports the RandomSampler class from a submodule, then defines the __all__ variable to include only the RandomSampler class.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/samplers/__init__.py\":0-16",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .random_sampler import RandomSampler\n__all__ = ['RandomSampler']"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4c899b69-445e-40cf-8ce1-c99c3ff3da42.json b/docs/doc/4c899b69-445e-40cf-8ce1-c99c3ff3da42.json
new file mode 100644
index 000000000..3ccef4c67
--- /dev/null
+++ b/docs/doc/4c899b69-445e-40cf-8ce1-c99c3ff3da42.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a PaddleVideo-based `InferModel` class for action detection using PPTSM with inference and prediction methods. It loads model, config file, specifies image paths, predicts on images, prints output shape, and time taken for prediction.",
+ "details": [
+ {
+ "comment": "This code defines a class `InferModel` that uses the PPTSM (Pose-aware Two-Stream Temporal Segmentation Model) for action detection. The model is initialized with a configuration file specifying the model and parameter files, as well as GPU memory and device ID settings. The configuration is optimized for efficient inference using feed fetch operations disabled and enabling memory optimization. The input tensor handle for the model is also retrieved.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/pptsm_infer.py\":0-37",
+ "content": "\"\"\"\nppTSM InferModel\n\"\"\"\nimport sys\nimport numpy as np\nimport time\nsys.path.append('../')\nfrom utils.preprocess import get_images\nfrom utils.config_utils import parse_config\nimport reader\nfrom paddle.inference import Config\nfrom paddle.inference import create_predictor\nclass InferModel(object):\n \"\"\"pptsm infer\"\"\"\n def __init__(self, cfg, name='PPTSM'): \n name = name.upper()\n self.name = name\n model_file = cfg[name]['model_file']\n params_file = cfg[name]['params_file']\n gpu_mem = cfg[name]['gpu_mem']\n device_id = cfg[name]['device_id']\n # model init\n config = Config(model_file, params_file)\n config.enable_use_gpu(gpu_mem, device_id)\n config.switch_ir_optim(True) # default true\n config.enable_memory_optim()\n # use zero copy\n config.switch_use_feed_fetch_ops(False)\n self.predictor = create_predictor(config)\n input_names = self.predictor.get_input_names()\n self.input_tensor = self.predictor.get_input_handle(input_names[0])"
+ },
+ {
+ "comment": "This code defines a class with methods for inferring and predicting actions from the PaddleVideo framework. It uses the PaddlePaddle library for inference and gets output names and handles to extract the results. The code also includes a main function that can be run if the file is executed directly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/pptsm_infer.py\":39-68",
+ "content": " output_names = self.predictor.get_output_names()\n print(\"output_names = \", output_names)\n #self.output_tensor = self.predictor.get_output_handle(output_names[1])\n self.output_tensor = self.predictor.get_output_handle(output_names[0])\n def infer(self, input):\n \"\"\"infer\"\"\"\n self.input_tensor.copy_from_cpu(input)\n self.predictor.run()\n output = self.output_tensor.copy_to_cpu()\n return output\n def predict(self, infer_config):\n \"\"\"predict\"\"\"\n infer_reader = reader.get_reader(self.name, 'infer', infer_config)\n feature_list = []\n for infer_iter, data in enumerate(infer_reader()):\n inputs = [items[:-1] for items in data]\n inputs = np.array(inputs)\n output = self.infer(inputs)\n #print(\"inputs\", inputs.shape)\n #print(\"outputs\", output.shape)\n feature_list.append(np.squeeze(output))\n feature_list = np.vstack(feature_list)\n return feature_list\nif __name__ == \"__main__\":"
+ },
+ {
+ "comment": "This code loads a model, config file, and specifies image paths. It then predicts using the loaded model on images in the specified path and prints the shape of the output as well as the time taken to perform prediction. The comment is suitable for code chunks that explain what each section does, like loading a model, reading input files, or performing computations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/models/pptsm_infer.py\":69-82",
+ "content": " cfg_file = '/home/work/inference/configs/configs.yaml' \n cfg = parse_config(cfg_file)\n model = InferModel(cfg)\n imgs_path = '/home/work/datasets/WorldCup2018/frames/6e577252c4004961ac7caa738a52c238/' \n imgs_list = get_images(imgs_path)\n t0 = time.time()\n cfg['PPTSM']['frame_list'] = imgs_list\n outputs = model.predict(cfg)\n # outputs = model.infer(np.random.rand(32, 8, 3, 224, 224).astype(np.float32))\n t1 = time.time()\n print(outputs.shape)\n print('cost time = {} min'.format((t1 - t0) / 60.0))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4cb0d2f5-3f59-492f-8273-48b5fec17c86.json b/docs/doc/4cb0d2f5-3f59-492f-8273-48b5fec17c86.json
new file mode 100644
index 000000000..779e33bee
--- /dev/null
+++ b/docs/doc/4cb0d2f5-3f59-492f-8273-48b5fec17c86.json
@@ -0,0 +1,25 @@
+{
+ "summary": "TSMHead, a classification task-oriented class extending TSNHead, initializes weights and registers in the HEADS registry. It is part of PaddleVideo's temporal segment network head, with parameters for weights and data format, forward function with average pooling and optional dropout, and possible tensor reshaping.",
+ "details": [
+ {
+ "comment": "This code defines the TSMHead class, which extends the TSNHead class. It is used for classification tasks with a specific number of classes and input feature channels. The class is registered in the HEADS registry and follows a certain weight initialization method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsm_head.py\":0-32",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn import Linear\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom .tsn_head import TSNHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass TSMHead(TSNHead):\n \"\"\" TSM Head\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature."
+ },
+ {
+ "comment": "The code defines a class with an __init__ method that initializes the TsmHead object. It takes arguments like num_classes, in_channels, drop_ratio, std, and data_format to set up the internal structure of the class. The Linear layer is also initialized with specific learning rates for weights and biases.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsm_head.py\":33-56",
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n drop_ratio(float): drop ratio. Default: 0.5.\n std(float): Std(Scale) value in normal initilizar. Default: 0.001.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n drop_ratio=0.5,\n std=0.001,\n data_format=\"NCHW\",\n **kwargs):\n super().__init__(num_classes,\n in_channels,\n drop_ratio=drop_ratio,\n std=std,\n data_format=data_format,\n **kwargs)\n self.fc = Linear(self.in_channels,\n self.num_classes,\n weight_attr=ParamAttr(learning_rate=5.0,\n regularizer=L2Decay(1e-4)),\n bias_attr=ParamAttr(learning_rate=10.0,"
+ },
+ {
+ "comment": "The code initializes a TSM head, sets the data format and standard deviation for weights, initializes FC layer parameters, defines the forward function to perform average pooling, dropout if applicable, and reshapes the tensor based on the data format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsm_head.py\":57-88",
+ "content": " regularizer=L2Decay(0.0)))\n assert (data_format in [\n 'NCHW', 'NHWC'\n ]), f\"data_format must be 'NCHW' or 'NHWC', but got {data_format}\"\n self.data_format = data_format\n self.stdv = std\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc, 'Normal', 'fc_0.w_0', 'fc_0.b_0', std=self.stdv)\n def forward(self, x, num_seg):\n \"\"\"Define how the tsm-head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # x.shape = [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x) # [N * num_segs, in_channels, 1, 1]\n if self.dropout is not None:\n x = self.dropout(x) # [N * num_seg, in_channels, 1, 1]\n if self.data_format == 'NCHW':\n x = paddle.reshape(x, x.shape[:2])"
+ },
+ {
+ "comment": "This code is part of a temporal segment network head in PaddleVideo. If the input is not 3-channel, it reshapes the feature map to have only the first third of channels. Then, it passes the reshaped feature through a fully connected layer and averages across segments to get a score for each class. The shape of the scores is then modified accordingly, and softmax could be applied (note: comment indicates that softmax might be removed).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/tsm_head.py\":89-98",
+ "content": " else:\n x = paddle.reshape(x, x.shape[::3])\n score = self.fc(x) # [N * num_seg, num_class]\n score = paddle.reshape(\n score, [-1, num_seg, score.shape[1]]) # [N, num_seg, num_class]\n score = paddle.mean(score, axis=1) # [N, num_class]\n score = paddle.reshape(score,\n shape=[-1, self.num_classes]) # [N, num_class]\n # score = F.softmax(score) #NOTE remove\n return score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4cff8309-3d7b-4715-9d30-609fc3431144.json b/docs/doc/4cff8309-3d7b-4715-9d30-609fc3431144.json
new file mode 100644
index 000000000..3389a7457
--- /dev/null
+++ b/docs/doc/4cff8309-3d7b-4715-9d30-609fc3431144.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The ASRFHead class is a model for action recognition using convolutional layers, and computes precision, recall, F1 score. It creates an ASRF head class for video processing with label retrieval, Levenshtein distance methods, edit scores, true positives, false positives, IoU measures, and selects the best scoring segment.",
+ "details": [
+ {
+ "comment": "The code defines a class for the ASRFHead, which is an instance of BaseHead and registered in HEADS registry. It imports necessary libraries, defines several models including SingleStageModel, and includes various utility functions from other modules. It also initializes weights with KaimingUniform_like_torch method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":0-31",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# https://github.com/yiskw713/asrf/libs/models/tcn.py\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nfrom paddle import ParamAttr\nfrom ..backbones.ms_tcn import SingleStageModel\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\nfrom ..framework.segmenters.utils import init_bias, KaimingUniform_like_torch\n@HEADS.register()\nclass ASRFHead(BaseHead):"
+ },
+ {
+ "comment": "The code above initializes an object of a class representing a feature extraction and classification model for action recognition. It takes several parameters such as the number of classes, features, stages in the action segmentation branch (ASB), stages in the boundary refinement branch (BRB), and layers per stage. The object is initialized by first calling the superclass constructor and then setting up the necessary components like the convolutional layers for class scores and boundary prediction, as well as multiple SingleStageModel instances for the action segmentation branch if needed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":33-62",
+ "content": " def __init__(self,\n num_classes,\n num_features,\n num_stages,\n num_layers,\n num_stages_asb=None,\n num_stages_brb=None):\n super().__init__(num_classes=num_classes, in_channels=num_features)\n if not isinstance(num_stages_asb, int):\n num_stages_asb = num_stages\n if not isinstance(num_stages_brb, int):\n num_stages_brb = num_stages\n self.num_layers = num_layers\n self.num_stages_asb = num_stages_asb\n self.num_stages_brb = num_stages_brb\n self.num_features = num_features\n # cls score\n self.overlap = 0.5\n self.conv_cls = nn.Conv1D(self.num_features, self.num_classes, 1)\n self.conv_boundary = nn.Conv1D(self.num_features, 1, 1)\n # action segmentation branch\n asb = [\n SingleStageModel(self.num_layers, self.num_features,\n self.num_classes, self.num_classes)\n for _ in range(self.num_stages_asb - 1)"
+ },
+ {
+ "comment": "This code defines a ASRF head model, initializes its weights and performs forward pass for classification and boundary regression tasks. It uses Conv1D layers and LayerList for flexibility. The weight initialization follows Kaiming uniform distribution and applies bias if present. The outputs of both tasks are stored separately in lists.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":63-97",
+ "content": " ]\n # boundary regression branch\n brb = [\n SingleStageModel(self.num_layers, self.num_features, 1, 1)\n for _ in range(self.num_stages_brb - 1)\n ]\n self.brb = nn.LayerList(brb)\n self.asb = nn.LayerList(asb)\n self.activation_asb = nn.Softmax(axis=1)\n self.activation_brb = nn.Sigmoid()\n def init_weights(self):\n \"\"\"\n initialize model layers' weight\n \"\"\"\n # init weight\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv1D):\n layer.weight.set_value(\n KaimingUniform_like_torch(layer.weight).astype('float32'))\n if layer.bias is not None:\n layer.bias.set_value(\n init_bias(layer.weight, layer.bias).astype('float32'))\n def forward(self, x):\n \"\"\"\n ASRF head\n \"\"\"\n out_cls = self.conv_cls(x)\n out_boundary = self.conv_boundary(x)\n outputs_cls = [out_cls]\n outputs_boundary = [out_boundary]"
+ },
+ {
+ "comment": "This code implements an ASRF head for a model, which takes in input and outputs classified classes and boundary scores. It also includes a get_F1_score function to calculate precision, recall, and F1 score for classification tasks. The F1 score is calculated based on the correctness of predicted class labels compared to ground truth labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":99-135",
+ "content": " for as_stage in self.asb:\n out_cls = as_stage(self.activation_asb(out_cls))\n outputs_cls.append(out_cls)\n for br_stage in self.brb:\n out_boundary = br_stage(self.activation_brb(out_boundary))\n outputs_boundary.append(out_boundary)\n return outputs_cls, outputs_boundary\n def get_F1_score(self, predicted, groundTruth):\n recog_content = list(predicted.numpy())\n gt_content = list(groundTruth[0].numpy())\n # cls score\n correct = 0\n total = 0\n edit = 0\n for i in range(len(gt_content)):\n total += 1\n if gt_content[i] == recog_content[i]:\n correct += 1\n edit_num = self.edit_score(recog_content, gt_content)\n edit += edit_num\n tp, fp, fn = self.f_score(recog_content, gt_content, self.overlap)\n # cls metric\n precision = tp / float(tp + fp)\n recall = tp / float(fp + fn)\n if precision + recall > 0.0:\n f1 = 2.0 * (precision * recall) / (precision + recall)"
+ },
+ {
+ "comment": "The code defines an ASRF head class that seems to be related to video processing and includes methods for retrieving label information and calculating the Levenshtein distance between two sequences. The get_labels_start_end_time method converts frame-wise labels into a list of labels, their respective start times, and end times. The levenstein method calculates the Levenshtein distance, which is used to compare two sequences of characters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":136-169",
+ "content": " else:\n f1 = 0.0\n f1 = np.nan_to_num(f1)\n return f1\n def get_labels_start_end_time(self, frame_wise_labels):\n labels = []\n starts = []\n ends = []\n last_label = frame_wise_labels[0]\n labels.append(frame_wise_labels[0])\n starts.append(0)\n for i in range(len(frame_wise_labels)):\n if frame_wise_labels[i] != last_label:\n labels.append(frame_wise_labels[i])\n starts.append(i)\n ends.append(i)\n last_label = frame_wise_labels[i]\n ends.append(i + 1)\n return labels, starts, ends\n def levenstein(self, p, y, norm=False):\n m_row = len(p)\n n_col = len(y)\n D = np.zeros([m_row + 1, n_col + 1], np.float)\n for i in range(m_row + 1):\n D[i, 0] = i\n for i in range(n_col + 1):\n D[0, i] = i\n for j in range(1, n_col + 1):\n for i in range(1, m_row + 1):\n if y[j - 1] == p[i - 1]:\n D[i, j] = D[i - 1, j - 1]"
+ },
+ {
+ "comment": "The code contains a function to calculate the edit score between two sequences. It uses the Levenshtein distance algorithm to compare recognized and ground truth labels, considering insertions, deletions, and substitutions. The f_score function calculates true positive (tp) and false positive (fp) values based on label overlaps, and normalizes the edit score if required.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":170-199",
+ "content": " else:\n D[i, j] = min(D[i - 1, j] + 1, D[i, j - 1] + 1,\n D[i - 1, j - 1] + 1)\n if norm:\n score = (1 - D[-1, -1] / max(m_row, n_col)) * 100\n else:\n score = D[-1, -1]\n return score\n def edit_score(self, recognized, ground_truth, norm=True):\n P, _, _ = self.get_labels_start_end_time(recognized)\n Y, _, _ = self.get_labels_start_end_time(ground_truth)\n return self.levenstein(P, Y, norm)\n def f_score(self, recognized, ground_truth, overlap):\n p_label, p_start, p_end = self.get_labels_start_end_time(recognized)\n y_label, y_start, y_end = self.get_labels_start_end_time(ground_truth)\n tp = 0\n fp = 0\n hits = np.zeros(len(y_label))\n for j in range(len(p_label)):\n intersection = np.minimum(p_end[j], y_end) - np.maximum(\n p_start[j], y_start)\n union = np.maximum(p_end[j], y_end) - np.minimum(\n p_start[j], y_start)"
+ },
+ {
+ "comment": "This code calculates true positives (tp), false positives (fp) and false negatives (fn). It measures IoU between predicted and actual labels, selects best scoring segment and tracks hits and misses. The method returns tp, fp, fn as float values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/asrf_head.py\":200-211",
+ "content": " IoU = (1.0 * intersection / union) * (\n [p_label[j] == y_label[x] for x in range(len(y_label))])\n # Get the best scoring segment\n idx = np.array(IoU).argmax()\n if IoU[idx] >= overlap and not hits[idx]:\n tp += 1\n hits[idx] = 1\n else:\n fp += 1\n fn = len(y_label) - sum(hits)\n return float(tp), float(fp), float(fn)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4d06c7ec-ce63-4739-aff3-681d841badf7.json b/docs/doc/4d06c7ec-ce63-4739-aff3-681d841badf7.json
new file mode 100644
index 000000000..c371c374b
--- /dev/null
+++ b/docs/doc/4d06c7ec-ce63-4739-aff3-681d841badf7.json
@@ -0,0 +1,60 @@
+{
+ "summary": "This code calculates BMN metric for object detection in computer vision frameworks, supports batch_size and world_size as 1, initializes class variables, processes video data, logs progress, saves results, performs soft NMS, calculates proposal lists, evaluates performance using the \"cal_metrics\" function.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and defines a function to compute the Intersection over Union (IoU) between a box and anchors. It appears to be related to object detection or proposal generation within a computer vision framework.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":0-31",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nimport multiprocessing as mp\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom .ActivityNet import ANETproposal\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\ndef iou_with_anchors(anchors_min, anchors_max, box_min, box_max):\n \"\"\"Compute jaccard score between a box and the anchors.\n \"\"\"\n len_anchors = anchors_max - anchors_min\n int_xmin = np.maximum(anchors_min, box_min)\n int_xmax = np.minimum(anchors_max, box_max)"
+ },
+ {
+ "comment": "inter_len calculates the intersection length between two bounding boxes. union_len computes the total length of both boxes and jaccard index is calculated by dividing inter_len with union_len. This function returns the Jaccard index.\nThe boundary_choose() function selects start and end boundaries based on a given score list. It identifies the highest score and creates three arrays - score_list, score_front, score_back for comparison. mask_peak is created by comparing these arrays, followed by generating a binary mask of True values.\nSoft_nms function sorts proposals generated by network based on scores in descending order. It takes alpha value (Gaussian decaying function), and two threshold values t1, t2.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":32-62",
+ "content": " inter_len = np.maximum(int_xmax - int_xmin, 0.)\n union_len = len_anchors - inter_len + box_max - box_min\n jaccard = np.divide(inter_len, union_len)\n return jaccard\ndef boundary_choose(score_list):\n \"\"\"Choose start and end boundary from score.\n \"\"\"\n max_score = max(score_list)\n mask_high = (score_list > max_score * 0.5)\n score_list = list(score_list)\n score_middle = np.array([0.0] + score_list + [0.0])\n score_front = np.array([0.0, 0.0] + score_list)\n score_back = np.array(score_list + [0.0, 0.0])\n mask_peak = ((score_middle > score_front) & (score_middle > score_back))\n mask_peak = mask_peak[1:-1]\n mask = (mask_high | mask_peak).astype('float32')\n return mask\ndef soft_nms(df, alpha, t1, t2):\n '''\n df: proposals generated by network;\n alpha: alpha value of Gaussian decaying function;\n t1, t2: threshold for soft nms.\n '''\n df = df.sort_values(by=\"score\", ascending=False)\n tstart = list(df.xmin.values[:])\n tend = list(df.xmax.values[:])\n tscore = list(df.score.values[:])"
+ },
+ {
+ "comment": "The code calculates BMN metric for object detection by iterating through a list of scores and appending the maximum score, along with its corresponding start and end positions, to new lists. It then creates a new DataFrame using these lists before returning it as the final output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":64-97",
+ "content": " rstart = []\n rend = []\n rscore = []\n while len(tscore) > 1 and len(rscore) < 101:\n max_index = tscore.index(max(tscore))\n tmp_iou_list = iou_with_anchors(np.array(tstart), np.array(tend),\n tstart[max_index], tend[max_index])\n for idx in range(0, len(tscore)):\n if idx != max_index:\n tmp_iou = tmp_iou_list[idx]\n tmp_width = tend[max_index] - tstart[max_index]\n if tmp_iou > t1 + (t2 - t1) * tmp_width:\n tscore[idx] = tscore[idx] * np.exp(\n -np.square(tmp_iou) / alpha)\n rstart.append(tstart[max_index])\n rend.append(tend[max_index])\n rscore.append(tscore[max_index])\n tstart.pop(max_index)\n tend.pop(max_index)\n tscore.pop(max_index)\n newDf = pd.DataFrame()\n newDf['score'] = rscore\n newDf['xmin'] = rstart\n newDf['xmax'] = rend\n return newDf\n@METRIC.register\nclass BMNMetric(BaseMetric):\n \"\"\"\n Metrics for BMN. Two Stages in this metric:"
+ },
+ {
+ "comment": "This code initializes an instance of BMNMetric class with various parameters such as data_size, batch_size, tscale, dscale, file_path, ground_truth_filename, subset, output_path, result_path, get_metrics, and log_interval. It also performs assertions to ensure batch_size is 1 and world_size is 1, as the code currently supports only these conditions. The class is a part of PaddleVideo library for video analysis tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":98-126",
+ "content": " (1) Get test results using trained model, results will be saved in BMNMetric.result_path;\n (2) Calculate metrics using results file from stage (1).\n \"\"\"\n def __init__(self,\n data_size,\n batch_size,\n tscale,\n dscale,\n file_path,\n ground_truth_filename,\n subset,\n output_path,\n result_path,\n get_metrics=True,\n log_interval=1):\n \"\"\"\n Init for BMN metrics.\n Params:\n get_metrics: whether to calculate AR@N and AUC metrics or not, default True.\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n assert self.batch_size == 1, \" Now we just support batch_size==1 test\"\n assert self.world_size == 1, \" Now we just support single-card test\"\n self.tscale = tscale\n self.dscale = dscale\n self.file_path = file_path\n self.ground_truth_filename = ground_truth_filename"
+ },
+ {
+ "comment": "The code initializes the class variables and checks if the output and result directories exist, creating them if not. It then calls a method to get the dataset dictionary and list based on the provided file path and subset. The update method takes batch ID, data, and outputs as inputs to update metrics during each iteration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":127-155",
+ "content": " self.subset = subset\n self.output_path = output_path\n self.result_path = result_path\n self.get_metrics = get_metrics\n if not os.path.isdir(self.output_path):\n os.makedirs(self.output_path)\n if not os.path.isdir(self.result_path):\n os.makedirs(self.result_path)\n self.video_dict, self.video_list = self.get_dataset_dict(\n self.file_path, self.subset)\n def get_dataset_dict(self, file_path, subset):\n annos = json.load(open(file_path))\n video_dict = {}\n for video_name in annos.keys():\n video_subset = annos[video_name][\"subset\"]\n if subset in video_subset:\n video_dict[video_name] = annos[video_name]\n video_list = list(video_dict.keys())\n video_list.sort()\n return video_dict, video_list\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n fid = data[4].numpy()\n pred_bm, pred_start, pred_end = outputs"
+ },
+ {
+ "comment": "Code snippet performs boundary detection and creates a score vector list for each detection. It uses the provided prediction, start and end values to calculate the xmin and xmax values within the defined time scale. Then it checks if the start and end mask conditions are met and adds the corresponding score value to the score vector list. This information is used for further analysis or evaluation of video frames.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":156-181",
+ "content": " pred_bm = pred_bm.numpy()\n pred_start = pred_start[0].numpy()\n pred_end = pred_end[0].numpy()\n snippet_xmins = [1.0 / self.tscale * i for i in range(self.tscale)]\n snippet_xmaxs = [\n 1.0 / self.tscale * i for i in range(1, self.tscale + 1)\n ]\n cols = [\"xmin\", \"xmax\", \"score\"]\n video_name = self.video_list[fid[0]]\n pred_bm = pred_bm[0, 0, :, :] * pred_bm[0, 1, :, :]\n start_mask = boundary_choose(pred_start)\n start_mask[0] = 1.\n end_mask = boundary_choose(pred_end)\n end_mask[-1] = 1.\n score_vector_list = []\n for idx in range(self.dscale):\n for jdx in range(self.tscale):\n start_index = jdx\n end_index = start_index + idx\n if end_index < self.tscale and start_mask[\n start_index] == 1 and end_mask[end_index] == 1:\n xmin = snippet_xmins[start_index]\n xmax = snippet_xmaxs[end_index]\n xmin_score = pred_start[start_index]"
+ },
+ {
+ "comment": "This code snippet performs post-processing on video data and calculates metrics. It first processes the video data, then accumulates the metrics for each batch during processing. The code uses numpy arrays to handle score vectors, Pandas DataFrame to store and manipulate data, and logging to provide progress updates. The results are saved in a CSV file at the specified output path.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":182-205",
+ "content": " xmax_score = pred_end[end_index]\n bm_score = pred_bm[idx, jdx]\n conf_score = xmin_score * xmax_score * bm_score\n score_vector_list.append([xmin, xmax, conf_score])\n score_vector_list = np.stack(score_vector_list)\n video_df = pd.DataFrame(score_vector_list, columns=cols)\n video_df.to_csv(os.path.join(self.output_path, \"%s.csv\" % video_name),\n index=False)\n if batch_id % self.log_interval == 0:\n logger.info(\"Processing................ batch {}\".format(batch_id))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n # check clip index of each video\n #Stage1\n self.bmn_post_processing(self.video_dict, self.subset, self.output_path,\n self.result_path)\n if self.get_metrics:\n logger.info(\"[TEST] calculate metrics...\")\n #Stage2\n uniform_average_nr_proposals_valid, uniform_average_recall_valid, uniform_recall_valid = self.cal_metrics("
+ },
+ {
+ "comment": "This code is initializing a bmn_post_processing function that will process multiple videos in parallel using multiple processes. It creates a result dictionary and divides the video list into equal parts to assign each part to a separate process. It also logs the average recall at different thresholds for different numbers of detections.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":206-228",
+ "content": " self.ground_truth_filename,\n os.path.join(self.result_path, \"bmn_results_validation.json\"),\n max_avg_nr_proposals=100,\n tiou_thresholds=np.linspace(0.5, 0.95, 10),\n subset='validation')\n logger.info(\"AR@1; AR@5; AR@10; AR@100\")\n logger.info(\"%.02f %.02f %.02f %.02f\" %\n (100 * np.mean(uniform_recall_valid[:, 0]),\n 100 * np.mean(uniform_recall_valid[:, 4]),\n 100 * np.mean(uniform_recall_valid[:, 9]),\n 100 * np.mean(uniform_recall_valid[:, -1])))\n def bmn_post_processing(self, video_dict, subset, output_path, result_path):\n video_list = list(video_dict.keys())\n global result_dict\n result_dict = mp.Manager().dict()\n pp_num = 12\n num_videos = len(video_list)\n num_videos_per_thread = int(num_videos / pp_num)\n processes = []\n for tid in range(pp_num - 1):\n tmp_video_list = video_list[tid * num_videos_per_thread:(tid + 1) *"
+ },
+ {
+ "comment": "The code creates multiple processes to handle video processing tasks in parallel, using multiprocessing. It then joins all the results together into a single output dictionary before writing it to a JSON file. This approach allows for efficient and concurrent processing of large numbers of videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":229-255",
+ "content": " num_videos_per_thread]\n p = mp.Process(target=self.video_process,\n args=(tmp_video_list, video_dict, output_path,\n result_dict))\n p.start()\n processes.append(p)\n tmp_video_list = video_list[(pp_num - 1) * num_videos_per_thread:]\n p = mp.Process(target=self.video_process,\n args=(tmp_video_list, video_dict, output_path,\n result_dict))\n p.start()\n processes.append(p)\n for p in processes:\n p.join()\n result_dict = dict(result_dict)\n output_dict = {\n \"version\": \"VERSION 1.3\",\n \"results\": result_dict,\n \"external_data\": {}\n }\n outfile = open(\n os.path.join(result_path, \"bmn_results_%s.json\" % subset), \"w\")\n # json.dump(output_dict, outfile)\n # in case of file name in chinese\n json.dump(output_dict, outfile, ensure_ascii=False)"
+ },
+ {
+ "comment": "This function takes a list of video names, corresponding metadata dictionaries, output path, and result dictionary. It processes each video by reading its CSV file, performs soft NMS if the dataframe has more than one row, calculates proposal list for each video, and appends them to the result dictionary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":256-281",
+ "content": " outfile.close()\n def video_process(self,\n video_list,\n video_dict,\n output_path,\n result_dict,\n snms_alpha=0.4,\n snms_t1=0.55,\n snms_t2=0.9):\n for video_name in video_list:\n logger.info(\"Processing video........\" + video_name)\n df = pd.read_csv(os.path.join(output_path, video_name + \".csv\"))\n if len(df) > 1:\n df = soft_nms(df, snms_alpha, snms_t1, snms_t2)\n video_duration = video_dict[video_name][\"duration_second\"]\n proposal_list = []\n for idx in range(min(100, len(df))):\n tmp_prop={\"score\":df.score.values[idx], \\\n \"segment\":[max(0,df.xmin.values[idx])*video_duration, \\\n min(1,df.xmax.values[idx])*video_duration]}\n proposal_list.append(tmp_prop)\n video_name = video_name[2:] if video_name[:2] == 'v_' else video_name"
+ },
+ {
+ "comment": "The code defines a function \"cal_metrics\" that takes in ground truth and proposal filenames, calculates the average recall, average proposals per video, and overall recall using ANETproposal class. This function is used to evaluate performance based on given thresholds and subsets of data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/bmn_metric.py\":282-303",
+ "content": " result_dict[video_name] = proposal_list\n def cal_metrics(self,\n ground_truth_filename,\n proposal_filename,\n max_avg_nr_proposals=100,\n tiou_thresholds=np.linspace(0.5, 0.95, 10),\n subset='validation'):\n anet_proposal = ANETproposal(ground_truth_filename,\n proposal_filename,\n tiou_thresholds=tiou_thresholds,\n max_avg_nr_proposals=max_avg_nr_proposals,\n subset=subset,\n verbose=True,\n check_status=False)\n anet_proposal.evaluate()\n recall = anet_proposal.recall\n average_recall = anet_proposal.avg_recall\n average_nr_proposals = anet_proposal.proposals_per_video\n return (average_nr_proposals, average_recall, recall)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4d09faa8-ce8a-428b-983c-98e28504bd5a.json b/docs/doc/4d09faa8-ce8a-428b-983c-98e28504bd5a.json
new file mode 100644
index 000000000..02e1df172
--- /dev/null
+++ b/docs/doc/4d09faa8-ce8a-428b-983c-98e28504bd5a.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The PaintBoard class, derived from QWidget, handles data initialization and view size, provides methods for clearing board, changing pen attributes, painting events, and retrieving content. The code implements a mouse event handler for drawing on the board in PaintBoard mode or eraser mode based on user selection.",
+ "details": [
+ {
+ "comment": "The code defines a class `PaintBoard` which inherits from `QWidget`. It initializes data such as the size of the board, an empty QPixmap, a boolean for EraserMode, and variables to store the position and pen attributes. Then it sets the view's fixed size based on the initialized data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/widget/PaintBoard.py\":0-39",
+ "content": "from PyQt5.QtWidgets import QWidget\nfrom PyQt5.Qt import QPixmap, QPainter, QPoint, QPaintEvent, QMouseEvent, QPen, \\\n QColor, QSize\nfrom PyQt5.QtCore import Qt\nclass PaintBoard(QWidget):\n def __init__(self, parent=None):\n '''\n Constructor\n '''\n super().__init__(parent)\n self.__init_data() # \u5148\u521d\u59cb\u5316\u6570\u636e\uff0c\u518d\u521d\u59cb\u5316\u754c\u9762\n self.__init_view()\n def __init_data(self):\n self.__size = QSize(810, 458)\n # \u65b0\u5efaQPixmap\u4f5c\u4e3a\u753b\u677f\uff0c\u5c3a\u5bf8\u4e3a__size\n self.__board = QPixmap(self.__size)\n self.__board.fill(Qt.transparent) # \u7528\u900f\u660e\u586b\u5145\u753b\u677f\n self.__IsEmpty = True # \u9ed8\u8ba4\u4e3a\u7a7a\u753b\u677f\n self.EraserMode = False # \u9ed8\u8ba4\u4e3a\u7981\u7528\u6a61\u76ae\u64e6\u6a21\u5f0f\n self.__lastPos = QPoint(0, 0) # \u4e0a\u4e00\u6b21\u9f20\u6807\u4f4d\u7f6e\n self.__currentPos = QPoint(0, 0) # \u5f53\u524d\u7684\u9f20\u6807\u4f4d\u7f6e\n self.__painter = QPainter() # \u65b0\u5efa\u7ed8\u56fe\u5de5\u5177\n self.__thickness = 15 # \u9ed8\u8ba4\u753b\u7b14\u7c97\u7ec6\u4e3a10px\n self.__penColor = QColor(\"black\") # \u8bbe\u7f6e\u9ed8\u8ba4\u753b\u7b14\u989c\u8272\u4e3a\u9ed1\u8272\n self.__colorList = QColor.colorNames() # \u83b7\u53d6\u989c\u8272\u5217\u8868\n def __init_view(self):\n # \u8bbe\u7f6e\u754c\u9762\u7684\u5c3a\u5bf8\u4e3a__size\n self.setFixedSize(self.__size)"
+ },
+ {
+ "comment": "This code defines a class with methods to clear the paint board, change pen color and thickness, check if the board is empty, retrieve content as QImage, and handle painting events. The paintEvent method utilizes QPainter to draw the pixmap on the board in the correct location.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/widget/PaintBoard.py\":41-77",
+ "content": " def clear(self):\n # \u6e05\u7a7a\u753b\u677f\n # self.__board.fill(Qt.white)\n self.__board = QPixmap(self.__size)\n self.__board.fill(Qt.transparent) # \u7528\u900f\u660e\u586b\u5145\u753b\u677f\n self.update()\n self.__IsEmpty = True\n def change_pen_color(self, color=\"black\"):\n # \u6539\u53d8\u753b\u7b14\u989c\u8272\n # rgbaColor = QColor(255, 255, 0, 100)\n self.__penColor = QColor(color)\n def change_pen_thickness(self, thickness=10):\n # \u6539\u53d8\u753b\u7b14\u7c97\u7ec6\n self.__thickness = thickness\n def is_empty(self):\n # \u8fd4\u56de\u753b\u677f\u662f\u5426\u4e3a\u7a7a\n return self.__IsEmpty\n def get_content_as_q_image(self):\n # \u83b7\u53d6\u753b\u677f\u5185\u5bb9\uff08\u8fd4\u56deQImage\uff09\n image = self.__board.toImage()\n return image\n def paintEvent(self, paint_event):\n # \u7ed8\u56fe\u4e8b\u4ef6\n # \u7ed8\u56fe\u65f6\u5fc5\u987b\u4f7f\u7528QPainter\u7684\u5b9e\u4f8b\uff0c\u6b64\u5904\u4e3a__painter\n # \u7ed8\u56fe\u5728begin()\u51fd\u6570\u4e0eend()\u51fd\u6570\u95f4\u8fdb\u884c\n # begin(param)\u7684\u53c2\u6570\u8981\u6307\u5b9a\u7ed8\u56fe\u8bbe\u5907\uff0c\u5373\u628a\u56fe\u753b\u5728\u54ea\u91cc\n # drawPixmap\u7528\u4e8e\u7ed8\u5236QPixmap\u7c7b\u578b\u7684\u5bf9\u8c61\n self.__painter.begin(self)\n # 0,0\u4e3a\u7ed8\u56fe\u7684\u5de6\u4e0a\u89d2\u8d77\u70b9\u7684\u5750\u6807\uff0c__board\u5373\u8981\u7ed8\u5236\u7684\u56fe\n self.__painter.drawPixmap(0, 0, self.__board)\n self.__painter.end()"
+ },
+ {
+ "comment": "This code implements a mouse event handler for drawing on a PaintBoard. When the mouse is pressed, the current position is saved as the previous position. As the mouse moves, it draws lines between the last and current positions based on whether eraser mode is enabled or not. Upon mouse release, the board is marked as not empty. The drawing is updated to reflect the changes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/widget/PaintBoard.py\":79-105",
+ "content": " def mousePressEvent(self, mouse_event):\n # \u9f20\u6807\u6309\u4e0b\u65f6\uff0c\u83b7\u53d6\u9f20\u6807\u7684\u5f53\u524d\u4f4d\u7f6e\u4fdd\u5b58\u4e3a\u4e0a\u4e00\u6b21\u4f4d\u7f6e\n self.__currentPos = mouse_event.pos()\n self.__lastPos = self.__currentPos\n def mouseMoveEvent(self, mouse_event):\n # \u9f20\u6807\u79fb\u52a8\u65f6\uff0c\u66f4\u65b0\u5f53\u524d\u4f4d\u7f6e\uff0c\u5e76\u5728\u4e0a\u4e00\u4e2a\u4f4d\u7f6e\u548c\u5f53\u524d\u4f4d\u7f6e\u95f4\u753b\u7ebf\n self.__currentPos = mouse_event.pos()\n self.__painter.begin(self.__board)\n if self.EraserMode == False:\n # \u975e\u6a61\u76ae\u64e6\u6a21\u5f0f\n self.__painter.setPen(QPen(self.__penColor, self.__thickness)) # \u8bbe\u7f6e\u753b\u7b14\u989c\u8272\uff0c\u7c97\u7ec6\n else:\n # \u6a61\u76ae\u64e6\u6a21\u5f0f\u4e0b\u753b\u7b14\u4e3a\u7eaf\u767d\u8272\uff0c\u7c97\u7ec6\u4e3a10\n self.__painter.setPen(QPen(Qt.transparent, 10))\n # \u753b\u7ebf\n # print(self.__lastPos + self.__currentPos)\n self.__painter.drawLine(self.__lastPos, self.__currentPos)\n self.__painter.end()\n self.__lastPos = self.__currentPos\n self.update() # \u66f4\u65b0\u663e\u793a\n def mouseReleaseEvent(self, mouseEvent):\n self.__IsEmpty = False # \u753b\u677f\u4e0d\u518d\u4e3a\u7a7a"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4e27e6ff-cf4b-435a-ac68-53362cde9b5d.json b/docs/doc/4e27e6ff-cf4b-435a-ac68-53362cde9b5d.json
new file mode 100644
index 000000000..40f07e82f
--- /dev/null
+++ b/docs/doc/4e27e6ff-cf4b-435a-ac68-53362cde9b5d.json
@@ -0,0 +1,45 @@
+{
+ "summary": "The Python script prepares MSRVTTDataset by importing libraries, creating a class, tokenizing captions, retrieving features, and preparing sequences for processing. It processes image data, performs array operations, pads, resizes, calculates features, and converts to float32 for training/testing.",
+ "details": [
+ {
+ "comment": "The code is a Python script that imports various libraries and packages, checks for the availability of 'lmdb' library, and tries to import 'BertTokenizer' from 'paddlenlp'. It also includes license information and copyright notice.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":0-30",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\ntry:\n import lmdb\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [lmdb] package and it's dependencies is required for ActBERT.\"\n )\nimport pickle\ntry:\n from paddlenlp.transformers import BertTokenizer\nexcept ImportError as e:\n print(\n f\"Warning! {e}, [paddlenlp] package and it's dependencies is required for ActBERT.\"\n )"
+ },
+ {
+ "comment": "The code defines the `MSRVTTDataset` class for text-video clip retrieval from MSR-VTT dataset, registering it in the registry. It takes parameters such as file path, pipeline, and maximum sequence length for initializing the dataset, and provides attributes like bert model, padding index, and other dimensions for processing the dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":31-66",
+ "content": "from ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass MSRVTTDataset(BaseDataset):\n \"\"\"MSR-VTT dataset for text-video clip retrieval.\n \"\"\"\n def __init__(\n self,\n file_path,\n pipeline,\n features_path,\n bert_model=\"bert-base-uncased\",\n padding_index=0,\n max_seq_length=36,\n max_region_num=36,\n max_action_num=5,\n vision_feature_dim=2048,\n action_feature_dim=2048,\n spatials_dim=5,\n data_prefix=None,\n test_mode=False,\n ):\n self.features_path = features_path\n self.bert_model = bert_model\n self.padding_index = padding_index\n self.max_seq_length = max_seq_length\n self.max_region_num = max_region_num\n self._max_action_num = max_action_num\n self.vision_feature_dim = vision_feature_dim\n self.action_feature_dim = action_feature_dim\n self.spatials_dim = spatials_dim"
+ },
+ {
+ "comment": "The code snippet initializes a BertTokenizer object, loads file containing video information, tokenizes each entry's caption using the initialized tokenizer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":67-92",
+ "content": " self._tokenizer = BertTokenizer.from_pretrained(bert_model,\n do_lower_case=True)\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n self.tokenize()\n self.gen_feature()\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n with open(self.file_path) as fin:\n self.image_entries = []\n self.caption_entries = []\n for line in fin.readlines():\n line = line.strip()\n vid_id = line.split(',')[0]\n self.image_entries.append(vid_id)\n self.caption_entries.append({\n \"caption\": line.split(',')[1],\n \"vid_id\": vid_id\n })\n self.env = lmdb.open(self.features_path)\n def tokenize(self):\n for entry in self.caption_entries:\n tokens = []\n tokens.append(\"[CLS]\")\n for token in self._tokenizer.tokenize(entry[\"caption\"]):"
+ },
+ {
+ "comment": "This code is part of a class that processes video data. It appends tokens to an entry, converts tokens to ids, creates segment and input masks, and pads the sequence if necessary. The \"get_image_feature\" function retrieves image features from a database for a given video ID.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":93-119",
+ "content": " tokens.append(token)\n tokens.append(\"[SEP]\")\n tokens = self._tokenizer.convert_tokens_to_ids(tokens)\n segment_ids = [0] * len(tokens)\n input_mask = [1] * len(tokens)\n if len(tokens) < self.max_seq_length:\n padding = [self.padding_index\n ] * (self.max_seq_length - len(tokens))\n tokens = tokens + padding\n input_mask += padding\n segment_ids += padding\n entry[\"token\"] = np.array(tokens).astype('int64')\n entry[\"input_mask\"] = np.array(input_mask)\n entry[\"segment_ids\"] = np.array(segment_ids).astype('int64')\n def get_image_feature(self, video_id):\n video_id = str(video_id).encode()\n with self.env.begin(write=False) as txn:\n item = pickle.loads(txn.get(video_id))\n video_id = item[\"video_id\"]\n image_h = int(item[\"image_h\"])\n image_w = int(item[\"image_w\"])\n features = item[\"features\"].reshape(-1, self.vision_feature_dim)"
+ },
+ {
+ "comment": "This code is resizing and calculating the image location for each box in a dataset. It also concatenates the average feature to the start of the features array, and handles reshaping action_features. The code uses numpy functions extensively for array operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":120-141",
+ "content": " boxes = item[\"boxes\"].reshape(-1, 4)\n num_boxes = features.shape[0]\n g_feat = np.sum(features, axis=0) / num_boxes\n num_boxes = num_boxes + 1\n features = np.concatenate(\n [np.expand_dims(g_feat, axis=0), features], axis=0)\n action_features = item[\"action_features\"].reshape(\n -1, self.action_feature_dim)\n image_location = np.zeros((boxes.shape[0], self.spatials_dim),\n dtype=np.float32)\n image_location[:, :4] = boxes\n image_location[:,\n 4] = ((image_location[:, 3] - image_location[:, 1]) *\n (image_location[:, 2] - image_location[:, 0]) /\n (float(image_w) * float(image_h)))\n image_location[:, 0] = image_location[:, 0] / float(image_w)\n image_location[:, 1] = image_location[:, 1] / float(image_h)\n image_location[:, 2] = image_location[:, 2] / float(image_w)"
+ },
+ {
+ "comment": "The code defines a function that returns features, number of boxes, image location, and action features after processing an input image. It also initializes arrays for all instances of features, action features, spatial locations, and masks. The code then iterates over each image ID and calls another function to get the respective features, num_boxes, boxes, and action_features.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":142-162",
+ "content": " image_location[:, 3] = image_location[:, 3] / float(image_h)\n g_location = np.array([0, 0, 1, 1, 1])\n image_location = np.concatenate(\n [np.expand_dims(g_location, axis=0), image_location], axis=0)\n return features, num_boxes, image_location, action_features\n def gen_feature(self):\n num_inst = len(self.image_entries) #1000\n self.features_all = np.zeros(\n (num_inst, self.max_region_num, self.vision_feature_dim))\n self.action_features_all = np.zeros(\n (num_inst, self._max_action_num, self.action_feature_dim))\n self.spatials_all = np.zeros(\n (num_inst, self.max_region_num, self.spatials_dim))\n self.image_mask_all = np.zeros((num_inst, self.max_region_num))\n self.action_mask_all = np.zeros((num_inst, self._max_action_num))\n for i, image_id in enumerate(self.image_entries):\n features, num_boxes, boxes, action_features = self.get_image_feature(\n image_id)"
+ },
+ {
+ "comment": "The code handles the padding of features, boxes and masks for a dataset. It ensures that all sequences have the same length by padding them with zeros if necessary. The mixed features (maximum region number), boxes, and action features are assigned to respective lists. These lists will be used later in the program. The code also converts the features list to float32 data type.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":164-186",
+ "content": " mix_num_boxes = min(int(num_boxes), self.max_region_num)\n mix_boxes_pad = np.zeros((self.max_region_num, self.spatials_dim))\n mix_features_pad = np.zeros(\n (self.max_region_num, self.vision_feature_dim))\n image_mask = [1] * (int(mix_num_boxes))\n while len(image_mask) < self.max_region_num:\n image_mask.append(0)\n action_mask = [1] * (self._max_action_num)\n while len(action_mask) < self._max_action_num:\n action_mask.append(0)\n mix_boxes_pad[:mix_num_boxes] = boxes[:mix_num_boxes]\n mix_features_pad[:mix_num_boxes] = features[:mix_num_boxes]\n self.features_all[i] = mix_features_pad\n x = action_features.shape[0]\n self.action_features_all[i][:x] = action_features[:]\n self.image_mask_all[i] = np.array(image_mask)\n self.action_mask_all[i] = np.array(action_mask)\n self.spatials_all[i] = mix_boxes_pad\n self.features_all = self.features_all.astype(\"float32\")"
+ },
+ {
+ "comment": "This code initializes data types and provides methods for preparing training and testing data. The `prepare_train` method is left empty, while `prepare_test` takes an index, retrieves the corresponding entry, creates a target array, and returns various data arrays to be used in testing. The length of the dataset is determined by the number of caption entries.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/msrvtt.py\":187-219",
+ "content": " self.action_features_all = self.action_features_all.astype(\"float32\")\n self.image_mask_all = self.image_mask_all.astype(\"int64\")\n self.action_mask_all = self.action_mask_all.astype(\"int64\")\n self.spatials_all = self.spatials_all.astype(\"float32\")\n def prepare_train(self, idx):\n pass\n def prepare_test(self, idx):\n entry = self.caption_entries[idx]\n caption = entry[\"token\"]\n input_mask = entry[\"input_mask\"]\n segment_ids = entry[\"segment_ids\"]\n target_all = np.zeros(1000)\n for i, image_id in enumerate(self.image_entries):\n if image_id == entry[\"vid_id\"]:\n target_all[i] = 1\n return (\n caption,\n self.action_features_all,\n self.features_all,\n self.spatials_all,\n segment_ids,\n input_mask,\n self.image_mask_all,\n self.action_mask_all,\n target_all,\n )\n def __len__(self):\n return len(self.caption_entries)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4e2977e4-6b4e-4937-b357-76a12da3b124.json b/docs/doc/4e2977e4-6b4e-4937-b357-76a12da3b124.json
new file mode 100644
index 000000000..16cb93e0b
--- /dev/null
+++ b/docs/doc/4e2977e4-6b4e-4937-b357-76a12da3b124.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This function processes skeleton data, extracts body information, updates a dictionary with the data and returns the skeleton name, body data, and frame count. It handles missing frames and calculates motion using NTU RGB-D dataset data. Additionally, it combines and processes raw skeleton data from multiple files, updating progress, filters out missing frames, logs events, and saves the filtered data into pickle files.",
+ "details": [
+ {
+ "comment": "This function gets raw bodies data from a skeleton sequence by loading the file and checking its existence. It returns a dictionary with three key-value pairs: name (skeleton filename), data (raw data of each body), and num_frames (number of valid frames).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/ntu-rgb-d/get_raw_skes_data.py\":0-27",
+ "content": "# ref: https://github.com/Uason-Chen/CTR-GCN/blob/main/data/ntu/get_raw_skes_data.py\nimport os.path as osp\nimport os\nimport numpy as np\nimport pickle\nimport logging\ndef get_raw_bodies_data(skes_path, ske_name, frames_drop_skes,\n frames_drop_logger):\n \"\"\"\n Get raw bodies data from a skeleton sequence.\n Each body's data is a dict that contains the following keys:\n - joints: raw 3D joints positions. Shape: (num_frames x 25, 3)\n - colors: raw 2D color locations. Shape: (num_frames, 25, 2)\n - interval: a list which stores the frame indices of this body.\n - motion: motion amount (only for the sequence with 2 or more bodyIDs).\n Return:\n a dict for a skeleton sequence with 3 key-value pairs:\n - name: the skeleton filename.\n - data: a dict which stores raw data of each body.\n - num_frames: the number of valid frames.\n \"\"\"\n ske_file = osp.join(skes_path, ske_name + '.skeleton')\n assert osp.exists(ske_file), 'Error: Skeleton file %s not found' % ske_file"
+ },
+ {
+ "comment": "Reading and processing .skeleton file data into a list, storing number of frames, ignoring frames with no bodies, extracting body IDs, and counting the number of joints for each body.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/ntu-rgb-d/get_raw_skes_data.py\":28-57",
+ "content": " # Read all data from .skeleton file into a list (in string format)\n print('Reading data from %s' % ske_file[-29:])\n with open(ske_file, 'r') as fr:\n str_data = fr.readlines()\n num_frames = int(str_data[0].strip('\\r\\n'))\n frames_drop = []\n bodies_data = dict()\n valid_frames = -1 # 0-based index\n current_line = 1\n for f in range(num_frames):\n num_bodies = int(str_data[current_line].strip('\\r\\n'))\n current_line += 1\n if num_bodies == 0: # no data in this frame, drop it\n frames_drop.append(f) # 0-based index\n continue\n valid_frames += 1\n joints = np.zeros((num_bodies, 25, 3), dtype=np.float32)\n colors = np.zeros((num_bodies, 25, 2), dtype=np.float32)\n for b in range(num_bodies):\n bodyID = str_data[current_line].strip('\\r\\n').split()[0]\n current_line += 1\n num_joints = int(str_data[current_line].strip('\\r\\n')) # 25 joints\n current_line += 1\n for j in range(num_joints):"
+ },
+ {
+ "comment": "This code reads data from a file, extracts joint and color information for each body, and updates or adds body data to a dictionary based on the body ID. The joint and color arrays are created using numpy functions, and the joints array is stacked along the frame order if the body's data already exists in the dictionary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/ntu-rgb-d/get_raw_skes_data.py\":58-75",
+ "content": " temp_str = str_data[current_line].strip('\\r\\n').split()\n joints[b, j, :] = np.array(temp_str[:3], dtype=np.float32)\n colors[b, j, :] = np.array(temp_str[5:7], dtype=np.float32)\n current_line += 1\n if bodyID not in bodies_data: # Add a new body's data\n body_data = dict()\n body_data['joints'] = joints[b] # ndarray: (25, 3)\n body_data['colors'] = colors[b,\n np.newaxis] # ndarray: (1, 25, 2)\n body_data['interval'] = [valid_frames\n ] # the index of the first frame\n else: # Update an already existed body's data\n body_data = bodies_data[bodyID]\n # Stack each body's data of each frame along the frame order\n body_data['joints'] = np.vstack(\n (body_data['joints'], joints[b]))\n body_data['colors'] = np.vstack("
+ },
+ {
+ "comment": "This code retrieves raw data for a specific subject's skeleton (ske_name) from the NTU RGB-D dataset. It handles missing frames, calculates motion based on body data with multiple bodyIDs and returns the skeleton name, body data and updated frame count.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/ntu-rgb-d/get_raw_skes_data.py\":76-99",
+ "content": " (body_data['colors'], colors[b, np.newaxis]))\n pre_frame_idx = body_data['interval'][-1]\n body_data['interval'].append(pre_frame_idx +\n 1) # add a new frame index\n bodies_data[bodyID] = body_data # Update bodies_data\n num_frames_drop = len(frames_drop)\n assert num_frames_drop < num_frames, \\\n 'Error: All frames data (%d) of %s is missing or lost' % (num_frames, ske_name)\n if num_frames_drop > 0:\n frames_drop_skes[ske_name] = np.array(frames_drop, dtype=np.int)\n frames_drop_logger.info('{}: {} frames missed: {}\\n'.format(\n ske_name, num_frames_drop, frames_drop))\n # Calculate motion (only for the sequence with 2 or more bodyIDs)\n if len(bodies_data) > 1:\n for body_data in bodies_data.values():\n body_data['motion'] = np.sum(np.var(body_data['joints'], axis=0))\n return {\n 'name': ske_name,\n 'data': bodies_data,\n 'num_frames': num_frames - num_frames_drop"
+ },
+ {
+ "comment": "This function retrieves raw skeleton data from multiple files, processes it, and saves the combined data in a file. It keeps track of the number of frames for each file and prints progress updates every 1000 files processed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/ntu-rgb-d/get_raw_skes_data.py\":100-129",
+ "content": " }\ndef get_raw_skes_data():\n skes_name = np.loadtxt(skes_name_file, dtype=str)\n num_files = skes_name.size\n print('Found %d available skeleton files.' % num_files)\n raw_skes_data = []\n frames_cnt = np.zeros(num_files, dtype=np.int)\n for (idx, ske_name) in enumerate(skes_name):\n bodies_data = get_raw_bodies_data(skes_path, ske_name, frames_drop_skes,\n frames_drop_logger)\n raw_skes_data.append(bodies_data)\n frames_cnt[idx] = bodies_data['num_frames']\n if (idx + 1) % 1000 == 0:\n print('Processed: %.2f%% (%d / %d)' % \\\n (100.0 * (idx + 1) / num_files, idx + 1, num_files))\n with open(save_data_pkl, 'wb') as fw:\n pickle.dump(raw_skes_data, fw, pickle.HIGHEST_PROTOCOL)\n np.savetxt(osp.join(save_path, 'raw_data', 'frames_cnt.txt'),\n frames_cnt,\n fmt='%d')\n print('Saved raw bodies data into %s' % save_data_pkl)\n print('Total frames: %d' % np.sum(frames_cnt))"
+ },
+ {
+ "comment": "This code reads data from the NTU-RGB+D dataset, filters out frames with missing skeleton data, and saves it into two pickle files. The data is read from a specific path, and if the raw_data directory does not exist, it creates one. A logger for frames drop events is also set up and logs to a file. Finally, the code dumps the filtered frames data into another pickle file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/data/ntu-rgb-d/get_raw_skes_data.py\":131-156",
+ "content": " with open(frames_drop_pkl, 'wb') as fw:\n pickle.dump(frames_drop_skes, fw, pickle.HIGHEST_PROTOCOL)\nif __name__ == '__main__':\n save_path = './'\n skes_path = '../ntu-rgb-d/nturgb+d_skeletons/'\n stat_path = osp.join(save_path, 'statistics')\n if not osp.exists('./raw_data'):\n os.makedirs('./raw_data')\n skes_name_file = osp.join(stat_path, 'skes_available_name.txt')\n save_data_pkl = osp.join(save_path, 'raw_data', 'raw_skes_data.pkl')\n frames_drop_pkl = osp.join(save_path, 'raw_data', 'frames_drop_skes.pkl')\n frames_drop_logger = logging.getLogger('frames_drop')\n frames_drop_logger.setLevel(logging.INFO)\n frames_drop_logger.addHandler(\n logging.FileHandler(osp.join(save_path, 'raw_data', 'frames_drop.log')))\n frames_drop_skes = dict()\n get_raw_skes_data()\n with open(frames_drop_pkl, 'wb') as fw:\n pickle.dump(frames_drop_skes, fw, pickle.HIGHEST_PROTOCOL)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/4f50e070-a940-4db2-83a2-23168083178f.json b/docs/doc/4f50e070-a940-4db2-83a2-23168083178f.json
new file mode 100644
index 000000000..d648769d3
--- /dev/null
+++ b/docs/doc/4f50e070-a940-4db2-83a2-23168083178f.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a 3D ROI extractor class and head, performing feature extraction with optional temporal pooling. The forward method executes feature extraction based on input features, RoIs, and number of RoIs, and returns the final output after applying ROI layer and stacking features along axis 2.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and registers a new ROI (Region of Interest) extractor named \"SingleRoIExtractor3D\". This class inherits from nn.Layer and is designed to extract RoI features from a single level feature map with specific options such as roi_layer_type, featmap_stride, output_size, and sampling_ratio.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/single_straight3d.py\":0-27",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport numpy as np\nfrom ..registry import ROI_EXTRACTORS\nfrom .roi_extractor import RoIAlign\n@ROI_EXTRACTORS.register()\nclass SingleRoIExtractor3D(nn.Layer):\n \"\"\"Extract RoI features from a single level feature map. \"\"\"\n def __init__(self,\n roi_layer_type='RoIAlign',\n featmap_stride=16,\n output_size=16,\n sampling_ratio=0,"
+ },
+ {
+ "comment": "This code defines a class for a 3D head that takes input features and regions of interest (ROIs) to extract features using the RoIAlign layer. It also includes an optional temporal pooling operation and an initialization function. The forward method performs feature extraction given the input features, ROIs, and number of ROIs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/single_straight3d.py\":28-54",
+ "content": " pool_mode='avg',\n aligned=True,\n with_temporal_pool=True,\n with_global=False):\n super().__init__()\n self.roi_layer_type = roi_layer_type\n assert self.roi_layer_type in ['RoIPool', 'RoIAlign']\n self.featmap_stride = featmap_stride\n self.spatial_scale = 1. / self.featmap_stride\n self.output_size = output_size\n self.sampling_ratio = sampling_ratio\n self.pool_mode = pool_mode\n self.aligned = aligned\n self.with_temporal_pool = with_temporal_pool\n self.with_global = with_global\n self.roi_layer = RoIAlign(resolution=self.output_size,\n spatial_scale=self.spatial_scale,\n sampling_ratio=self.sampling_ratio,\n aligned=self.aligned)\n def init_weights(self):\n pass\n # The shape of feat is N, C, T, H, W\n def forward(self, feat, rois, rois_num):\n if len(feat) >= 2:"
+ },
+ {
+ "comment": "This code performs temporal pooling, concatenates slow and fast features, extracts frame-wise features using index selection, squeezes the dimensions to prevent deletion when N=1, applies a ROI layer on each frame, stacks the resulting features along axis 2, and returns the final output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/single_straight3d.py\":55-78",
+ "content": " assert self.with_temporal_pool\n if self.with_temporal_pool:\n xi = 0\n for x in feat:\n xi = xi + 1\n y = paddle.mean(x, 2, keepdim=True)\n feat = [paddle.mean(x, 2, keepdim=True) for x in feat]\n feat = paddle.concat(feat, axis=1) # merge slow and fast\n roi_feats = []\n for t in range(feat.shape[2]):\n if type(t) == paddle.static.Variable:\n index = paddle.to_tensor(t)\n else:\n data_index = np.array([t]).astype('int32')\n index = paddle.to_tensor(data_index)\n frame_feat = paddle.index_select(feat, index, axis=2)\n frame_feat = paddle.squeeze(frame_feat,\n axis=2) #axis=2,\u907f\u514dN=1\u65f6, \u7b2c\u4e00\u7ef4\u5ea6\u88ab\u5220\u9664.\n roi_feat = self.roi_layer(frame_feat, rois, rois_num)\n roi_feats.append(roi_feat)\n ret = paddle.stack(roi_feats, axis=2)\n return ret"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5014415b-96eb-4188-85fe-6c50f490f696.json b/docs/doc/5014415b-96eb-4188-85fe-6c50f490f696.json
new file mode 100644
index 000000000..ac8ba843f
--- /dev/null
+++ b/docs/doc/5014415b-96eb-4188-85fe-6c50f490f696.json
@@ -0,0 +1,30 @@
+{
+ "summary": "This code uses MeanAveragePrecisionCalculator to calculate mAP for ranked lists, initializes AveragePrecisionCalculator objects, supports interpolated precisions, and ensures shape compatibility. It averages average precisions of each class to provide the final result as mAP.",
+ "details": [
+ {
+ "comment": "This code calculates the mean average precision for a ranked list of items. It provides an interface to calculate this metric for the entire list or top-n ranked items. The example usage demonstrates accumulating data in parts and then using peek_map_at_n function to calculate the final result. The provided numpy array is used for demonstration purposes, representing a ranked list of values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py\":0-26",
+ "content": "# Copyright 2016 Google Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Calculate the mean average precision.\nIt provides an interface for calculating mean average precision\nfor an entire list or the top-n ranked items.\nExample usages:\nWe first call the function accumulate many times to process parts of the ranked\nlist. After processing all the parts, we call peek_map_at_n\nto calculate the mean average precision.\n```\nimport random\np = np.array([[random.random() for _ in xrange(50)] for _ in xrange(1000)])"
+ },
+ {
+ "comment": "Creates a numpy array with 1000 samples, each containing 50 binary random choices. Initializes MeanAveragePrecisionCalculator object with specified number of classes (in this case, 50). Accumulates predictions and ground truth for calculating average precision. Retrieves the average precision map at a given point in time.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py\":27-58",
+ "content": "a = np.array([[random.choice([0, 1]) for _ in xrange(50)]\n for _ in xrange(1000)])\n# mean average precision for 50 classes.\ncalculator = mean_average_precision_calculator.MeanAveragePrecisionCalculator(\n num_class=50)\ncalculator.accumulate(p, a)\naps = calculator.peek_map_at_n()\n```\n\"\"\"\nimport numpy\nfrom . import average_precision_calculator\nclass MeanAveragePrecisionCalculator(object):\n \"\"\"This class is to calculate mean average precision.\n \"\"\"\n def __init__(self, num_class):\n \"\"\"Construct a calculator to calculate the (macro) average precision.\n Args:\n num_class: A positive Integer specifying the number of classes.\n top_n_array: A list of positive integers specifying the top n for each\n class. The top n in each class will be used to calculate its average\n precision at n.\n The size of the array must be num_class.\n Raises:\n ValueError: An error occurred when num_class is not a positive integer;\n or the top_n_array is not a list of positive integers."
+ },
+ {
+ "comment": "This code defines a class for calculating Mean Average Precision (mAP) in the context of video classification. The constructor checks if num_class is a positive integer and initializes a list to store AveragePrecisionCalculator objects. The accumulate method takes predictions and actuals as input, accumulating prediction scores with their corresponding ground truth labels. If num_positives is provided, it represents the number of true positives for each class; otherwise, it defaults to no value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py\":59-79",
+ "content": " \"\"\"\n if not isinstance(num_class, int) or num_class <= 1:\n raise ValueError(\"num_class must be a positive integer.\")\n self._ap_calculators = [] # member of AveragePrecisionCalculator\n self._num_class = num_class # total number of classes\n for i in range(num_class):\n self._ap_calculators.append(\n average_precision_calculator.AveragePrecisionCalculator())\n def accumulate(self, predictions, actuals, num_positives=None):\n \"\"\"Accumulate the predictions and their ground truth labels.\n Args:\n predictions: A list of lists storing the prediction scores. The outer\n dimension corresponds to classes.\n actuals: A list of lists storing the ground truth labels. The dimensions\n should correspond to the predictions input. Any value\n larger than 0 will be treated as positives, otherwise as negatives.\n num_positives: If provided, it is a list of numbers representing the\n number of true positives for each class. If not provided, the number of"
+ },
+ {
+ "comment": "This code calculates the mean average precision for each class in a dataset, and provides methods to clear and check if the calculators are empty. The peek_map_at_n function returns an array of non-interpolated average precisions at n for each class. It also checks for shape compatibility between predictions and actuals arrays.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py\":80-111",
+ "content": " true positives will be inferred from the 'actuals' array.\n Raises:\n ValueError: An error occurred when the shape of predictions and actuals\n does not match.\n \"\"\"\n if not num_positives:\n num_positives = [None for i in predictions.shape[1]]\n calculators = self._ap_calculators\n for i in range(len(predictions)):\n calculators[i].accumulate(predictions[i], actuals[i],\n num_positives[i])\n def clear(self):\n for calculator in self._ap_calculators:\n calculator.clear()\n def is_empty(self):\n return ([calculator.heap_size for calculator in self._ap_calculators] ==\n [0 for _ in range(self._num_class)])\n def peek_map_at_n(self):\n \"\"\"Peek the non-interpolated mean average precision at n.\n Returns:\n An array of non-interpolated average precision at n (default 0) for each\n class.\n \"\"\"\n aps = [\n self._ap_calculators[i].peek_ap_at_n()\n for i in range(self._num_class)"
+ },
+ {
+ "comment": "This code calculates the mean average precision (mAP) by averaging the average precisions of each class. It returns the mAP value as a result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/youtube8m/mean_average_precision_calculator.py\":112-113",
+ "content": " ]\n return aps"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5040f134-25fe-4f2c-acdc-a26d37111ea0.json b/docs/doc/5040f134-25fe-4f2c-acdc-a26d37111ea0.json
new file mode 100644
index 000000000..53cb0959a
--- /dev/null
+++ b/docs/doc/5040f134-25fe-4f2c-acdc-a26d37111ea0.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code repository features a PaddlePaddle implementation of Table Tennis action recognition with the VideoSwinTransformer model, supporting feature extraction, classification, single/multi-GPU training, and pre-trained models. Running prediction generates gif files overlaid with predictions, while optimization can be done by adjusting sampling parameters or hyperparameters.",
+ "details": [
+ {
+ "comment": "This code repository contains a PaddlePaddle implementation of Table Tennis action recognition using the VideoSwinTransformer model. The code is based on PaddlePaddle 2.2, and the training data should be placed in the \"data\" directory. It includes steps for image feature extraction and action classification using SwinTransformer3D and I3DHead respectively. Training can be done with single or multi-GPU configurations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/ActionRecognition/README.md\":0-42",
+ "content": "# \u4e52\u4e53\u7403\u52a8\u4f5c\u8bc6\u522b\u6a21\u578b\n## \u5185\u5bb9\n- [\u6a21\u578b\u7b80\u4ecb](#\u6a21\u578b\u7b80\u4ecb)\n- [\u6570\u636e\u51c6\u5907](#\u6570\u636e\u51c6\u5907)\n- [\u6a21\u578b\u8bad\u7ec3](#\u6a21\u578b\u8bad\u7ec3)\n- [\u6a21\u578b\u63a8\u7406](#\u6a21\u578b\u63a8\u7406)\n- [\u6a21\u578b\u4f18\u5316](#\u6a21\u578b\u4f18\u5316)\n- [\u6a21\u578b\u90e8\u7f72](#\u6a21\u578b\u90e8\u7f72)\n- [\u53c2\u8003\u8bba\u6587](#\u53c2\u8003\u8bba\u6587)\n\u5728\u5f00\u59cb\u4f7f\u7528\u4e4b\u524d\uff0c\u60a8\u9700\u8981\u6309\u7167\u4ee5\u4e0b\u547d\u4ee4\u5b89\u88c5\u989d\u5916\u7684\u4f9d\u8d56\u5305\uff1a\n```bash\npython -m pip install imageio\n```\n## \u6a21\u578b\u7b80\u4ecb\n\u8be5\u4ee3\u7801\u5e93\u7528\u4e8e\u4e52\u4e53\u7403\u52a8\u4f5c\u8bc6\u522b, \u57fa\u4e8epaddle2.2\u7248\u672c\u5f00\u53d1\uff0c\u7ed3\u5408PaddleVideo\u4e2d\u7684VideoSwinTransformer\u6a21\u578b\uff0c\u5bf9\u7ed9\u5b9a\u7684\u4e52\u4e53\u7403\u89c6\u9891\u8fdb\u884c\u52a8\u4f5c\u5206\u7c7b\u3002\n\u4e3b\u8981\u5206\u4e3a\u5982\u4e0b\u51e0\u6b65\n - \u56fe\u50cf\u7279\u5f81\u62bd\u53d6\uff0cSwinTransformer3D\n - \u52a8\u4f5c\u5206\u7c7b\uff0cI3DHead\n## \u6570\u636e\u51c6\u5907\nTODO\n## \u6a21\u578b\u8bad\u7ec3\n\u4e3b\u8981\u4ee3\u7801\u6765\u81eaVideoSwin\u6a21\u578b\uff1a[VideoSwin](../../../docs/zh-CN/model_zoo/recognition/videoswin.md)\n1. \u4f7f\u7528VideoSwin\u5728K400\u4e0a\u7684\u9884\u8bad\u7ec3\u6a21\u578b\u57fa\u7840\u4e0a\u8fdb\u884cfinetune\uff0c\u56e0\u6b64\u9996\u5148\u4e0b\u8f7dK400\u7684\u9884\u8bad\u7ec3\u6a21\u578b\u5e76\u653e\u7f6e\u5230`data`\u76ee\u5f55\u4e0b\n ```bash\n wget -P data/ https://videotag.bj.bcebos.com/PaddleVideo-release2.2/VideoSwin_k400.pdparams\n ```\n2. \u4f7f\u7528`TableTennis/ActionRecognition/configs/videoswin_tabletennis.yaml`\u914d\u7f6e\u6587\u4ef6\u8fdb\u884c\u8bad\u7ec3\n \u8bad\u7ec3\u542f\u52a8\u547d\u4ee4\u5982\u4e0b\uff1a\n ```bash\n # \u5355\u5361\n python3.7 -u main.py --amp --validate -c applications/TableTennis/ActionRecognition/configs/videoswin_tabletennis.yaml\n # \u591a\u5361\n python3.7 -u -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_videoswin_tabletennis main.py --amp --validate -c applications/TableTennis/ActionRecognition/configs/videoswin_tabletennis.yaml"
+ },
+ {
+ "comment": "This code snippet is for training and testing the VideoSwin transformer model on the TableTennis dataset using PaddlePaddle. It provides instructions to download pre-trained models and example input video files, and demonstrates how to export an inference model using the provided config file and model parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/ActionRecognition/README.md\":43-65",
+ "content": " ```\n## \u6a21\u578b\u8bc4\u4f30\n```bash\npython3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_videoswin_tabletennis main.py --test -c configs/recognition/video_swin_transformer/videoswin_tabletennis.yaml -w \"output/VideoSwin_TableTennis/VideoSwin_TableTennis_best.pdparams\"\n```\n## \u6a21\u578b\u63a8\u7406\n\u6211\u4eec\u63d0\u4f9b\u4e86\u4e00\u4e2a\u5728\u4e52\u4e53\u7403\u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u597d\u7684\u6a21\u578b\u4ee5\u53ca\u4e00\u4e2a\u4e52\u4e53\u7403\u6837\u4f8b\u7684\u89c6\u9891pkl\u6587\u4ef6\uff0c\u4ee5\u4f9b\u6d4b\u8bd5\n```\nwget -P data/ https://videotag.bj.bcebos.com/PaddleVideo-release2.2/VideoSwin_tennis.pdparams # \u4e0b\u8f7d\u4e52\u4e53\u7403\u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u597d\u7684\u6a21\u578b\nwget -P data/ https://videotag.bj.bcebos.com/Data/example_tennis.pkl # \u4e0b\u8f7d\u4e52\u4e53\u7403\u6837\u4f8b\u8f93\u5165\u89c6\u9891pkl\u6587\u4ef6\n```\n### \u5bfc\u51fa\u63a8\u7406\u6a21\u578b\n```\npython3.7 tools/export_model.py -c applications/TableTennis/ActionRecognition/configs/videoswin_tabletennis.yaml \\\n -p output/VideoSwin_TableTennis/VideoSwin_TableTennis_best.pdparams \\\n -o inference/VideoSwin_TableTennis\n```\n\u4e0a\u8ff0\u547d\u4ee4\u4f1a\u6839\u636e\u4f20\u5165\u7684`.pdparams`\u6a21\u578b\uff0c\u5728`inference/VideoSwin_TableTennis`\u6587\u4ef6\u5939\u4e0b\u751f\u6210\u63a8\u7406\u6a21\u578b\uff0c\u4e3b\u8981\u5305\u62ec3\u4e2a\u6587\u4ef6\uff1a`VideoSwin_TableTennis.pdiparams`\u3001`VideoSwin_TableTennis.pdmodel`\u3001`VideoSwin_TableTennis.info`"
+ },
+ {
+ "comment": "Running prediction code with provided arguments will generate a gif file showing the video overlaid with predicted results (top 1 class and probability) in the results folder.\nThe model can be optimized based on video content, by adjusting sampling parameters like num_seg and seg_len or hyperparameters for training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/ActionRecognition/README.md\":67-97",
+ "content": "### \u4f7f\u7528\u63a8\u7406\u6a21\u578b\n\u6d4b\u8bd5\u6587\u4ef6\u4f7f\u7528`.pkl`\u6587\u4ef6\uff0c\u5176\u5305\u542b\u4e86\u5df2\u62bd\u53d6\u7684\u7528\u4e8e\u9884\u6d4b\u7684\u4e52\u4e53\u7403\u89c6\u9891\u5e27\u3002\n\u8fd0\u884c\u9884\u6d4b\u4ee3\u7801\n```bash\npython3.7 tools/predict.py --input_file data/example_tennis_7.pkl \\\n --config applications/TableTennis/ActionRecognition/configs/videoswin_tabletennis.yaml \\\n --model_file inference/VideoSwin_TableTennis/VideoSwin_TableTennis.pdmodel \\\n --params_file inference/VideoSwin_TableTennis/VideoSwin_TableTennis.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\n\u6267\u884c\u4ee5\u4e0a\u547d\u4ee4\u4f1a\u4ea7\u51fa\u4e00\u4e2a\u539f\u89c6\u9891\u53e0\u52a0\u9884\u6d4b\u7ed3\u679c\u6587\u672c(Top1\u7c7b\u522b+\u6982\u7387)\u7684gif\u56fe\u7247\uff0c\u4fdd\u5b58\u5728\u672c\u76ee\u5f55\u7684results\u6587\u4ef6\u5939\u4e0b\uff0cgif\u6587\u4ef6\u540d\u4e0e\u8f93\u5165\u7684pkl\u6587\u4ef6\u540d\u76f8\u540c\u3002\n\u6548\u679c\u5982\u4e0b\u56fe\uff1a\n\n## \u6a21\u578b\u4f18\u5316\n\u5728\u5b9e\u9645\u4f7f\u7528\u573a\u666f\u4e2d\u53ef\u6839\u636e\u89c6\u9891\u5185\u5bb9\u5c1d\u8bd5\u4f18\u5316\u7b56\u7565\n- \u53ef\u6839\u636e\u52a8\u4f5c\u6301\u7eed\u65f6\u95f4\u7684\u957f\u77ed\uff0c\u8c03\u6574\u91c7\u6837\u7684\u6bb5\u6570num_seg\u548c\u6bb5\u5185\u91c7\u6837\u7684\u5e27\u6570seg_len\n- \u53ef\u4ee5\u6839\u636e\u6570\u636e\u96c6\u5927\u5c0f\u8c03\u6574\u6a21\u578b\u8bad\u7ec3\u7684\u8d85\u53c2\u6570\uff0c\u5305\u62ec\u6743\u91cd\u8870\u51cf\u3001DropOut\u6982\u7387\u3001\u5b66\u4e60\u7387\u3001\u66f4\u6362\u4f18\u5316\u5668\u7b49\uff0c\u4ee5\u83b7\u5f97\u66f4\u4f18\u7684\u7ed3\u679c\u3002\n- \u672c\u4ee3\u7801\u7684backbone\u90e8\u5206\u53ef\u4ee5\u4f5c\u4e3a\u89c6\u9891\u7279\u5f81\u63d0\u53d6\u6a21\u5757\uff0c\u4ee3\u66ff\u5176\u5b83\u7684\u52a8\u4f5c\u8bc6\u522bbackbone\uff0c\u4ee5\u83b7\u5f97\u8868\u5f81\u80fd\u529b\u66f4\u5f3a\u7684\u89c6\u9891\u7279\u5f81\uff0c\u4ee5\u63d0\u5347\u6574\u4f53\u4efb\u52a1\u7684\u7cbe\u5ea6\u3002\n## \u6a21\u578b\u90e8\u7f72\nTODO\n## \u53c2\u8003\u8bba\u6587\n- [Video Swin Transformer](https://arxiv.org/pdf/2106.13230.pdf), Ze Liu, Jia Ning, Yue Cao, Yixuan Wei"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/50866672-274d-4c7f-94c1-0ff51774db70.json b/docs/doc/50866672-274d-4c7f-94c1-0ff51774db70.json
new file mode 100644
index 000000000..162700dab
--- /dev/null
+++ b/docs/doc/50866672-274d-4c7f-94c1-0ff51774db70.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code imports functions from submodules \"optimizer\" and \"lr\" to build optimizer and learning rate for PaddleVideo's Video Quality Assessment application.",
+ "details": [
+ {
+ "comment": "This code imports functions from submodules \"optimizer\" and \"lr\" to build optimizer and learning rate for PaddleVideo's Video Quality Assessment application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/__init__.py\":0-16",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .optimizer import build_optimizer\nfrom .lr import build_lr"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/50c4b9b3-d253-4b49-96f6-0c5f6f76e603.json b/docs/doc/50c4b9b3-d253-4b49-96f6-0c5f6f76e603.json
new file mode 100644
index 000000000..20a4307bc
--- /dev/null
+++ b/docs/doc/50c4b9b3-d253-4b49-96f6-0c5f6f76e603.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The BaseMetric class serves as a foundation for various video quality assessment metrics, requiring subclasses to implement the update and overridden methods. It utilizes numpy, paddle, and PaddleVideo's utils for data manipulation and distribution information.",
+ "details": [
+ {
+ "comment": "This Python class, named BaseMetric, is a base class for different video quality assessment metrics. It initializes with data size, batch size, log interval, and optional keyword arguments. The update method must be implemented by subclasses to update the metric values. The class also has abstract methods that must be overridden in subclasses for actual functionality. It utilizes numpy, paddle, and PaddleVideo's utils for data manipulation and distribution information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/base.py\":0-35",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n\"\"\"\nfrom abc import abstractmethod\nimport numpy as np\nimport paddle\nfrom paddlevideo.utils import get_dist_info\nfrom .registry import METRIC\nclass BaseMetric(object):\n \"\"\"Base Metric\"\"\"\n def __init__(self, data_size, batch_size, log_interval=1, **kwargs):\n self.data_size = data_size\n self.batch_size = batch_size\n _, self.world_size = get_dist_info()\n self.log_interval = log_interval\n @abstractmethod\n def update(self):\n \"\"\"update\"\"\"\n raise NotImplementedError\n @abstractmethod"
+ },
+ {
+ "comment": "This code defines an \"accumulate\" method in a base class, but it raises a NotImplementedError to indicate that subclasses must override this method with their own implementation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/metrics/base.py\":36-38",
+ "content": " def accumulate(self):\n \"\"\"accumulate\"\"\"\n raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/50c87944-4812-4383-b04b-bddfb3fcb327.json b/docs/doc/50c87944-4812-4383-b04b-bddfb3fcb327.json
new file mode 100644
index 000000000..57986ef66
--- /dev/null
+++ b/docs/doc/50c87944-4812-4383-b04b-bddfb3fcb327.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code installs PaddleVideo's VideoTag app, provides instructions for data preparation and model inference, and represents a dictionary containing information about classified video objects.",
+ "details": [
+ {
+ "comment": "This code provides installation instructions for PaddleVideo's VideoTag application, including dependencies and downloading pre-trained weights. It also outlines the data preparation process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/Run.md\":0-53",
+ "content": "# \u6837\u4f8b\u4ee3\u7801\u8fd0\u884c\u6307\u5357\n---\n## \u5185\u5bb9\n\u53c2\u8003\u672c\u6587\u6863\uff0c\u60a8\u53ef\u4ee5\u5feb\u901f\u719f\u6089VideoTag\u7684\u4f7f\u7528\u65b9\u6cd5\uff0c\u89c2\u5bdfVideoTag\u7684\u9884\u8bad\u7ec3\u6a21\u578b\u5728\u793a\u4f8b\u89c6\u9891\u4e0a\u7684\u9884\u6d4b\u7ed3\u679c\u3002\n\u6587\u6863\u5185\u5bb9\u5305\u62ec:\n- [\u5b89\u88c5\u8bf4\u660e](#\u5b89\u88c5\u8bf4\u660e)\n- [\u6570\u636e\u51c6\u5907](#\u6570\u636e\u51c6\u5907)\n- [\u6a21\u578b\u63a8\u65ad](#\u6a21\u578b\u63a8\u65ad)\n## \u5b89\u88c5\u8bf4\u660e\n### \u73af\u5883\u4f9d\u8d56\uff1a\n```\n CUDA >= 9.0\n cudnn >= 7.5\n```\n### \u4f9d\u8d56\u5b89\u88c5:\n- 1.7.0 <= PaddlePaddle\u7248\u672c <= 2.0.0: pip install paddlepaddle-gpu==1.8.4.post97 -i https://mirror.baidu.com/pypi/simple\n- opencv\u7248\u672c >= 4.1.0: pip install opencv-python==4.2.0.32\n## \u6570\u636e\u51c6\u5907\n### \u9884\u8bad\u7ec3\u6743\u91cd\u4e0b\u8f7d\n\u6211\u4eec\u63d0\u4f9b\u4e86[TSN](https://videotag.bj.bcebos.com/video_tag_tsn.tar)\u548c[AttentionLSTM](https://videotag.bj.bcebos.com/video_tag_lstm.tar)\u9884\u8bad\u7ec3\u6743\u91cd\uff0c\u8bf7\u5728video\\_tag\u76ee\u5f55\u4e0b\u65b0\u5efaweights\u76ee\u5f55\uff0c\u5e76\u5c06\u4e0b\u8f7d\u89e3\u538b\u540e\u7684\u53c2\u6570\u6587\u4ef6\u653e\u5728weights\u76ee\u5f55\u4e0b:\n```\n mkdir weights\n cd weights\n wget https://videotag.bj.bcebos.com/video_tag_tsn.tar\n wget https://videotag.bj.bcebos.com/video_tag_lstm.tar\n tar -zxvf video_tag_tsn.tar\n tar -zxvf video_tag_lstm.tar\n rm video_tag_tsn.tar -rf\n rm video_tag_lstm.tar -rf\n mv video_tag_tsn/* .\n mv attention_lstm/* .\n rm video_tag_tsn/ -rf\n rm attention_lstm -rf\n```\n\u6240\u5f97\u76ee\u5f55\u7ed3\u6784\u5982\u4e0b\uff1a\n```\nvideo_tag\n \u251c\u2500\u2500weights\n \u251c\u2500\u2500 attention_lstm.pdmodel\n \u251c\u2500\u2500 attention_lstm.pdopt "
+ },
+ {
+ "comment": "This code provides instructions on how to download an example video for testing, how to run model inference, and how to save the results. The example video can be downloaded from a provided link and should be extracted into a specific directory structure. The model inference script is named videotag_test.py and prints prediction probabilities. Users can specify a different output directory using the --save\\_dir parameter. The predictions are saved as JSON files in the specified directory, with each file corresponding to a video.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/Run.md\":54-104",
+ "content": " \u251c\u2500\u2500 attention_lstm.pdparams\n \u251c\u2500\u2500 tsn.pdmodel\n \u251c\u2500\u2500 tsn.pdopt\n \u2514\u2500\u2500 tsn.pdparams\n```\n### \u793a\u4f8b\u89c6\u9891\u4e0b\u8f7d\n\u6211\u4eec\u63d0\u4f9b\u4e86[\u6837\u4f8b\u89c6\u9891](https://videotag.bj.bcebos.com/mp4.tar)\u65b9\u4fbf\u7528\u6237\u6d4b\u8bd5\uff0c\u8bf7\u4e0b\u8f7d\u540e\u89e3\u538b\uff0c\u5e76\u5c06\u89c6\u9891\u6587\u4ef6\u653e\u7f6e\u5728video\\_tag/data/mp4\u76ee\u5f55\u4e0b:\n```\n cd data/\n wget https://videotag.bj.bcebos.com/mp4.tar\n tar -zxvf mp4.tar\n rm mp4.tar -rf\n```\n\u6240\u5f97\u76ee\u5f55\u7ed3\u6784\u5982\u4e0b\uff1a\n```\nvideo_tag\n \u251c\u2500\u2500data\n \u251c\u2500\u2500 mp4\n \u251c\u2500\u2500 1.mp4\n \u251c\u2500\u2500 2.mp4\n \u2514\u2500\u2500 ...\n```\n## \u6a21\u578b\u63a8\u65ad\n\u6a21\u578b\u63a8\u65ad\u7684\u542f\u52a8\u65b9\u5f0f\u5982\u4e0b\uff1a\n python videotag_test.py\n- \u9884\u6d4b\u7ed3\u679c\u4f1a\u4ee5\u65e5\u5fd7\u65b9\u5f0f\u6253\u5370\uff0c\u793a\u4f8b\u5982\u4e0b:\n```\n[========video_id [ data/mp4/1.mp4 ] , topk(20) preds: ========]\nclass_id: 3110, class_name: \u8bad\u7ec3 , probability: 0.97730666399\nclass_id: 2159, class_name: \u8e72 , probability: 0.945082366467\n...\n[========video_id [ data/mp4/2.mp4 ] , topk(20) preds: ========]\nclass_id: 2773, class_name: \u821e\u8e48 , probability: 0.850423932076\nclass_id: 1128, class_name: \u8868\u6f14\u827a\u672f , probability: 0.0446354188025\n...\n```\n- \u901a\u8fc7--save\\_dir\u53ef\u6307\u5b9a\u9884\u6d4b\u7ed3\u679c\u5b58\u50a8\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3avideo\\_tag/data/VideoTag\\_results\uff0c\u4e0d\u540c\u8f93\u5165\u89c6\u9891\u7684\u9884\u6d4b\u7ed3\u679c\u5206\u6587\u4ef6\u4fdd\u5b58\u5728\u4e0d\u540c\u7684json\u6587\u4ef6\u4e2d\uff0c\u6587\u4ef6\u7684\u5185\u5bb9\u683c\u5f0f\u4e3a\uff1a\n```\n [file_path,\n {\"class_name\": class_name1, \"probability\": probability1, \"class_id\": class_id1},"
+ },
+ {
+ "comment": "This code represents a dictionary containing information about a classified video object. The 'class_name' key holds the name of the class, 'probability' stores the confidence level of the classification, and 'class_id' contains the identifier of the recognized class. These dictionaries are stored in an array, potentially for multiple classifications within the same video or different videos.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/Run.md\":105-108",
+ "content": " {\"class_name\": class_name2, \"probability\": probability2, \"class_id\": class_id2},\n ...\n ]\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/50ce1e9f-ecb1-49a0-9e7f-bab5a7669020.json b/docs/doc/50ce1e9f-ecb1-49a0-9e7f-bab5a7669020.json
new file mode 100644
index 000000000..bc50443c9
--- /dev/null
+++ b/docs/doc/50ce1e9f-ecb1-49a0-9e7f-bab5a7669020.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code downloads, extracts and provides label information for 19228 videos' feature frames in \"activitynet_1.3_annotations.json\" for PaddleVideo model pre-training, using decompressed data from \"bmn_feat.tar.gz\". Users need to modify `feat_path` and `file_path` in the configuration file.",
+ "details": [
+ {
+ "comment": "ActivityNet is a large-scale dataset for video understanding tasks like action localization and recognition. The code provides instructions on how to download the processed ActivityNet 1.3 dataset, consisting of videos with corresponding labels, durations, and frames. Users can choose between two methods: downloading precompressed packages or clicking provided hyperlinks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ActivityNet.md\":0-23",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../zh-CN/dataset/ActivityNet.md) | English\n# ActivityNet data preparation\n- [Introduction](#Introduction)\n- [Download](#Download)\n## Introduction\nActivityNet is a dataset for large-scale video understanding tasks, which can be used for tasks such as action localization, action recognition, etc.\n## Download\n1. The BMN model uses the processed ActivityNet 1.3 dataset. There are two ways to use it:\n - Using our processed ActivityNet 1.3 dataset (compressed package is about 5.5G), each video has corresponding action labels, duration intervals, duration frames, duration seconds and other information\n Download with the following command:\n ```bash\n wget https://paddlemodels.bj.bcebos.com/video_detection/bmn_feat.tar.gz # Download the processed video feature data\n wget https://paddlemodels.bj.bcebos.com/video_detection/activitynet_1.3_annotations.json # Download the processed label data\n ```\n Or click the following hyperlinks to download:\n [Video feature data](https://paddlemodels.bj.bcebos.com/video_detection/bmn_feat.tar.gz)"
+ },
+ {
+ "comment": "The code is explaining how to download and extract video feature data from the \"activitynet_1.3_annotations.json\" file for a model in PaddleVideo. It mentions decompressing the \"bmn_feat.tar.gz\" file, extracting features by yourself using TSN, and providing the necessary files and instructions to download and pre-train the TSN model. The \"activitynet_1.3_annotations.json\" file contains information about video annotations for training purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ActivityNet.md\":24-44",
+ "content": " [Video feature data](https://paddlemodels.bj.bcebos.com/video_detection/activitynet_1.3_annotations.json)\n then decompression `bmn_feat.tar.gz`\n ```bash\n tar -xf bmn_feat.tar.gz\n ```\n - Extract features by yourself\n First refer to [Download Instructions](https://github.com/activitynet/ActivityNet/tree/master/Crawler) to download the original dataset. When training this model, you need to use TSN to extract features from the source files first. You can [self-extract](https://github.com/yjxiong/temporal-segment-networks) video frame and optical flow information, and the pre-trained TSN model can be downloaded from [here](https://github.com/ yjxiong/anet2016-cuhk) download.\n The information in the `activitynet_1.3_annotations.json` tag file is as follows:\n ```json\n {\n \"v_QOlSCBRmfWY\": {\n \"duration_second\": 82.73,\n \"subset\": \"training\",\n \"duration_frame\": 2067,\n \"annotations\": [{\n \"segment\": [6.195294851794072, 77.73085420904837],"
+ },
+ {
+ "comment": "The code represents a dictionary containing label information and video feature frame data for 19228 videos. Each key represents a video, and the corresponding value is another dictionary with 'duration_second', 'subset', 'duration_frame', 'feature_frame' keys, and an array of 'annotations' which includes 'segment' (time range) and 'label' information. The code also mentions that there will be 19228 video feature npy files obtained from the activitynet_1.3_annotations.json file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ActivityNet.md\":45-76",
+ "content": " \"label\": \"Ballet\"\n }],\n \"feature_frame\": 2064\n },\n \"v_ehGHCYKzyZ8\": {\n \"duration_second\": 61.7189999999999994,\n \"subset\": \"training\",\n \"duration_frame\": 1822,\n \"annotations\": [{\n \"segment\": [43.95990729267573, 45.401932082395355],\n \"label\": \"Doing crunches\"\n }],\n \"feature_frame\": 1808\n },\n ...,\n ...\n }\n ```\n In the end, `19228` video feature npy files are obtained, corresponding to the `19228` label information in the `activitynet_1.3_annotations.json` file.\n2. Create a new `data/bmn_data` folder, and then unzip the video feature data after downloading and put it in this folder, and finally it should be organized into the following form:\n ```\n PaddleVideo\n \u251c\u2500\u2500 data\n \u2502 \u251c\u2500\u2500 bmn_data\n \u2502 \u2502 \u251c\u2500\u2500 fix_feat_100\n \u2502 \u2502 \u2502 \u251c\u2500\u2500 v___c8enCfzqw.npy\n \u2502 \u2502 \u2502 \u251c\u2500\u2500 v___dXUJsj3yo.npy\n \u2502 \u2502 \u2502 \u251c\u2500\u2500 ...\n \u2502 \u2502 \u2502\n \u2502 \u2502 \u2514\u2500\u2500 activitynet_1.3_annotations.json"
+ },
+ {
+ "comment": "In the code, it is instructing to modify two fields in the configuration file. The `feat_path` field needs updating with the feature directory path, and the `file_path` should be specified for the label file path. This ensures proper data access during program execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ActivityNet.md\":77-79",
+ "content": " ```\n3. Finally, modify the `feat_path` field in the configuration file configs/localization/bmn.yaml to specify the feature directory path, and the `file_path` field to specify the label file path."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/51e06ebf-f1dc-4c8b-b8ca-8d4734756a79.json b/docs/doc/51e06ebf-f1dc-4c8b-b8ca-8d4734756a79.json
new file mode 100644
index 000000000..28c346f0a
--- /dev/null
+++ b/docs/doc/51e06ebf-f1dc-4c8b-b8ca-8d4734756a79.json
@@ -0,0 +1,45 @@
+{
+ "summary": "Two learning rate scheduler classes, CustomWarmupCosineDecay and CosineAnnealingDecay, are provided for optimizing models with warm-up and stepwise cosine decay. The `CustomWarmupPiecewiseDecay` class is a custom scheduler for PaddleVideo, implementing piecewise function and warmup phase with linear decay.",
+ "details": [
+ {
+ "comment": "This code defines a custom learning rate scheduler called CustomWarmupCosineDecay, which combines warm-up and stepwise cosine decay for optimizing models. It extends the LRScheduler class and allows users to define specific start learning rates and the number of epochs for warm-up before applying stepwise cosine decay.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":0-32",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport math\nfrom paddle.optimizer.lr import *\n\"\"\"\nPaddleVideo Learning Rate Schedule:\nYou can use paddle.optimizer.lr\nor define your custom_lr in this file.\n\"\"\"\nclass CustomWarmupCosineDecay(LRScheduler):\n \"\"\"\n We combine warmup and stepwise-cosine which is used in slowfast model.\n Args:\n warmup_start_lr (float): start learning rate used in warmup stage.\n warmup_epochs (int): the number epochs of warmup."
+ },
+ {
+ "comment": "This code defines a class \"CosineAnnealingDecay\" for scheduling the learning rate. It takes parameters such as base learning rate, total epochs, number of iterations per epoch, and initializes instance variables accordingly. The step() method will update the last_lr/last_epoch/base_lr based on the provided parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":33-54",
+ "content": " cosine_base_lr (float|int, optional): base learning rate in cosine schedule.\n max_epoch (int): total training epochs.\n num_iters(int): number iterations of each epoch.\n last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.\n verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .\n Returns:\n ``CosineAnnealingDecay`` instance to schedule learning rate.\n \"\"\"\n def __init__(self,\n warmup_start_lr,\n warmup_epochs,\n cosine_base_lr,\n max_epoch,\n num_iters,\n last_epoch=-1,\n verbose=False):\n self.warmup_start_lr = warmup_start_lr\n self.warmup_epochs = warmup_epochs\n self.cosine_base_lr = cosine_base_lr\n self.max_epoch = max_epoch\n self.num_iters = num_iters\n #call step() in base class, last_lr/last_epoch/base_lr will be update"
+ },
+ {
+ "comment": "The code defines a CustomWarmupCosineDecay class that extends the Optimizer. It has an __init__ method to initialize last_epoch and verbose, and a step method to update learning rate based on current epoch. The step method also handles cases where epoch is None or provided manually. Additionally, there is a _lr_func_cosine method for calculating the learning rate using a cosine function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":55-80",
+ "content": " super(CustomWarmupCosineDecay, self).__init__(last_epoch=last_epoch,\n verbose=verbose)\n def step(self, epoch=None):\n \"\"\"\n ``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .\n The new learning rate will take effect on next ``optimizer.step`` .\n Args:\n epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.\n Returns:\n None\n \"\"\"\n if epoch is None:\n if self.last_epoch == -1:\n self.last_epoch += 1\n else:\n self.last_epoch += 1 / self.num_iters # update step with iters\n else:\n self.last_epoch = epoch\n self.last_lr = self.get_lr()\n if self.verbose:\n print('Epoch {}: {} set learning rate to {}.'.format(\n self.last_epoch, self.__class__.__name__, self.last_lr))\n def _lr_func_cosine(self, cur_epoch, cosine_base_lr, max_epoch):"
+ },
+ {
+ "comment": "This code defines a custom learning rate (LR) scheduler that combines warmup and stepwise-cosine decay. It starts with a warmup phase, then uses a cosine annealing LR schedule. The `get_lr` function calculates the current learning rate based on the current epoch, maximum epoch, warmup epochs, and other parameters. This scheduler is used in the \"slowfast\" model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":81-107",
+ "content": " \"\"\"start to cosine\"\"\"\n return cosine_base_lr * (math.cos(math.pi * cur_epoch / max_epoch) +\n 1.0) * 0.5\n def get_lr(self):\n \"\"\"Define lr policy\"\"\"\n lr = self._lr_func_cosine(self.last_epoch, self.cosine_base_lr,\n self.max_epoch)\n lr_end = self._lr_func_cosine(self.warmup_epochs, self.cosine_base_lr,\n self.max_epoch)\n # Perform warm up.\n if self.last_epoch < self.warmup_epochs:\n lr_start = self.warmup_start_lr\n alpha = (lr_end - lr_start) / self.warmup_epochs\n lr = self.last_epoch * alpha + lr_start\n return lr\nclass CustomWarmupPiecewiseDecay(LRScheduler):\n \"\"\"\n This op combine warmup and stepwise-cosine which is used in slowfast model.\n Args:\n warmup_start_lr (float): start learning rate used in warmup stage.\n warmup_epochs (int): the number epochs of warmup.\n step_base_lr (float|int, optional): base learning rate in step schedule."
+ },
+ {
+ "comment": "This code defines a class `CustomWarmupPiecewiseDecay` for scheduling learning rates. It takes several parameters like warmup start lr, warmup epochs, step base lr, lrs (list of lr values), gamma, steps, max_epoch, num_iters, last_epoch and verbose. The constructor initializes the class with these parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":108-133",
+ "content": " max_epoch (int): total training epochs.\n num_iters(int): number iterations of each epoch.\n last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.\n verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .\n Returns:\n ``CustomWarmupPiecewiseDecay`` instance to schedule learning rate.\n \"\"\"\n def __init__(self,\n warmup_start_lr,\n warmup_epochs,\n step_base_lr,\n lrs,\n gamma,\n steps,\n max_epoch,\n num_iters,\n last_epoch=0,\n verbose=False):\n self.warmup_start_lr = warmup_start_lr\n self.warmup_epochs = warmup_epochs\n self.step_base_lr = step_base_lr\n self.lrs = lrs\n self.gamma = gamma\n self.steps = steps\n self.max_epoch = max_epoch\n self.num_iters = num_iters"
+ },
+ {
+ "comment": "This code defines a custom learning rate scheduler for optimizers, allowing the learning rate to be updated based on epochs. The `step` function is used to update the learning rate, and the `_lr_func_steps_with_relative_lrs` function seems to set the learning rates for each parameter group.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":134-159",
+ "content": " self.last_epoch = last_epoch\n self.last_lr = self.warmup_start_lr # used in first iter\n self.verbose = verbose\n self._var_name = None\n def step(self, epoch=None, rebuild=False):\n \"\"\"\n ``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .\n The new learning rate will take effect on next ``optimizer.step`` .\n Args:\n epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.\n Returns:\n None\n \"\"\"\n if epoch is None:\n if not rebuild:\n self.last_epoch += 1 / self.num_iters # update step with iters\n else:\n self.last_epoch = epoch\n self.last_lr = self.get_lr()\n if self.verbose:\n print('Epoch {}: {} set learning rate to {}.'.format(\n self.last_epoch, self.__class__.__name__, self.last_lr))\n def _lr_func_steps_with_relative_lrs(self, cur_epoch, lrs, base_lr, steps,"
+ },
+ {
+ "comment": "This code defines a custom learning rate (LR) scheduler for the PaddleVideo library. It uses a piecewise function to define different LRs at various epochs, and also implements a warmup phase with a linear decay from an initial LR to the first defined LR after the warmup period. The code provides functions to calculate the LR at each epoch based on the given steps, LR values, base LR, and maximum epoch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":160-195",
+ "content": " max_epoch):\n \"\"\"lr func steps with relative lrs\"\"\"\n # get step index\n steps = steps + [max_epoch]\n for ind, step in enumerate(steps):\n if cur_epoch < step:\n break\n return lrs[ind - 1] * base_lr\n def get_lr(self):\n \"\"\"Define lr policy\"\"\"\n lr = self._lr_func_steps_with_relative_lrs(\n self.last_epoch,\n self.lrs,\n self.step_base_lr,\n self.steps,\n self.max_epoch,\n )\n lr_end = self._lr_func_steps_with_relative_lrs(\n self.warmup_epochs,\n self.lrs,\n self.step_base_lr,\n self.steps,\n self.max_epoch,\n )\n # Perform warm up.\n if self.last_epoch < self.warmup_epochs:\n lr_start = self.warmup_start_lr\n alpha = (lr_end - lr_start) / self.warmup_epochs\n lr = self.last_epoch * alpha + lr_start\n return lr\nclass CustomPiecewiseDecay(PiecewiseDecay):"
+ },
+ {
+ "comment": "This code defines a custom learning rate scheduler, which initializes an instance of the class and takes keyword arguments. The 'num_iters' argument is specifically excluded from being passed as a parameter.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/solver/custom_lr.py\":196-200",
+ "content": " \"\"\"CustomPiecewiseDecay\"\"\"\n def __init__(self, **kargs):\n \"\"\"start\"\"\"\n kargs.pop('num_iters')\n super().__init__(**kargs)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/526a992a-fb80-424c-8273-2748fcece73e.json b/docs/doc/526a992a-fb80-424c-8273-2748fcece73e.json
new file mode 100644
index 000000000..9945041a6
--- /dev/null
+++ b/docs/doc/526a992a-fb80-424c-8273-2748fcece73e.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code imports the function 'get_metrics' from the 'metrics_util' module in the same application directory. This function is likely used to calculate and retrieve various metrics related to video processing or analysis.",
+ "details": [
+ {
+ "comment": "This code imports the function 'get_metrics' from the 'metrics_util' module in the same application directory. This function is likely used to calculate and retrieve various metrics related to video processing or analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/__init__.py\":0-0",
+ "content": "from .metrics_util import get_metrics"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/52e7b804-02ff-454a-9a3d-fc306cd4077e.json b/docs/doc/52e7b804-02ff-454a-9a3d-fc306cd4077e.json
new file mode 100644
index 000000000..e19be9747
--- /dev/null
+++ b/docs/doc/52e7b804-02ff-454a-9a3d-fc306cd4077e.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The code loads an action detection model, reads a list of videos from a URL file, and for each video, it prints its name, creates paths for image frames and audio, calls the infer function to get bone-coordinate matrices (bmn_results) and action results, stores them in a list named 'results', and writes JSON data to \"results.json\" file.",
+ "details": [
+ {
+ "comment": "This code is importing necessary libraries, setting the path to access an action detection model. The model is loaded and a list of videos are read from a URL file. For each video, its name is printed, the required paths for image frames and audio are created, and the action detection model's infer function is called to get bone-coordinate matrices (bmn_results) and action results. These results are then stored in a list named 'results'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/predict.py\":0-32",
+ "content": "import os\nimport sys\nimport json\nsys.path.append('action_detect')\nfrom action import ActionDetection\nif __name__ == '__main__':\n #dataset_dir = \"/workspace/PaddleVideo/applications/FootballAction/datasets/EuroCup2016\"\n dataset_dir = \"../datasets/EuroCup2016\"\n model_predict = ActionDetection(cfg_file=\"./configs/configs.yaml\")\n model_predict.load_model()\n video_url = os.path.join(dataset_dir, 'url_val.list')\n with open(video_url, 'r') as f:\n lines = f.readlines()\n lines = [os.path.join(dataset_dir, k.strip()) for k in lines]\n results = []\n for line in lines:\n video_name = line\n print(video_name)\n imgs_path = video_name.replace(\".mp4\", \"\").replace(\"mp4\", \"frames\")\n pcm_path = video_name.replace(\".mp4\", \".pcm\").replace(\"mp4\", \"pcm\")\n bmn_results, action_results = model_predict.infer(imgs_path, pcm_path)\n results.append({\n 'video_name': line,\n 'bmn_results': bmn_results,\n 'action_results': action_results\n })"
+ },
+ {
+ "comment": "Writes JSON data to \"results.json\" file, ensuring UTF-8 encoding and readable indentation for improved readability.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/predict/predict.py\":34-36",
+ "content": " with open('results.json', 'w', encoding='utf-8') as f:\n data = json.dumps(results, indent=4, ensure_ascii=False)\n f.write(data)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/52ee5f2d-38b9-4f24-b70a-c634554414e2.json b/docs/doc/52ee5f2d-38b9-4f24-b70a-c634554414e2.json
new file mode 100644
index 000000000..9cd6ca3ce
--- /dev/null
+++ b/docs/doc/52ee5f2d-38b9-4f24-b70a-c634554414e2.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code contains the version information for PaddleVideo, licensed under the Apache License 2.0, and defines the current version as \"0.0.1\".",
+ "details": [
+ {
+ "comment": "This code contains the version information for PaddleVideo, licensed under the Apache License 2.0, and defines the current version as \"0.0.1\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/version.py\":0-15",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n__all__ = [\"paddlevideo_version\"]\npaddlevideo_version = \"0.0.1\""
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5319de35-5759-4c56-a6a3-0b026f570722.json b/docs/doc/5319de35-5759-4c56-a6a3-0b026f570722.json
new file mode 100644
index 000000000..ec7b3da97
--- /dev/null
+++ b/docs/doc/5319de35-5759-4c56-a6a3-0b026f570722.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code initializes and defines a class for MS-RNN/VTT model metrics computation, updates metrics using input data, calculates rank metrics (r1, r5, r10, medr, mean), logs these metrics, and signals the end of iterations.",
+ "details": [
+ {
+ "comment": "The code is from the MSRVTTMetric class in paddlevideo's metrics module. It initializes an instance of the class, prepares for metrics computation, and creates score_matrix and target_matrix using numpy with zeroes. These matrices will be used to store results during metric calculations. The class also registers with the base BaseMetric class.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/msrvtt_metric.py\":0-30",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport numpy as np\nimport paddle\nimport paddle.nn.functional as F\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass MSRVTTMetric(BaseMetric):\n def __init__(self, data_size, batch_size, log_interval=1):\n \"\"\"prepare for metrics\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.score_matrix = np.zeros((data_size, data_size))\n self.target_matrix = np.zeros((data_size, data_size))"
+ },
+ {
+ "comment": "This code initializes a rank matrix, updates score and target matrices based on input data, calculates r1, r5, r10 rank metrics and median rank (medr) and mean rank, then logs these metrics.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/msrvtt_metric.py\":31-55",
+ "content": " self.rank_matrix = np.ones((data_size)) * data_size\n def update(self, batch_id, data, outputs):\n \"\"\"update metrics during each iter\n \"\"\"\n target = data[-1]\n cm_logit = outputs[-1]\n self.score_matrix[batch_id, :] = F.softmax(\n cm_logit, axis=1)[:, 0].reshape([-1]).numpy()\n self.target_matrix[batch_id, :] = target.reshape([-1]).numpy()\n rank = np.where((np.argsort(-self.score_matrix[batch_id]) == np.where(\n self.target_matrix[batch_id] == 1)[0][0]) == 1)[0][0]\n self.rank_matrix[batch_id] = rank\n rank_matrix_tmp = self.rank_matrix[:batch_id + 1]\n r1 = 100.0 * np.sum(rank_matrix_tmp < 1) / len(rank_matrix_tmp)\n r5 = 100.0 * np.sum(rank_matrix_tmp < 5) / len(rank_matrix_tmp)\n r10 = 100.0 * np.sum(rank_matrix_tmp < 10) / len(rank_matrix_tmp)\n medr = np.floor(np.median(rank_matrix_tmp) + 1)\n meanr = np.mean(rank_matrix_tmp) + 1\n logger.info(\n \"[{}] Final r1:{:.3f}, r5:{:.3f}, r10:{:.3f}, mder:{:.3f}, meanr:{:.3f}\""
+ },
+ {
+ "comment": "This code defines a class for accumulating metrics related to the MS-RNN/VTT model. It seems to have methods for updating and finalizing the metric calculations. The update method takes in values r1, r5, r10, medr, and meanr, which are likely performance scores. The accumulate method signals the end of iterations by logging a message saying \"Eval Finished!\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/msrvtt_metric.py\":56-61",
+ "content": " .format(batch_id, r1, r5, r10, medr, meanr))\n def accumulate(self):\n \"\"\"accumulate metrics when finished all iters.\n \"\"\"\n logger.info(\"Eval Finished!\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/537299c8-cfa2-4eb5-a8b7-1f055d82abc1.json b/docs/doc/537299c8-cfa2-4eb5-a8b7-1f055d82abc1.json
new file mode 100644
index 000000000..66634b296
--- /dev/null
+++ b/docs/doc/537299c8-cfa2-4eb5-a8b7-1f055d82abc1.json
@@ -0,0 +1,120 @@
+{
+ "summary": "The `BaseDataset` class serves as a base for creating video feature datasets, handling missing values and encoding text while supporting efficient dataset partitioning.",
+ "details": [
+ {
+ "comment": "Copyright and license information, importing necessary libraries, and type guarding.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":0-35",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport time\nimport json\nimport random\nimport paddle\nimport inspect\nimport logging\nimport functools\nimport data_loader\nimport numpy as np\nimport pickle as pkl\nfrom pathlib import Path\nfrom abc import abstractmethod\nfrom typing import Dict, Union\nfrom numpy.random import randint\nfrom typeguard import typechecked\nfrom collections import OrderedDict\nfrom zsvision.zs_utils import memcache\ntry:\n from paddlenlp.transformers import BertTokenizer\nexcept ImportError as e:\n print(\n f\"{e}, [paddlenlp] package and it's dependencies is required for T2VLAD.\""
+ },
+ {
+ "comment": "This code defines a base class `BaseDataset` for creating and loading video features dataset. It contains methods for generating required paths, performing sanity checks on loaded data, and loading features from disk. The class is abstract and requires subclass implementation of these methods. It also includes utility functions and settings like `dataset_paths`, `sanity_checks`, and `load_features`.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":36-75",
+ "content": " )\nfrom utils import ensure_tensor, expert_tensor_storage\n# For SLURM usage, buffering makes it difficult to see events as they happen, so we set\n# the global print statement to enforce flushing\nprint = functools.partial(print, flush=True)\nclass BaseDataset(paddle.io.Dataset):\n @staticmethod\n @abstractmethod\n @typechecked\n def dataset_paths() -> Dict[str, Union[Path, str]]:\n \"\"\"Generates a datastructure containing all the paths required to load features\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def sanity_checks(self):\n \"\"\"Run sanity checks on loaded data\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def load_features(self):\n \"\"\"Load features from disk\n \"\"\"\n raise NotImplementedError\n @typechecked\n def __init__(\n self,\n data_dir: Path,\n eval_only: bool,\n use_zeros_for_missing: bool,\n text_agg: str,\n text_feat: str,\n split_name: str,\n cls_partition: str,\n root_feat_folder: str,"
+ },
+ {
+ "comment": "The code above defines a class for a dataset, with various parameters such as text_dim, num_test_captions, and max_tokens. It sets the necessary attributes including logger, text_feat, data_dir, and experts. The class also initializes the tokenizer and sets the restrict_test_captions and text_features attributes before calling load_features() method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":76-100",
+ "content": " text_dim: int,\n num_test_captions: int,\n restrict_train_captions: int,\n max_tokens: Dict[str, int],\n logger: logging.Logger,\n raw_input_dims: Dict[str, int],\n feat_aggregation: Dict[str, Dict],\n ):\n self.eval_only = eval_only\n self.logger = logger\n self.text_feat = text_feat\n self.data_dir = data_dir\n self.text_dim = text_dim\n self.restrict_train_captions = restrict_train_captions\n self.max_tokens = max_tokens\n self.cls_partition = cls_partition\n self.num_test_captions = num_test_captions\n self.feat_aggregation = feat_aggregation\n self.root_feat = data_dir / root_feat_folder\n self.experts = set(raw_input_dims.keys())\n self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n # This attributes can be overloaded by different datasets, so it must be set\n # before the `load_features() method call`\n self.restrict_test_captions = None\n self.text_features = None"
+ },
+ {
+ "comment": "This code initializes class variables for a dataset object. It sets the label features, video labels, raw captions, and features to None. It loads the word2int mapping from a JSON file. The code allows for one caption per video in training minibatches. It creates an ordered list of experts based on input dimensions. The training and test lists are set by dataset-specific subclasses. The code is for retrieval tasks and uses a single dataloader, handling retrieval data separately. It sets the sample list to the training partition and calculates the total number of samples.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":101-124",
+ "content": " self.label_features = None\n self.video_labels = None\n self.raw_captions = None\n self.features = None\n self.word2int = json.load(open('word2int.json'))\n # Use a single caption per video when forming training minibatches (different\n # captions from the same video may still be used across different minibatches)\n self.captions_per_video = 1\n self.ordered_experts = list(raw_input_dims.keys())\n # Training and test lists are set by dataset-specific subclasses\n self.partition_lists = {}\n self.configure_train_test_splits(split_name=split_name)\n # All retrieval-based tasks use a single dataloader (and handle the retrieval\n # data separately), whereas for classification we use one dataloader for\n # training and one for validation.\n self.logger.info(\"The current task is retrieval\")\n self.sample_list = self.partition_lists[\"train\"]\n self.num_samples = len(self.sample_list)\n num_val = len(self.partition_lists[\"val\"])"
+ },
+ {
+ "comment": "The code sets default paths for video retrieval, defines missing value strategy based on use_zeros_for_missing argument, loads dataset-specific features into memory and averages text features when text_agg is set to \"avg\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":126-151",
+ "content": " self.raw_input_dims = raw_input_dims\n # we store default paths to enable visualisations (this can be overloaded by\n # dataset-specific classes)\n self.video_path_retrieval = [\n f\"videos/{x}.mp4\" for x in self.partition_lists[\"val\"]\n ]\n # NOTE: We use nans rather than zeros to indicate missing faces, unless we wish\n # to test single modality strength, which requires passing zeroed features for\n # missing videos\n if use_zeros_for_missing:\n self.MISSING_VAL = 0\n else:\n self.MISSING_VAL = np.nan\n # load the dataset-specific features into memory\n self.load_features()\n if text_agg == \"avg\":\n self.logger.info(\"averaging the text features...\")\n for key, val in self.text_features.items():\n self.text_features[key] = [\n np.mean(x, 0, keepdims=1) for x in val\n ]\n self.logger.info(\"finished averaging the text features\")"
+ },
+ {
+ "comment": "This code initializes training and raw configuration dictionaries, creates a tensor storage object, iterates through static experts, adds their relevant configurations to the dictionaries, and then builds a retrieval dictionary for both fixed and variable experts.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":153-174",
+ "content": " self.trn_config = {}\n self.raw_config = {}\n self.tensor_storage = expert_tensor_storage(self.experts,\n self.feat_aggregation)\n for static_expert in self.tensor_storage[\"fixed\"]:\n if static_expert in self.feat_aggregation:\n if \"trn_seg\" in self.feat_aggregation[static_expert].keys():\n self.trn_config[static_expert] = \\\n self.feat_aggregation[static_expert][\"trn_seg\"]\n if \"raw\" in self.feat_aggregation[static_expert][\"temporal\"]:\n self.raw_config[static_expert] = 1\n retrieval = {\n expert: np.zeros(\n (num_val, self.max_tokens[expert], raw_input_dims[expert]))\n for expert in self.tensor_storage[\"variable\"]\n }\n retrieval.update({\n expert: np.zeros((num_val, raw_input_dims[expert]))\n for expert in self.tensor_storage[\"fixed\"]\n })\n self.retrieval = retrieval"
+ },
+ {
+ "comment": "The code is initializing various arrays and tensors for evaluating the model on validation data. It sets up masks, retrieval tensors for text, captions, and attention, and prepares an empty list for saving the validation captions. This code is part of a larger function that appears to be setting up a dataset for video captioning or related task.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":175-196",
+ "content": " self.test_ind = {\n expert: paddle.ones([num_val])\n for expert in self.experts\n }\n self.raw_captions_retrieval = [None] * num_val\n # avoid evaluation on missing queries\n self.query_masks = np.zeros((num_val, num_test_captions))\n self.text_token_mask = np.zeros((num_val, num_test_captions))\n self.text_retrieval = np.zeros((num_val, self.num_test_captions,\n self.max_tokens[\"text\"], self.text_dim))\n self.cap_retrieval = paddle.zeros(\n [num_val, self.num_test_captions, self.max_tokens[\"text\"]],\n dtype='int64'\n ) #self.cap_retrieval = th.zeros((num_val, self.num_test_captions, self.max_tokens[\"text\"]))\n self.att_retrieval = paddle.zeros(\n [num_val, self.num_test_captions, self.max_tokens[\"text\"]],\n dtype='int64'\n ) #self.att_retrieval = th.zeros((num_val, self.num_test_captions, self.max_tokens[\"text\"]))\n save_cap = []\n for ii, video_name in enumerate(self.partition_lists[\"val\"]):"
+ },
+ {
+ "comment": "This code initializes the retrieval and test indices for each expert in both fixed and variable tensor storage. It handles missing values by replacing them with 'MISSING_VAL' and binarizing non-missing features using marker values if requested.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":198-216",
+ "content": " self.raw_captions_retrieval[ii] = self.raw_captions[video_name]\n for expert in self.tensor_storage[\"fixed\"].intersection(\n self.experts):\n feats = self.features[expert][video_name]\n drop = self.has_missing_values(feats)\n self.test_ind[expert][ii] = not drop\n self.retrieval[expert][ii] = feats\n if drop:\n self.retrieval[expert][ii][:] = self.MISSING_VAL\n if self.feat_aggregation[expert].get(\"binarise\", False):\n keep = np.logical_not(\n np.isnan(self.retrieval[expert][:, 0, 0]))\n marker = np.ones_like(self.retrieval[expert][keep])\n self.retrieval[expert][keep] = marker\n for expert in self.tensor_storage[\"variable\"].intersection(\n self.experts):\n feats = self.features[expert][video_name]\n drop = self.has_missing_values(feats)"
+ },
+ {
+ "comment": "The code is handling the process of selecting video features and test captions for a specific expert. It drops certain entries, sets missing values where needed, applies binarization if required, and limits the number of tokens based on maximum token limit. It also restricts test captions if specified by the user. Finally, it sets query masks to prepare for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":217-236",
+ "content": " self.test_ind[expert][ii] = not drop\n if drop:\n self.retrieval[expert][ii][:] = self.MISSING_VAL\n if self.feat_aggregation[expert].get(\"binarise\", False):\n keep = np.logical_not(\n np.isnan(self.retrieval[expert][:, 0, 0]))\n marker = np.ones_like(self.retrieval[expert][keep])\n self.retrieval[expert][keep] = marker\n if self.test_ind[expert][ii]:\n keep = min(self.max_tokens[expert], len(feats))\n self.retrieval[expert][ii, :keep, :] = feats[:keep]\n candidates_sentences = self.text_features[video_name]\n if self.restrict_test_captions is not None:\n keep_sent_idx = self.restrict_test_captions[video_name]\n candidates_sentences = [candidates_sentences[keep_sent_idx]]\n self.query_masks[ii, :len(candidates_sentences)] = 1\n for test_caption_idx in range(self.num_test_captions):"
+ },
+ {
+ "comment": "This code is iterating over a list of candidate sentences, breaking when the index exceeds the list length. For each sentence, it sets the number of tokens to keep based on the maximum allowed and masks the corresponding tokens. It then encodes the sentence into tokenized input IDs and attention mask for PaddlePaddle's model, appending the original sentence to a save list, storing the tokenized inputs in 'cap_retrieval', and the attention masks in 'att_retrieval'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":237-256",
+ "content": " if len(candidates_sentences) <= test_caption_idx:\n break\n keep = min(len(candidates_sentences[test_caption_idx]),\n self.max_tokens[\"text\"])\n self.text_token_mask[ii, test_caption_idx] = keep\n sent = self.raw_captions_retrieval[ii][test_caption_idx]\n sent = \" \".join(sent)\n sent = sent.strip()\n encoded_dict = self.tokenizer.__call__(\n sent,\n max_seq_len=self.max_tokens[\"text\"],\n pad_to_max_seq_len=True,\n return_attention_mask=True,\n truncation_strategy='longest_first')\n cap_ids = paddle.to_tensor(encoded_dict['input_ids'])\n attention_mask = paddle.to_tensor(\n encoded_dict['attention_mask'])\n save_cap.append(sent)\n self.cap_retrieval[ii, test_caption_idx, :] = cap_ids\n self.att_retrieval[ii, test_caption_idx, :] = attention_mask"
+ },
+ {
+ "comment": "The code is checking the progress of a dataset evaluation, creating text features for each sentence in the list, storing them in an array and then dumping the saved captions into a file called 'run_cap.pkl'. It also includes a function to configure train/test splits of the dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":257-279",
+ "content": " if ii % 500 == 0 and test_caption_idx == 0:\n msg = (\n f\"{ii}/{len(self.partition_lists['val'])} will evaluate \"\n f\"sentence {test_caption_idx} out of \"\n f\"{len(candidates_sentences)} (has {keep} words) \"\n f\"{video_name}\")\n self.logger.info(msg)\n text_feats = candidates_sentences[test_caption_idx][:keep]\n if text_feats.shape[0] == 0:\n text_feats = 0\n raise ValueError(\"empty text features!\")\n self.text_retrieval[ii, test_caption_idx, :keep, :] = text_feats\n with open('run_cap.pkl', 'wb') as f:\n pkl.dump(save_cap, f)\n self.sanity_checks()\n def configure_train_test_splits(self, split_name):\n \"\"\"Partition the datset into train/val/test splits.\n Args:\n split_name (str): the name of the split\n \"\"\"\n self.paths = type(self).dataset_paths()"
+ },
+ {
+ "comment": "The code loads training/validation splits, reads and stores them in partition lists for later use, and initializes tensor storage for the PaddleVideo application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":280-303",
+ "content": " print(\"loading training/val splits....\")\n tic = time.time()\n for subset, path in self.paths[\"subset_list_paths\"][split_name].items():\n root_feat = Path(self.root_feat)\n subset_list_path = root_feat / path\n if subset == \"train\" and self.eval_only:\n rows = []\n else:\n with open(subset_list_path) as f:\n rows = f.read().splitlines()\n self.partition_lists[subset] = rows\n print(\"done in {:.3f}s\".format(time.time() - tic))\n self.split_name = split_name\n def collate_data(self, data):\n batch_size = len(data)\n tensors = {}\n for expert in self.tensor_storage[\"fixed\"]:\n if expert in self.trn_config.keys():\n tensors[expert] = paddle.to_tensor(\n np.zeros((batch_size, self.trn_config[expert],\n self.raw_input_dims[expert])))\n else:\n tensors[expert] = paddle.to_tensor("
+ },
+ {
+ "comment": "This code initializes tensors for a batch of data in a dataset. It creates zero-initialized tensors for each expert (modality), and separate tensors for text data including token masks, cap IDs, and attention mask. These will be filled with actual data as the batch is processed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":304-326",
+ "content": " np.zeros((batch_size, self.raw_input_dims[expert])))\n # Track which indices of each modality are available in the present batch\n ind = {\n expert: paddle.to_tensor(np.zeros(batch_size))\n for expert in self.experts\n }\n tensors.update({\n expert: paddle.to_tensor(\n np.zeros((batch_size, self.max_tokens[expert],\n self.raw_input_dims[expert])))\n for expert in self.tensor_storage[\"variable\"]\n })\n text_tensor = paddle.to_tensor(\n np.zeros((batch_size, self.captions_per_video,\n self.max_tokens[\"text\"], self.text_dim)))\n text_token_mask = paddle.to_tensor(\n np.zeros((batch_size, self.captions_per_video)))\n text_cap_id = paddle.zeros([batch_size, self.max_tokens[\"text\"]],\n dtype='int64')\n text_att_mask = paddle.zeros([batch_size, self.max_tokens[\"text\"]],\n dtype='int64')"
+ },
+ {
+ "comment": "This code iterates through a dataset, extracting data for various experts and creating tensors from it. It handles missing values and stores text and mask information in separate tensors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":328-349",
+ "content": " for ii, _ in enumerate(data):\n datum = data[ii]\n for expert in self.experts:\n ind[expert][ii] = datum[f\"{expert}_ind\"]\n for expert in self.tensor_storage[\"fixed\"]:\n tensors[expert][ii] = datum[expert]\n for expert in self.tensor_storage[\"variable\"]:\n if ind[expert][ii]:\n keep = min(len(datum[expert]), self.max_tokens[expert])\n if keep:\n tensors[expert][ii, :keep, :] = datum[expert][:keep]\n else:\n tensors[expert][ii, :, :] = self.MISSING_VAL\n text = datum[\"text\"]\n cap_id = datum[\"cap_id\"]\n att_mask = datum[\"att_mask\"]\n text_cap_id[ii, :] = paddle.to_tensor(cap_id)\n text_att_mask[ii, :] = paddle.to_tensor(att_mask)\n for jj in range(self.captions_per_video):\n keep = min(len(text[jj]), self.max_tokens[\"text\"])\n text_tensor[ii, jj, :keep, :] = text[jj][:keep]"
+ },
+ {
+ "comment": "This code creates a minibatch for video features and text data. It applies binarization to some features, converts tensors, and prepares inputs for machine learning models. The process_sent function sets default values for EOS and UNK consistent with the word2int.json file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":350-371",
+ "content": " text_token_mask[ii, jj] = keep\n ind = {key: ensure_tensor(val) for key, val in ind.items()}\n experts = OrderedDict(\n (expert, paddle.to_tensor(tensors[expert], dtype='float32'))\n for expert in self.ordered_experts)\n for expert in self.experts:\n if self.feat_aggregation[expert].get(\"binarise\", False):\n replace = np.logical_not(paddle.isnan(experts[expert][:, 0, 0]))\n experts[expert][replace] = paddle.ones_like(\n experts[expert][replace])\n minibatch = {\"experts\": experts, \"ind\": ind}\n minibatch[\"text\"] = paddle.to_tensor(text_tensor, dtype='float32')\n minibatch[\"cap_id\"] = paddle.to_tensor(text_cap_id, dtype='int64')\n minibatch[\"att_mask\"] = paddle.to_tensor(text_att_mask, dtype='int64')\n minibatch[\"text_token_mask\"] = paddle.to_tensor(text_token_mask)\n return minibatch\n def process_sent(self, sent, max_words, EOS: int = 1, UNK: int = 2):\n # set EOS=1, UNK=2 by default, consistent with file 'word2int.json'."
+ },
+ {
+ "comment": "This code defines a dataset class that loads and processes video features for text-to-video retrieval. It takes a list of videos, extracts expert features, and pads them to a fixed length. The class also supports indexing and has methods for getting the number of samples in the dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":372-396",
+ "content": " tokens = [self.word2int.get(w, UNK) for w in sent]\n tokens = tokens[:max_words]\n tokens_len = len(tokens)\n tokens = np.array(tokens + [EOS] * (max_words - tokens_len))\n return tokens, tokens_len\n def __len__(self):\n return self.num_samples\n def __getitem__(self, idx):\n if idx < self.num_samples:\n vid = self.sample_list[idx]\n features = {}\n for expert in self.experts:\n if expert not in self.trn_config.keys():\n if expert in self.raw_config.keys():\n features[expert] = np.mean(self.features[expert][vid],\n axis=0)\n else:\n features[expert] = self.features[expert][vid]\n else:\n raw_frame_feats = self.features[expert][vid]\n new_length = 1\n num_frames = raw_frame_feats.shape[0]\n avg_duration = ((num_frames - new_length + 1) //"
+ },
+ {
+ "comment": "The code segments video frame features into smaller segments with a specified average duration, accounts for the last segment if the duration is not divisible by the specified interval, and ensures the number of new feature segments matches the expected number.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":397-412",
+ "content": " self.trn_config[expert])\n assert avg_duration > 0, \"average duration must be positive\"\n if avg_duration > 0:\n # maybe we could change to use average for each tiny segment\n # seems like use everything per iter\n offsets = np.multiply(\n list(range(self.trn_config[expert])), avg_duration)\n offsets += randint(avg_duration,\n size=self.trn_config[expert])\n new_frame_feats = np.zeros(\n (self.trn_config[expert], raw_frame_feats.shape[1]))\n for idx, xx in enumerate(offsets):\n new_frame_feats[idx, :] = raw_frame_feats[xx, :]\n msg = \"returning a wrong feature != segment num\"\n assert new_frame_feats.shape[0] == self.trn_config[\n expert], msg"
+ },
+ {
+ "comment": "This code is responsible for handling inconsistencies in text features storage. It randomly selects a caption from a list of captions for a given video, applies tokenization, and ensures that the sequence length does not exceed a maximum threshold. The result is stored in the encoded_dict variable.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":413-436",
+ "content": " features[expert] = new_frame_feats\n ind = {}\n for expert in self.ordered_experts:\n if expert in self.tensor_storage[\"flaky\"]:\n ind[expert] = not self.has_missing_values(features[expert])\n else:\n ind[expert] = 1\n # Handle some inconsistencies between how the text features are stored\n text = self.text_features[vid]\n if isinstance(text, list):\n pick = np.random.choice(len(text), size=self.captions_per_video)\n sent = self.raw_captions[vid][pick[0]]\n sent = \" \".join(sent)\n sent = sent.strip()\n text = np.array(text)[pick]\n encoded_dict = self.tokenizer.__call__(\n sent,\n max_seq_len=self.max_tokens[\"text\"],\n pad_to_max_seq_len=True,\n return_attention_mask=True,\n truncation_strategy='longest_first')"
+ },
+ {
+ "comment": "This code is initializing a sample for video dataset, using either given or randomly chosen text. It creates a dictionary with cap_id, attention mask, and other tensors as key-value pairs, and returns the sample. The get_retrieval_data function converts retrieval data to tensors and adds them to a dictionary containing text and experts keys before returning it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":437-462",
+ "content": " cap_id = encoded_dict['input_ids']\n token_type_ids = encoded_dict['token_type_ids']\n attention_mask = encoded_dict['attention_mask']\n else:\n pick = None\n text = np.random.choice(text, size=self.captions_per_video)\n # Return both the missing indices as well as the tensors\n sample = {\"text\": text}\n sample.update({\"cap_id\": cap_id})\n sample.update({\"att_mask\": attention_mask})\n sample.update({f\"{key}_ind\": val for key, val in ind.items()})\n sample.update(features)\n return sample\n def get_retrieval_data(self):\n experts = OrderedDict(\n (expert, paddle.to_tensor(self.retrieval[expert], dtype='float32'))\n for expert in self.ordered_experts)\n retrieval_data = {\n \"text\":\n paddle.to_tensor(ensure_tensor(self.text_retrieval),\n dtype='float32'),\n \"experts\":\n experts,\n \"cap_id\":"
+ },
+ {
+ "comment": "The function defines a dictionary 'retrieval_data' containing cap_retrieval, att_mask, test_ind, and text_token_mask. It also defines the 'meta' dictionary containing query_masks, raw_captions, and paths. The function returns both 'retrieval_data' and 'meta'. The code provides a path lookup for visual features and skips loading if the feature is not requested.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":463-491",
+ "content": " paddle.to_tensor(self.cap_retrieval, dtype='int64'),\n \"att_mask\":\n paddle.to_tensor(self.att_retrieval, dtype='int64'),\n \"ind\":\n self.test_ind,\n \"text_token_mask\":\n paddle.to_tensor(self.text_token_mask)\n }\n meta = {\n \"query_masks\": self.query_masks,\n \"raw_captions\": self.raw_captions_retrieval,\n \"paths\": self.video_path_retrieval,\n }\n return retrieval_data, meta\n def has_missing_values(self, x):\n return isinstance(x, float) and np.isnan(x)\n def visual_feat_paths(self, model_spec, tag=None):\n \"\"\"Canonical path lookup for visual features\n \"\"\"\n if model_spec not in self.ordered_experts:\n self.logger.info(\n f\"Skipping load for {model_spec} (feature not requested)\")\n return f\"SKIPPED-{model_spec}\"\n feat_type, model_name, _ = model_spec.split(\".\")\n aggs = self.feat_aggregation[model_spec]\n base = f\"aggregated_{feat_type.replace('-', '_')}\""
+ },
+ {
+ "comment": "The code defines a function that generates feature paths based on the provided arguments. It assembles a base string with parameters like fps, pixel_dim, and stride. If the feature type is \"facecrops\" or \"faceboxes\", it includes those parameters in the base string. For other types except for \"ocr\", \"speech\", and \"audio\", it also includes those parameters in the base string. It then adds optional parameters like offset and inner_stride if present. Finally, it generates a feature path list with file names and appends the tag if provided. The function also defines a logging assertion function that writes assertions to logs using a recipe from an external link.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":492-515",
+ "content": " required = (\"fps\", \"pixel_dim\", \"stride\")\n fps, pixel_dim, stride = [aggs.get(x, None) for x in required]\n if feat_type in {\"facecrops\", \"faceboxes\"}:\n base = f\"{base}_{fps}fps_{pixel_dim}px_stride{stride}\"\n elif feat_type not in {\"ocr\", \"speech\", \"audio\"}:\n base = f\"{base}_{fps}fps_{pixel_dim}px_stride{stride}\"\n for option in \"offset\", \"inner_stride\":\n if aggs.get(option, None) is not None:\n base += f\"_{option}{aggs[option]}\"\n feat_paths = []\n for agg in aggs[\"temporal\"].split(\"-\"):\n fname = f\"{model_name}-{agg}\"\n if aggs[\"type\"] == \"logits\":\n fname = f\"{fname}-logits\"\n if tag is not None:\n fname += f\"-{tag}\"\n feat_paths.append(Path(base) / f\"{fname}.pickle\")\n return feat_paths\n def log_assert(self, bool_, msg=\"\", verbose=True):\n \"\"\"Use assertions that will be written to the logs. This is a recipe from:\n http://code.activestate.com/recipes/577074-logging-asserts/"
+ },
+ {
+ "comment": "The code snippet is a function that checks an assertion. If the assertion fails, it constructs an exception message containing the traceback from the calling frame and raises an AssertionError with this message. Another function called \"summary_stats\" reports basic statistics about feature availability and variable lengths across different data subsets.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":516-538",
+ "content": " \"\"\"\n try:\n assert bool_, msg\n except AssertionError:\n # construct an exception message from the code of the calling frame\n last_stackframe = inspect.stack()[-2]\n source_file, line_no, func = last_stackframe[1:4]\n source = f\"Traceback (most recent call last):\\n\" + \\\n f\" File {source_file}, line {line_no}, in {func}\\n\"\n if verbose:\n # include more lines than that where the statement was made\n source_code = open(source_file).readlines()\n source += \"\".join(source_code[line_no - 3:line_no + 1])\n else:\n source += last_stackframe[-2][0].strip()\n self.logger.debug(f\"{msg}\\n{source}\")\n raise AssertionError(f\"{msg}\\n{source}\")\n def summary_stats(self):\n \"\"\"Report basic statistics about feature availability and variable lengths\n across the different subsets of the data.\n \"\"\"\n self.logger.info(\"Computing feature stats...\")"
+ },
+ {
+ "comment": "This code partitions datasets based on predefined subsets and checks the sizes of the features. It prints a summary for each subset, counting missing values and displaying the minimum, maximum, and mean sizes of features. This ensures that the dataset is properly partitioned and allows for efficient analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/base/base_dataset.py\":539-561",
+ "content": " queries = self.ordered_experts + [\"text\"]\n for subset, keep in self.partition_lists.items():\n keep = set(keep)\n print(f\"Summary for {subset}\")\n for expert in queries:\n if expert in self.features:\n feats = self.features[expert]\n else:\n feats = self.text_features\n vals = [feats[key] for key in keep]\n missing = 0\n sizes = []\n for val in vals:\n if self.has_missing_values(val):\n missing += 1\n else:\n sizes.append(len(val))\n if sizes:\n stat_str = (f\"min: {np.min(sizes):4}, \"\n f\"max: {np.max(sizes):4}, \"\n f\"mean: {np.mean(sizes):.1f}\")\n print(\n f\"{subset}: missing: {missing:4}, {stat_str} {expert}\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/548fa608-fac1-4cf7-8d94-aa8296587051.json b/docs/doc/548fa608-fac1-4cf7-8d94-aa8296587051.json
new file mode 100644
index 000000000..29654b97c
--- /dev/null
+++ b/docs/doc/548fa608-fac1-4cf7-8d94-aa8296587051.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code introduces the PP-TSM, a high-performance and efficient video recognition model optimized based on TSM in PaddleVideo. It outlines various strategies like ImageNet pretraining, data augmentation, and optimizer improvements to enhance performance and achieve fast inference speed on V101 GPU with top-1 accuracy on UCF101 and Kinetics400 datasets.",
+ "details": [
+ {
+ "comment": "This code describes the PP-TSM, a high-performance and efficient video recognition model optimized based on TSM in PaddleVideo. It mentions its better performance and inference speed compared to TSM paper and other open source TSM models. Requires PaddlePaddle2.0 for execution. When using ImageNet for pretrain and 8X1 sample, it achieves high top-1 accuracy on UCF101 and Kinetics400 datasets with fast inference speed on V100 GPU.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/pp-tsm.md\":0-21",
+ "content": "# High performance recognition 2D architecture PP-TSM\nPP-TSM\uff1aAn Effective and Efficient video-recognition model \nPP-TSM is an optimized model based on TSM in PaddleVideo, \nwhose performance (top-1 on UCF101 and Kinetics400) and inference spped \nare better than TSM paper(https://arxiv.org/abs/1811.08383 ) and \nother open source TSM\uff0cPaddlePaddle2.0(available on pip now) or \nDaily Version( https://www.paddlepaddle.org.cn/documentation/docs/zh/install/Tables.html#whl-dev ) \nis required to run PP-TSM. \nWhen only use ImageNet for pretrain and only use 8X1 sample\uff0c \nPP-TSM\u2019s top1 reached to 89.5% and 73.5% on UCF101 and Kinetics400, \nand inference speed of FP32 on single V100 is 147 VPS on Kinectics400 dataset. \ninference speed of FP16 with TensorRT on single V100 isTODO. \nAs far as we know, under the same conditions, \ntop1=73.5% on Kinetics400 is the best performance for 2D video model until now. \nPP-TSM improved performance and speed of TSM with following methods: \n1\u3001Model Tweaks: ResNet50vd \uff0c+2.5% "
+ },
+ {
+ "comment": "This code outlines several strategies implemented to improve the performance of a model, including ImageNet pretraining, better batch size and L2 values, label smoothing, better learning rate decay, data augmentation, and updated epoch numbers. The code also mentions using Knowledge Distillation, optimizer improvements, and plans for integrating PaddleInference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/pp-tsm.md\":22-31",
+ "content": "2\u3001ImageNet pretrain weights based on Knowledge Distillation \uff0c +1.3% \n3\u3001beter batch size \uff0c+0.2% \n4\u3001beter L2 \uff0c+0.3% \n5\u3001label_smoothing \uff0c+0.2% \n6\u3001beter lr decay \uff0c+0.15% \n7\u3001Data augmentation \uff0c+0.3% \n8\u3001beter epoch num \uff0c+0.15% \n9\u3001bn strategy \uff0c+0.4% \n10\u3001integrated PaddleInference \n11\u3001more strategies todo: Knowledge Distillation\u3001optimizer and so on. "
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/54c8f988-1d1e-473c-8ad0-7b86dabb0428.json b/docs/doc/54c8f988-1d1e-473c-8ad0-7b86dabb0428.json
new file mode 100644
index 000000000..98e05ce55
--- /dev/null
+++ b/docs/doc/54c8f988-1d1e-473c-8ad0-7b86dabb0428.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a base class for multimodal models in PaddleVideo, requiring subclasses to override train_step, valid_step, test_step, and define abstract methods for validating, testing, and inference steps.",
+ "details": [
+ {
+ "comment": "This code defines a base class for multimodal models in PaddleVideo. It requires subclasses to override train_step, valid_step, and test_step methods. The constructor accepts optional backbone, head, and loss parameters which are built using the builder module. If provided, the backbone is initialized with its init_weights method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/multimodal/base.py\":0-31",
+ "content": "from abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseMultimodal(nn.Layer):\n \"\"\"Base class for Multimodal.\n All Multimodal model should subclass it.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Head to process feature.\n loss(dict): Loss function.\n \"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__()\n if backbone is not None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n if head is not None:\n self.head_name = head.name\n self.head = builder.build_head(head)"
+ },
+ {
+ "comment": "The code defines a base class for multimodal models, with an initializer to set up the head and loss functions. The `forward` method selects the appropriate step function based on the given mode (train, valid, test, or infer). The abstract `train_step` method must be implemented in subclasses for training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/multimodal/base.py\":32-62",
+ "content": " if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n if loss is not None:\n self.loss = builder.build_loss(loss)\n else:\n self.loss = None\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError"
+ },
+ {
+ "comment": "This code defines three abstract methods: val_step, test_step, and infer_step. These methods represent validating, testing, and inference steps respectively. The methods are not yet implemented and will need to be filled by subclasses according to the specific requirements of the model being developed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/multimodal/base.py\":64-80",
+ "content": " @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\"\n raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/54e499bb-46a2-451a-98ec-6b736112b2db.json b/docs/doc/54e499bb-46a2-451a-98ec-6b736112b2db.json
new file mode 100644
index 000000000..77ddba084
--- /dev/null
+++ b/docs/doc/54e499bb-46a2-451a-98ec-6b736112b2db.json
@@ -0,0 +1,130 @@
+{
+ "summary": "The code prepares for DAVIS2017 image processing, initializes variables, and utilizes PaddlePaddle for video object detection. It involves an interactive image classification system with 8 turns, optimizing scribble labels and filtering keys.",
+ "details": [
+ {
+ "comment": "The code imports necessary libraries and defines functions for processing image data from the DAVIS2017 dataset. It sets up the required transforms, initializes the network models, and reads in the dataset sequences.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":0-38",
+ "content": "import cv2\nimport os\nimport json\nimport paddle\nfrom PIL import Image\nimport timeit\nimport numpy as np\nfrom paddle.vision import transforms\nfrom dataloaders.davis_2017_f import DAVIS2017_Feature_Extract\nimport dataloaders.custom_transforms_f as tr\nfrom davisinteractive.session import DavisInteractiveSession\nfrom networks.deeplab import DeepLab\nfrom networks.IntVOS import IntVOS\nimport time\nfrom davisinteractive.utils.scribbles import scribbles2mask, annotated_frames\nfrom config import cfg\nfrom paddle import nn\nfrom paddle.io import DataLoader\nfrom utils.api import float_, byte_\n@paddle.no_grad()\ndef main():\n paddle.set_device(\"gpu:0\")\n total_frame_num_dic = {}\n #################\n seqs = []\n with open(os.path.join(cfg.DATA_ROOT, 'ImageSets', '2017',\n 'val' + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))\n seqs.extend(seqs_tmp)\n h_w_dic = {}\n for seq_name in seqs:\n images = np.sort(\n os.listdir("
+ },
+ {
+ "comment": "This code reads a configuration file and initializes variables for video analysis. It loads image information from disk, checks if an existing imgnum dictionary is available, and if not, it populates it by iterating through directories and counting images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":39-61",
+ "content": " os.path.join(cfg.DATA_ROOT, 'JPEGImages/480p/',\n seq_name.strip())))\n total_frame_num_dic[seq_name] = len(images)\n im_ = cv2.imread(\n os.path.join(cfg.DATA_ROOT, 'JPEGImages/480p/', seq_name,\n '00000.jpg'))\n im_ = np.array(im_, dtype=np.float32)\n hh_, ww_ = im_.shape[:2]\n h_w_dic[seq_name] = (hh_, ww_)\n _seq_list_file = os.path.join(cfg.DATA_ROOT, 'ImageSets', '2017',\n 'v_a_l' + '_instances.txt')\n seq_dict = json.load(open(_seq_list_file, 'r'))\n ##################\n seq_imgnum_dict_ = {}\n seq_imgnum_dict = os.path.join(cfg.DATA_ROOT, 'ImageSets', '2017',\n 'val_imgnum.txt')\n if os.path.isfile(seq_imgnum_dict):\n seq_imgnum_dict_ = json.load(open(seq_imgnum_dict, 'r'))\n else:\n for seq in os.listdir(os.path.join(cfg.DATA_ROOT, 'JPEGImages/480p/')):\n seq_imgnum_dict_[seq] = len(\n os.listdir(os.path.join(cfg.DATA_ROOT, 'JPEGImages/480p/',"
+ },
+ {
+ "comment": "Creating a dictionary of image numbers and saving it, setting save flags for predicted masks, checking if results directory exists and creating it if not, defining maximum interactive parameters, importing DeepLab model and Instant VOS model, and loading the saved model from specified location.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":62-86",
+ "content": " seq)))\n with open(seq_imgnum_dict, 'w') as f:\n json.dump(seq_imgnum_dict_, f)\n ##################\n is_save_image = False # Save the predicted masks\n report_save_dir = cfg.RESULT_ROOT\n save_res_dir = cfg.SAVE_RESULT_DIR # changed to path\n if not os.path.exists(cfg.RESULT_ROOT):\n os.makedirs(cfg.RESULT_ROOT)\n # Configuration used in the challenges\n max_nb_interactions = 8 # Maximum number of interactions\n max_time_per_interaction = 30 # Maximum time per interaction per object\n # Total time available to interact with a sequence and an initial set of scribbles\n max_time = max_nb_interactions * max_time_per_interaction # Maximum time per object\n # Interactive parameters\n subset = 'val'\n host = 'localhost' # 'localhost' for subsets train and val.\n feature_extracter = DeepLab(backbone='resnet', freeze_bn=False)\n model = IntVOS(cfg, feature_extracter)\n print('model loading...')\n saved_model_dict = save_res_dir"
+ },
+ {
+ "comment": "This code loads a pre-trained model, evaluates it, and initializes variables for processing scribbles. The code also defines a transform to resize images to specific dimensions, opens a file for writing, and sets up a DavisInteractiveSession object for iterating over interaction data. The session will continue until there are no more interactions left in the dataset.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":87-112",
+ "content": " pretrained_dict = paddle.load(saved_model_dict)\n load_network(model, pretrained_dict)\n print(f'model loading from {saved_model_dict} finished!')\n model.eval()\n inter_file = open(os.path.join(cfg.RESULT_ROOT, 'inter_file.txt'), 'w')\n resized_h, resized_w = 480, 854\n ###############################\n composed_transforms = transforms.Compose(\n [tr.Resize((resized_h, resized_w)),\n tr.ToTensor()])\n ###############################\n seen_seq = []\n n = 0\n max_n = 1\n with DavisInteractiveSession(host=host,\n davis_root=cfg.DATA_ROOT,\n subset=subset,\n report_save_dir=report_save_dir,\n max_nb_interactions=max_nb_interactions,\n max_time=max_time,\n metric_to_optimize='J') as sess:\n while sess.next():\n t_total = timeit.default_timer()\n # Get the current iteration scribbles"
+ },
+ {
+ "comment": "The code is retrieving scribbles and their corresponding sequence, image dimensions are assigned based on the dictionary h_w_dic. If there are no annotated frames from the scribbles, it returns previous masks and submits them. Otherwise, it initializes memories for the first round.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":114-138",
+ "content": " sequence, scribbles, first_scribble = sess.get_scribbles(\n only_last=True)\n h, w = h_w_dic[sequence]\n if 'prev_label_storage' not in locals().keys():\n prev_label_storage = paddle.zeros(\n [104, h, w]) # because the maximum length of frames is 104.\n print(sequence)\n h, w = h_w_dic[sequence]\n if len(\n annotated_frames(scribbles)\n ) == 0: # if no scribbles return, keep masks in previous round\n final_masks = prev_label_storage[:seq_imgnum_dict_[sequence]]\n sess.submit_masks(final_masks.numpy())\n else:\n start_annotated_frame = annotated_frames(scribbles)[0]\n pred_masks = []\n pred_masks_reverse = []\n if first_scribble: # If in the first round, initialize memories\n n_interaction = 1\n eval_global_map_tmp_dic = {}\n local_map_dics = ({}, {})"
+ },
+ {
+ "comment": "This code is part of an interaction detection process. It writes the interaction details to a file, including the sequence name, type, and frame number. It also checks if the sequence has been seen before and prepares embedding memory for reference image processing. The code uses DAVIS2017_Feature_Extract to extract pixel embeddings in the first round of annotations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":139-162",
+ "content": " total_frame_num = total_frame_num_dic[sequence]\n obj_nums = seq_dict[sequence][-1]\n else:\n n_interaction += 1\n ##\n inter_file.write(sequence + ' ' + 'interaction' +\n str(n_interaction) + ' ' + 'frame' +\n str(start_annotated_frame) + '\\n')\n ##\n ##########################Reference image process\n if first_scribble: # if in the first round, extract pixel embbedings.\n if sequence not in seen_seq:\n inter_turn = 1\n seen_seq.append(sequence)\n embedding_memory = []\n test_dataset = DAVIS2017_Feature_Extract(\n root=cfg.DATA_ROOT,\n transform=composed_transforms,\n seq_name=sequence)\n testloader = DataLoader(test_dataset,"
+ },
+ {
+ "comment": "This code is iterating through testloader and extracting frame embeddings for each image. The extracted embeddings are then concatenated to form a single embedding memory. If annotated frames are present, the reference frame embedding is extracted from the embedding memory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":163-181",
+ "content": " batch_size=14,\n shuffle=False,\n num_workers=cfg.NUM_WORKER)\n for ii, sample in enumerate(testloader):\n imgs = sample['img1']\n frame_embedding = model.extract_feature(imgs)\n embedding_memory.append(frame_embedding)\n del frame_embedding\n embedding_memory = paddle.concat(embedding_memory, 0)\n _, _, emb_h, emb_w = embedding_memory.shape\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n else:\n inter_turn += 1\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]"
+ },
+ {
+ "comment": "The code applies scribbles to an image using a mask and generates corresponding labels. It then creates a scribble sample and converts it into a tensor. If is_save_image is True, the scribble label image is saved as a PALETTE image.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":182-202",
+ "content": " ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n else:\n ref_frame_embedding = embedding_memory[\n start_annotated_frame]\n ref_frame_embedding = ref_frame_embedding.unsqueeze(0)\n ########\n scribble_masks = scribbles2mask(scribbles, (emb_h, emb_w))\n scribble_label = scribble_masks[start_annotated_frame]\n scribble_sample = {'scribble_label': scribble_label}\n scribble_sample = tr.ToTensor()(scribble_sample)\n # print(ref_frame_embedding, ref_frame_embedding.shape)\n scribble_label = scribble_sample['scribble_label']\n scribble_label = scribble_label.unsqueeze(0)\n ######\n if is_save_image:\n ref_scribble_to_show = scribble_label.squeeze().numpy()\n im_ = Image.fromarray(\n ref_scribble_to_show.astype('uint8')).convert('P', )"
+ },
+ {
+ "comment": "This code segment saves a scribble image with the palette applied to a specific directory path based on input parameters. It first checks if the necessary directory exists and creates it if it doesn't, then proceeds to save the image within this directory. The 'first_scribble' variable is used in decision making further down the code.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":203-223",
+ "content": " im_.putpalette(_palette)\n ref_img_name = str(start_annotated_frame)\n if not os.path.exists(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn))):\n os.makedirs(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn)))\n im_.save(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn),\n 'inter_' + ref_img_name + '.png'))\n scribble_label = scribble_label\n #######\n if first_scribble:"
+ },
+ {
+ "comment": "This code snippet initializes variables and checks for specific conditions in a segmentation model. It handles the previous label and label storage, updates them based on certain conditions, and submits the final masks to the session if necessary. The interaction segmentation head is printed as a comment.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":225-243",
+ "content": " prev_label = None\n prev_label_storage = paddle.zeros([104, h, w])\n prev_label_storage = prev_label_storage\n else:\n prev_label = prev_label_storage[start_annotated_frame]\n prev_label = prev_label.unsqueeze(0).unsqueeze(0)\n if not first_scribble and paddle.unique(\n scribble_label).shape[0] == 1:\n final_masks = prev_label_storage[:\n seq_imgnum_dict_[sequence]]\n sess.submit_masks(final_masks.numpy())\n else: ###inteaction segmentation head\n print('inteaction segmentation head')\n tmp_dic, local_map_dics = model.int_seghead(\n ref_frame_embedding=ref_frame_embedding,\n ref_scribble_label=scribble_label,\n prev_round_label=prev_label,\n global_map_tmp_dic=eval_global_map_tmp_dic,"
+ },
+ {
+ "comment": "This code snippet is part of a PaddleVideo application called Ma-Net, which seems to be related to object detection and video analysis. The code creates a dictionary for input data, retrieves the predicted label from a temporal dictionary, applies interpolation to resize the label, selects the maximum value along an axis, appends the mask to a list of masks, stores the previous label at a specific frame, and saves the predicted label as a numpy array if needed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":244-261",
+ "content": " local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n frame_num=[start_annotated_frame],\n first_inter=first_scribble)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks.append(float_(pred_label))\n prev_label_storage[start_annotated_frame] = float_(\n pred_label[0])\n if is_save_image: # save image\n pred_label_to_save = pred_label.squeeze(0).numpy()"
+ },
+ {
+ "comment": "This code snippet saves an interactive video frame as a labeled image. It first converts the predicted label to an array, then converts it into a format suitable for saving as an image with the 'P' mode and using a specific palette. The filename is created based on the current frame number and if the directory doesn't exist, it creates one. Finally, the labeled image is saved in the specified directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":262-278",
+ "content": " im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert('P', )\n im.putpalette(_palette)\n imgname = str(start_annotated_frame)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn))):\n os.makedirs(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn)))\n im.save(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),"
+ },
+ {
+ "comment": "This code is part of a video object detection algorithm. It's using a pre-trained model to generate segmentation masks for each frame in the video. The 'turn' and 'imgname' are used as file names for saving images. It applies initial scribble_label if it's the first frame, then it updates previous label and embedding for propagating prediction to next frames. It uses the model's prop_seghead function to generate predictions for each frame's segmentation mask.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":279-297",
+ "content": " 'turn' + str(inter_turn),\n imgname + '.png'))\n #######################################\n if first_scribble:\n scribble_label = rough_ROI(scribble_label)\n ##############################\n ref_prev_label = pred_label.unsqueeze(0)\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = ref_frame_embedding\n #### Propagation ->\n for ii in range(start_annotated_frame + 1, total_frame_num):\n current_embedding = embedding_memory[ii]\n current_embedding = current_embedding.unsqueeze(0)\n prev_label = prev_label\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = model.prop_seghead(\n ref_frame_embedding,\n prev_embedding,\n current_embedding,"
+ },
+ {
+ "comment": "Code snippet is calling a function, passing several arguments such as scribble_label, prev_label, and others. It assigns the returned value to pred_label after applying interpolation on it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":298-317",
+ "content": " scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg.KNNS,\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,\n frame_num=[ii],\n dynamic_seghead=model.dynamic_seghead)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(\n pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)"
+ },
+ {
+ "comment": "Code snippet handles image saving for each prediction label in a loop, storing the prediction label as a numpy array and converting it to an image using Pillow library. It then sets the palette and saves the image with a sequence number and interaction number, creating directories if they don't exist.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":319-337",
+ "content": " pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks.append(float_(pred_label))\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = current_embedding\n prev_label_storage[ii] = float_(pred_label[0])\n ####\n if is_save_image:\n pred_label_to_save = pred_label.squeeze(0).numpy()\n im = Image.fromarray(\n pred_label_to_save.astype('uint8')).convert(\n 'P', )\n im.putpalette(_palette)\n imgname = str(ii)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(\n os.path.join(\n cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),"
+ },
+ {
+ "comment": "Code creates folders and saves image, then resets variables for frame propagation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":338-355",
+ "content": " 'turn' + str(inter_turn))):\n os.makedirs(\n os.path.join(\n cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn)))\n im.save(\n os.path.join(cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn),\n imgname + '.png'))\n #######################################\n prev_label = ref_prev_label\n prev_embedding = ref_frame_embedding\n #######\n # Propagation <-\n for ii in range(start_annotated_frame):\n current_frame_num = start_annotated_frame - 1 - ii"
+ },
+ {
+ "comment": "This code section is using PaddlePaddle, a machine learning framework. It seems to be part of an object detection model for video sequences. The function 'model.prop_seghead' is being called with multiple embeddings and labels, and it returns three outputs (tmp_dic, eval_global_map_tmp_dic, local_map_dics) based on the input parameters. Normalization and nearest neighbor distances are used in this process as well. The 'cfg.KNS' likely refers to a pre-defined constant or configuration related to k-nearest neighbors (kNN). Finally, 'n_interaction', 'start_annotated_frame' variables represent interaction numbers and starting frame for annotated frames, respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":356-373",
+ "content": " current_embedding = embedding_memory[current_frame_num]\n current_embedding = current_embedding.unsqueeze(0)\n prev_label = prev_label\n tmp_dic, eval_global_map_tmp_dic, local_map_dics = model.prop_seghead(\n ref_frame_embedding,\n prev_embedding,\n current_embedding,\n scribble_label,\n prev_label,\n normalize_nearest_neighbor_distances=True,\n use_local_map=True,\n seq_names=[sequence],\n gt_ids=paddle.to_tensor([obj_nums]),\n k_nearest_neighbors=cfg.KNNS,\n global_map_tmp_dic=eval_global_map_tmp_dic,\n local_map_dics=local_map_dics,\n interaction_num=n_interaction,\n start_annotated_frame=start_annotated_frame,"
+ },
+ {
+ "comment": "This code appears to be part of a larger function or script. It seems to involve image processing and potentially object detection or classification. The code is looping through frames of an input video, extracting features and predictions from a model, interpolating the predictions for size consistency, then appending the predicted labels (or masks) to a list. It also stores the last prediction for each frame and optionally saves one of those predictions as an image. The code appears to be part of an object detection or classification task where it is updating the output based on new frames and previous frames' outputs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":374-393",
+ "content": " frame_num=[current_frame_num],\n dynamic_seghead=model.dynamic_seghead)\n pred_label = tmp_dic[sequence]\n pred_label = nn.functional.interpolate(\n pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_masks_reverse.append(float_(pred_label))\n prev_label = pred_label.unsqueeze(0)\n prev_embedding = current_embedding\n ####\n prev_label_storage[current_frame_num] = float_(\n pred_label[0])\n ###\n if is_save_image:\n pred_label_to_save = pred_label.squeeze(0).numpy()\n im = Image.fromarray("
+ },
+ {
+ "comment": "This code saves the predicted label as an image in a specific directory structure based on the current frame number, sequence name, and interaction turn. It ensures that the directory for the given combination of parameters exists before saving the image.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":394-411",
+ "content": " pred_label_to_save.astype('uint8')).convert(\n 'P', )\n im.putpalette(_palette)\n imgname = str(current_frame_num)\n while len(imgname) < 5:\n imgname = '0' + imgname\n if not os.path.exists(\n os.path.join(\n cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn))):\n os.makedirs(\n os.path.join(\n cfg.RESULT_ROOT, sequence,\n 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn)))\n im.save(\n os.path.join(cfg.RESULT_ROOT, sequence,"
+ },
+ {
+ "comment": "This code appears to be part of an interactive image classification system. The code is submitting masks for each turn and interacts up to 8 times, storing the results in memory and then clearing them after completion. At the end, it prints the total time taken for a single interaction and gets the report and summary from the session. The rough_ROI function seems to calculate distances based on input labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":412-435",
+ "content": " 'interactive' + str(n_interaction),\n 'turn' + str(inter_turn),\n imgname + '.png'))\n pred_masks_reverse.reverse()\n pred_masks_reverse.extend(pred_masks)\n final_masks = paddle.concat(pred_masks_reverse, 0)\n sess.submit_masks(final_masks.numpy())\n if inter_turn == 3 and n_interaction == 8:\n del eval_global_map_tmp_dic\n del local_map_dics\n del embedding_memory\n del prev_label_storage\n t_end = timeit.default_timer()\n print('Total time for single interaction: ' + str(t_end - t_total))\n report = sess.get_report()\n summary = sess.get_global_summary(\n save_file=os.path.join(report_save_dir, 'summary.json'))\n inter_file.close()\ndef rough_ROI(ref_scribble_labels):\n dist = 20\n b, _, h, w = ref_scribble_labels.shape"
+ },
+ {
+ "comment": "The code is applying a filter to refine the scribble labels, where it creates a filter based on the position of non-background pixels and then applies it to the original scribble labels. The function load_network filters out unnecessary keys from pretrained_dict and overwrites entries in the state dict of the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":436-467",
+ "content": " filter_ = paddle.zeros_like(ref_scribble_labels)\n to_fill = paddle.zeros_like(ref_scribble_labels)\n for i in range(b):\n no_background = (ref_scribble_labels[i] != -1)\n no_background = no_background.squeeze(0)\n no_b = no_background.nonzero()\n (h_min, w_min) = paddle.min(no_b, 0)\n (h_max, w_max) = paddle.max(no_b, 0)\n filter_[i, 0,\n max(h_min - dist, 0):min(h_max + dist, h - 1),\n max(w_min - dist, 0):min(w_max + dist, w - 1)] = 1\n final_scribble_labels = paddle.where(byte_(filter_), ref_scribble_labels,\n to_fill)\n return final_scribble_labels\ndef load_network(net, pretrained_dict):\n model_dict = net.state_dict()\n # 1. filter out unnecessary keys\n f_pretrained_dict = {}\n for k, v in pretrained_dict.items():\n if k in model_dict:\n f_pretrained_dict[k] = v\n else:\n print(k)\n print(len(model_dict.keys()), len(pretrained_dict.keys()))\n # 2. overwrite entries in the existing state dict"
+ },
+ {
+ "comment": "This code defines a palette with RGB values for 75 different colors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":468-484",
+ "content": " model_dict.update(pretrained_dict)\n net.set_state_dict(model_dict)\n_palette = [\n 0, 0, 0, 128, 0, 0, 0, 128, 0, 128, 128, 0, 0, 0, 128, 128, 0, 128, 0, 128,\n 128, 128, 128, 128, 64, 0, 0, 191, 0, 0, 64, 128, 0, 191, 128, 0, 64, 0,\n 128, 191, 0, 128, 64, 128, 128, 191, 128, 128, 0, 64, 0, 128, 64, 0, 0, 191,\n 0, 128, 191, 0, 0, 64, 128, 128, 64, 128, 22, 22, 22, 23, 23, 23, 24, 24,\n 24, 25, 25, 25, 26, 26, 26, 27, 27, 27, 28, 28, 28, 29, 29, 29, 30, 30, 30,\n 31, 31, 31, 32, 32, 32, 33, 33, 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 37,\n 37, 37, 38, 38, 38, 39, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 43, 43,\n 43, 44, 44, 44, 45, 45, 45, 46, 46, 46, 47, 47, 47, 48, 48, 48, 49, 49, 49,\n 50, 50, 50, 51, 51, 51, 52, 52, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56,\n 56, 56, 57, 57, 57, 58, 58, 58, 59, 59, 59, 60, 60, 60, 61, 61, 61, 62, 62,\n 62, 63, 63, 63, 64, 64, 64, 65, 65, 65, 66, 66, 66, 67, 67, 67, 68, 68, 68,\n 69, 69, 69, 70, 70, 70, 71, 71, 71, 72, 72, 72, 73, 73, 73, 74, 74, 74, 75,"
+ },
+ {
+ "comment": "This code appears to be a sequence of numbers, which could potentially represent a list or array in the code.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":485-497",
+ "content": " 75, 75, 76, 76, 76, 77, 77, 77, 78, 78, 78, 79, 79, 79, 80, 80, 80, 81, 81,\n 81, 82, 82, 82, 83, 83, 83, 84, 84, 84, 85, 85, 85, 86, 86, 86, 87, 87, 87,\n 88, 88, 88, 89, 89, 89, 90, 90, 90, 91, 91, 91, 92, 92, 92, 93, 93, 93, 94,\n 94, 94, 95, 95, 95, 96, 96, 96, 97, 97, 97, 98, 98, 98, 99, 99, 99, 100,\n 100, 100, 101, 101, 101, 102, 102, 102, 103, 103, 103, 104, 104, 104, 105,\n 105, 105, 106, 106, 106, 107, 107, 107, 108, 108, 108, 109, 109, 109, 110,\n 110, 110, 111, 111, 111, 112, 112, 112, 113, 113, 113, 114, 114, 114, 115,\n 115, 115, 116, 116, 116, 117, 117, 117, 118, 118, 118, 119, 119, 119, 120,\n 120, 120, 121, 121, 121, 122, 122, 122, 123, 123, 123, 124, 124, 124, 125,\n 125, 125, 126, 126, 126, 127, 127, 127, 128, 128, 128, 129, 129, 129, 130,\n 130, 130, 131, 131, 131, 132, 132, 132, 133, 133, 133, 134, 134, 134, 135,\n 135, 135, 136, 136, 136, 137, 137, 137, 138, 138, 138, 139, 139, 139, 140,\n 140, 140, 141, 141, 141, 142, 142, 142, 143, 143, 143, 144, 144, 144, 145,"
+ },
+ {
+ "comment": "This code contains 210 consecutive numbers, possibly representing the iteration or indexing in a loop.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":498-510",
+ "content": " 145, 145, 146, 146, 146, 147, 147, 147, 148, 148, 148, 149, 149, 149, 150,\n 150, 150, 151, 151, 151, 152, 152, 152, 153, 153, 153, 154, 154, 154, 155,\n 155, 155, 156, 156, 156, 157, 157, 157, 158, 158, 158, 159, 159, 159, 160,\n 160, 160, 161, 161, 161, 162, 162, 162, 163, 163, 163, 164, 164, 164, 165,\n 165, 165, 166, 166, 166, 167, 167, 167, 168, 168, 168, 169, 169, 169, 170,\n 170, 170, 171, 171, 171, 172, 172, 172, 173, 173, 173, 174, 174, 174, 175,\n 175, 175, 176, 176, 176, 177, 177, 177, 178, 178, 178, 179, 179, 179, 180,\n 180, 180, 181, 181, 181, 182, 182, 182, 183, 183, 183, 184, 184, 184, 185,\n 185, 185, 186, 186, 186, 187, 187, 187, 188, 188, 188, 189, 189, 189, 190,\n 190, 190, 191, 191, 191, 192, 192, 192, 193, 193, 193, 194, 194, 194, 195,\n 195, 195, 196, 196, 196, 197, 197, 197, 198, 198, 198, 199, 199, 199, 200,\n 200, 200, 201, 201, 201, 202, 202, 202, 203, 203, 203, 204, 204, 204, 205,\n 205, 205, 206, 206, 206, 207, 207, 207, 208, 208, 208, 209, 209, 209, 210,"
+ },
+ {
+ "comment": "The code consists of a sequence of integers. It's not executable and doesn't have any apparent function or variable assignments. The specific use case or purpose of these numbers is unclear without further context.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/test.py\":511-524",
+ "content": " 210, 210, 211, 211, 211, 212, 212, 212, 213, 213, 213, 214, 214, 214, 215,\n 215, 215, 216, 216, 216, 217, 217, 217, 218, 218, 218, 219, 219, 219, 220,\n 220, 220, 221, 221, 221, 222, 222, 222, 223, 223, 223, 224, 224, 224, 225,\n 225, 225, 226, 226, 226, 227, 227, 227, 228, 228, 228, 229, 229, 229, 230,\n 230, 230, 231, 231, 231, 232, 232, 232, 233, 233, 233, 234, 234, 234, 235,\n 235, 235, 236, 236, 236, 237, 237, 237, 238, 238, 238, 239, 239, 239, 240,\n 240, 240, 241, 241, 241, 242, 242, 242, 243, 243, 243, 244, 244, 244, 245,\n 245, 245, 246, 246, 246, 247, 247, 247, 248, 248, 248, 249, 249, 249, 250,\n 250, 250, 251, 251, 251, 252, 252, 252, 253, 253, 253, 254, 254, 254, 255,\n 255, 255\n]\nif __name__ == '__main__':\n main()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5538fc80-7b52-4b10-b854-88f8952bc1e4.json b/docs/doc/5538fc80-7b52-4b10-b854-88f8952bc1e4.json
new file mode 100644
index 000000000..7cd1668d2
--- /dev/null
+++ b/docs/doc/5538fc80-7b52-4b10-b854-88f8952bc1e4.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The code retrieves data, applies NMS to bounding box proposals, filters detected actions from videos using NMS, and stores relevant information in the \"video_results\" list. It defines a function `get_action_result` that takes inputs and performs NMS on processed results.",
+ "details": [
+ {
+ "comment": "This code defines two functions: `get_data_res` and `base_nms`. The first function takes in a label map, data (a list of features), and a topk value. It iterates through each video in the data, extracts relevant information from the feature, and appends this information to a new list called `video_result`. Finally, it returns the `video_result` list. The second function is an incomplete definition for a non-maximum suppression algorithm used for bounding boxes. It takes in bboxes (bounding box coordinates), thresh (threshold value), delta (optional parameter with default value 0), and nms_id (an identifier for the NMS operation, with a default value of 2).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/utils/process_result.py\":0-38",
+ "content": "\"\"\"\n# @File : process_result.py \n# @Author: macaihong\n# @Date : 2019/12/15\n# @Desc :\n\"\"\"\nimport sys\nimport os\nimport re\nimport numpy as np\nimport pickle\nimport json\nimport logger\nlogger = logger.Logger()\ndef get_data_res(label_map, data, topk):\n \"\"\"get_data_res\"\"\"\n sum_vid = len(data)\n video_result = []\n for i in range(sum_vid):\n vid_name = data[i][0][0]\n # true_label predict_start predict_end predict_score predict_len gt_iou gt_start gt_ioa\n feature_start_id = float(data[i][0][1]['start'])\n feature_end_id = float(data[i][0][1]['end'])\n feature_stage1_score = data[i][0][1]['score']\n predict_res = []\n for k in range(topk):\n score_top = data[i][1][k]\n labelid_top = data[i][2][k]\n label_iou = data[i][3]\n labelname_top = label_map[str(labelid_top)]\n video_result.append([feature_start_id, feature_end_id, labelid_top, labelname_top, score_top, label_iou])\n return video_result\ndef base_nms(bboxes, thresh, delta=0, nms_id=2):"
+ },
+ {
+ "comment": "This code performs non-maximal suppression on bounding box proposals. It filters out overlapping boxes by keeping only those with the highest scores and discarding the rest. The function process_proposal takes source bounding box proposals, applies non-maximal suppression with a threshold, and returns the filtered results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/utils/process_result.py\":39-75",
+ "content": " \"\"\"\n One-dimensional non-maximal suppression\n :param bboxes: [[vid, label, st, ed, score, ...], ...]\n :param thresh:\n :return:\n \"\"\"\n \"\"\"\n t1 = bboxes[:, 0]\n t2 = bboxes[:, 1]\n scores = bboxes[:, nms_id]\n \"\"\"\n t1 = np.array([max(0, x[0] - delta) for x in bboxes])\n t2 = np.array([x[1] + delta for x in bboxes])\n scores = np.array([x[nms_id] for x in bboxes])\n durations = t2 - t1\n order = scores.argsort()[::-1]\n keep = []\n while order.size > 0:\n i = order[0]\n keep.append(i)\n tt1 = np.maximum(t1[i], t1[order[1:]])\n tt2 = np.minimum(t2[i], t2[order[1:]])\n intersection = tt2 - tt1\n IoU = intersection / (durations[i] + durations[order[1:]] - intersection).astype(float)\n inds = np.where(IoU <= thresh)[0]\n order = order[inds + 1]\n return [bboxes[i] for i in keep]\ndef process_proposal(source_prop_box, min_frame_thread=5, nms_thresh=0.7, score_thresh=0.01):\n \"\"\"process_video_prop\"\"\"\n prop_box = []\n for items in source_prop_box:"
+ },
+ {
+ "comment": "This code is part of a video classification process. It filters and sorts the detected actions in a video, discarding background or weak detections. The results are stored in 'prop_res' and 'video_results'. The code applies non-maximum suppression (NMS) to filter and sort the detections based on frame duration, score threshold, and other parameters like fps, nms_thread, and nms_delta.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/utils/process_result.py\":76-106",
+ "content": " start_frame = float(items[0])\n end_frame = float(items[1])\n score = float(items[2])\n if end_frame - start_frame < min_frame_thread or score < score_thresh:\n continue\n prop_box.append([start_frame, end_frame, score])\n prop_box_keep = base_nms(prop_box, nms_thresh)\n prop_res = []\n for res in prop_box_keep:\n prop_res.append({'start': res[0], 'end': res[1], 'score': res[2]})\n return prop_res\ndef process_video_classify(video_prop, fps, score_thread, iou_thread, \\\n nms_id=5, nms_thread=0.01, nms_delta=10, backgroundid=0):\n \"\"\"process_video_classify\"\"\"\n prop_filter = []\n for item in video_prop:\n if item[2] == backgroundid:\n continue\n prop_filter.append(item)\n # prop_filter = sorted(prop_filter, key=lambda x: x[nms_id], reverse=True)\n prop_filter = base_nms(prop_filter, nms_thread, nms_delta, nms_id)\n prop_filter = sorted(prop_filter, key=lambda x: x[0])\n video_results = []\n for item in prop_filter:"
+ },
+ {
+ "comment": "This code calculates the start and end time in seconds, frame IDs, and other relevant details of detected actions from a video. It then appends these details as a dictionary to the \"video_results\" list if the classify score and IoU score exceed certain thresholds.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/utils/process_result.py\":107-128",
+ "content": " start_sec = item[0] / fps\n end_sec = item[1] / fps\n start_id_frame = item[0]\n end_id_frame = item[1]\n # start_time = \"%02d:%02d:%02d\" % ((start_id_frame / fps) / 3600, \\\n # ((start_id_frame / fps) % 3600) / 60, (start_id_frame / fps) % 60)\n # end_time = \"%02d:%02d:%02d\" % ((end_id_frame / fps) / 3600, \\\n # ((end_id_frame / fps) % 3600) / 60, (end_id_frame / fps) % 60)\n start_time = int(start_id_frame / fps)\n end_time = int(end_id_frame / fps)\n label_id = item[2]\n label_name = item[3]\n label_classify_score = item[4]\n label_iou_score = item[5]\n if label_classify_score > score_thread and label_iou_score > iou_thread:\n video_results.append({\"start_time\": start_time,\n \"end_time\": end_time,\n \"label_id\": label_id,\n \"label_name\": label_name,\n \"classify_score\": label_classify_score,"
+ },
+ {
+ "comment": "This code defines a function `get_action_result` that takes in `result_info`, `label_map_file`, `fps`, `score_thread`, `iou_thread`, `nms_id`, `nms_thread`, and `frame_offset` as inputs. It reads the label map from `label_map_file`, processes the result data using `get_data_res` function, performs non-maximum suppression (NMS) on the processed results with specified parameters, and returns the final NMS results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/utils/process_result.py\":129-143",
+ "content": " \"iou_score\": label_iou_score})\n return video_results\ndef get_action_result(result_info, label_map_file, fps, score_thread=0, \\\n iou_thread=0, nms_id=5, nms_thread=0.01, frame_offset=10, topk=1):\n \"\"\"get_action_result\"\"\"\n label_map = json.load(open(label_map_file, 'r', encoding='utf-8'))\n org_result = get_data_res(label_map, result_info, topk)\n nms_result = process_video_classify(org_result, fps, score_thread, iou_thread, nms_id, nms_thread, frame_offset)\n return nms_result"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/56c474d4-4253-4559-ad0f-108c809f42ab.json b/docs/doc/56c474d4-4253-4559-ad0f-108c809f42ab.json
new file mode 100644
index 000000000..398484f0b
--- /dev/null
+++ b/docs/doc/56c474d4-4253-4559-ad0f-108c809f42ab.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The code defines BMNINFReader class for reading and processing BMN model data, includes get_sw_prop function generating proposals, filters less than one-second proposals, performs calculations, and creates a reader class to load video data for training or prediction.",
+ "details": [
+ {
+ "comment": "This code is defining a class BMNINFReader, which extends DataReader and provides functionality for reading data from BMN model. It includes a function get_sw_prop, which generates proposals of a specific window size and step over a given duration. The class also filters out any proposals that are less than one second long.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/bmninf_reader.py\":0-48",
+ "content": "\"\"\"\n# @File : bmninf_reader.py \n# @Author: macaihong\n# @Date : 2019/12/15\n# @Desc :\n\"\"\"\nimport os\nimport random\nimport pickle\nimport json\nimport numpy as np\nimport multiprocessing\nimport numpy as np\nfrom .reader_utils import DataReader\ndef get_sw_prop(duration, window=200, step=10):\n \"\"\"\n get_sw_prop\n \"\"\"\n pr = []\n local_boxes = []\n for k in np.arange(0, duration - window + step, step):\n start_id = k\n end_id = min(duration, k + window)\n if end_id - start_id < window:\n start_id = end_id - window\n local_boxes = (start_id, end_id)\n pr.append(local_boxes)\n def valid_proposal(duration, span):\n \"\"\"\n valid_proposal\n \"\"\"\n # fileter proposals\n # a valid proposal should have at least one second in the video\n real_span = min(duration, span[1]) - span[0]\n return real_span >= 1\n pr = list(filter(lambda x: valid_proposal(duration, x), pr))\n return pr\nclass BMNINFReader(DataReader):\n \"\"\"\n Data reader for BMN model, which was stored as features extracted by prior networks"
+ },
+ {
+ "comment": "This code initializes a class that reads BMNINF data. It takes arguments for name, mode, and configuration (cfg). The tscale and dscale are set from the config file. The tgap, step, image_feature, and pcm_feature variables are calculated and reshaped accordingly. Minimum length is found to ensure both features have same length.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/bmninf_reader.py\":49-72",
+ "content": " dataset cfg: feat_path, feature path,\n tscale, temporal length of BM map,\n dscale, duration scale of BM map,\n anchor_xmin, anchor_xmax, the range of each point in the feature sequence,\n batch_size, batch size of input data,\n num_threads, number of threads of data processing\n \"\"\"\n def __init__(self, name, mode, cfg, material=None):\n self.name = name\n self.mode = mode\n self.tscale = cfg[self.name.upper()]['tscale'] # 200\n self.dscale = cfg[self.name.upper()]['dscale'] # 200\n self.tgap = 1. / self.tscale\n self.step = cfg[self.name.upper()]['window_step']\n self.material = material\n src_feature = self.material\n image_feature = src_feature['image_feature']\n pcm_feature = src_feature['pcm_feature']\n pcm_feature = pcm_feature.reshape((pcm_feature.shape[0] * 5, 640))\n min_length = min(image_feature.shape[0], pcm_feature.shape[0])\n image_feature = image_feature[:min_length, :]"
+ },
+ {
+ "comment": "This code defines a class with methods for getting dataset dictionary and match map. It takes configuration file as input, extracts relevant features from images and pcm data, sets batch size and number of threads based on mode, and creates video list and match map using duration, window size, step, and gap values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/bmninf_reader.py\":73-104",
+ "content": " pcm_feature = pcm_feature[:min_length, :]\n self.features = np.concatenate((image_feature, pcm_feature), axis=1)\n self.duration = len(self.features)\n self.window = self.tscale\n self.get_dataset_dict()\n self.get_match_map()\n self.batch_size = cfg[self.name.upper()]['batch_size']\n if (mode == 'test') or (mode == 'infer'):\n self.num_threads = 1 # set num_threads as 1 for test and infer\n def get_dataset_dict(self):\n \"\"\"\n get_dataset_dict\n \"\"\"\n self.video_list = get_sw_prop(self.duration, self.window, self.step)\n def get_match_map(self):\n \"\"\"\n get_match_map\n \"\"\"\n match_map = []\n for idx in range(self.tscale):\n tmp_match_window = []\n xmin = self.tgap * idx\n for jdx in range(1, self.tscale + 1):\n xmax = xmin + self.tgap * jdx\n tmp_match_window.append([xmin, xmax])\n match_map.append(tmp_match_window)\n match_map = np.array(match_map)"
+ },
+ {
+ "comment": "The code is a reader for BMNINF files. It transposes, reshapes, and stores match_map data. The load_file function loads video features based on start and end feature IDs. The create_reader function creates an inferencer reader. Finally, the make_infer_reader function returns a reader for inference tasks that iterates through video windows.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/bmninf_reader.py\":105-140",
+ "content": " match_map = np.transpose(match_map, [1, 0, 2])\n match_map = np.reshape(match_map, [-1, 2])\n self.match_map = match_map\n self.anchor_xmin = [self.tgap * i for i in range(self.tscale)]\n self.anchor_xmax = [self.tgap * i for i in range(1, self.tscale + 1)]\n def load_file(self, video_wind):\n \"\"\"\n load_file\n \"\"\"\n start_feat_id = video_wind[0]\n end_feat_id = video_wind[1]\n video_feat = self.features[video_wind[0]: video_wind[1]]\n video_feat = video_feat.T\n video_feat = video_feat.astype(\"float32\")\n return video_feat\n def create_reader(self):\n \"\"\"\n reader creator for ctcn model\n \"\"\"\n return self.make_infer_reader()\n def make_infer_reader(self):\n \"\"\"\n reader for inference\n \"\"\"\n def reader():\n \"\"\"\n reader\n \"\"\"\n batch_out = []\n # for video_name in self.video_list:\n for video_wind in self.video_list:\n video_idx = self.video_list.index(video_wind)"
+ },
+ {
+ "comment": "This code defines a reader class that loads and processes video data, creating batches of features for model training or prediction. It uses the `load_file` method to read video files and appends them to the `batch_out` list. When the list reaches the specified `batch_size`, it yields the batch and resets the list. If there are remaining items in the list upon exiting the function, it yields those final batches.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/bmninf_reader.py\":141-150",
+ "content": " video_feat = self.load_file(video_wind)\n batch_out.append((video_feat, video_wind, [self.duration, self.dscale]))\n if len(batch_out) == self.batch_size:\n yield batch_out\n batch_out = []\n if len(batch_out) > 0:\n yield batch_out\n return reader"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/577fb8a8-7e54-4187-8ddc-dd58514de87f.json b/docs/doc/577fb8a8-7e54-4187-8ddc-dd58514de87f.json
new file mode 100644
index 000000000..c0303d80b
--- /dev/null
+++ b/docs/doc/577fb8a8-7e54-4187-8ddc-dd58514de87f.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code deploys a PaddlePaddle model for serving using PaddleServing in PaddleVideo, supporting GPU and CPU installations on Linux platforms. Input/output variables are set for Python serving, and the RPC method is used for prediction. Results are displayed in cmd window.",
+ "details": [
+ {
+ "comment": "This code provides instructions on how to deploy a model service using PaddleServing in the PaddleVideo platform. It starts by explaining that this deployment example uses an HTTP prediction server and is currently only supported on Linux platforms. The instructions then cover how to install Serving, specifying steps for both GPU-accelerated docker installation and CPU-only docker installation, as well as installing the necessary Python packages.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme.md\":0-31",
+ "content": "\u7b80\u4f53\u4e2d\u6587 | [English](./readme_en.md)\n# \u6a21\u578b\u670d\u52a1\u5316\u90e8\u7f72\n## \u7b80\u4ecb\n[Paddle Serving](https://github.com/PaddlePaddle/Serving) \u65e8\u5728\u5e2e\u52a9\u6df1\u5ea6\u5b66\u4e60\u5f00\u53d1\u8005\u8f7b\u677e\u90e8\u7f72\u5728\u7ebf\u9884\u6d4b\u670d\u52a1\uff0c\u652f\u6301\u4e00\u952e\u90e8\u7f72\u5de5\u4e1a\u7ea7\u7684\u670d\u52a1\u80fd\u529b\u3001\u5ba2\u6237\u7aef\u548c\u670d\u52a1\u7aef\u4e4b\u95f4\u9ad8\u5e76\u53d1\u548c\u9ad8\u6548\u901a\u4fe1\u3001\u5e76\u652f\u6301\u591a\u79cd\u7f16\u7a0b\u8bed\u8a00\u5f00\u53d1\u5ba2\u6237\u7aef\u3002\n\u8be5\u90e8\u5206\u4ee5 HTTP \u9884\u6d4b\u670d\u52a1\u90e8\u7f72\u4e3a\u4f8b\uff0c\u4ecb\u7ecd\u600e\u6837\u5728 PaddleVideo \u4e2d\u4f7f\u7528 PaddleServing \u90e8\u7f72\u6a21\u578b\u670d\u52a1\u3002\u76ee\u524d\u53ea\u652f\u6301 Linux \u5e73\u53f0\u90e8\u7f72\uff0c\u6682\u4e0d\u652f\u6301 Windows \u5e73\u53f0\u3002\n## Serving \u5b89\u88c5\nServing \u5b98\u7f51\u63a8\u8350\u4f7f\u7528 docker \u5b89\u88c5\u5e76\u90e8\u7f72 Serving \u73af\u5883\u3002\u9996\u5148\u9700\u8981\u62c9\u53d6 docker \u73af\u5883\u5e76\u521b\u5efa\u57fa\u4e8e Serving \u7684 docker\u3002\n```bash\n# \u542f\u52a8GPU docker\ndocker pull paddlepaddle/serving:0.7.0-cuda10.2-cudnn7-devel\nnvidia-docker run -p 9292:9292 --name test -dit paddlepaddle/serving:0.7.0-cuda10.2-cudnn7-devel bash\nnvidia-docker exec -it test bash\n# \u542f\u52a8CPU docker\ndocker pull paddlepaddle/serving:0.7.0-devel\ndocker run -p 9292:9292 --name test -dit paddlepaddle/serving:0.7.0-devel bash\ndocker exec -it test bash\n```\n\u8fdb\u5165 docker \u540e\uff0c\u9700\u8981\u5b89\u88c5 Serving \u76f8\u5173\u7684 python \u5305\u3002\n```bash\npython3.7 -m pip install paddle-serving-client==0.7.0\npython3.7 -m pip install paddle-serving-app==0.7.0\npython3.7 -m pip install faiss-cpu==1.7.1post2\n#\u82e5\u4e3aCPU\u90e8\u7f72\u73af\u5883:\npython3.7 -m pip install paddle-serving-server==0.7.0 # CPU"
+ },
+ {
+ "comment": "Install PaddlePaddle for CPU and GPU environments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme.md\":32-57",
+ "content": "python3.7 -m pip install paddlepaddle==2.2.0 # CPU\n#\u82e5\u4e3aGPU\u90e8\u7f72\u73af\u5883\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post102 # GPU with CUDA10.2 + TensorRT6\npython3.7 -m pip install paddlepaddle-gpu==2.2.0 # GPU with CUDA10.2\n#\u5176\u4ed6GPU\u73af\u5883\u9700\u8981\u786e\u8ba4\u73af\u5883\u518d\u9009\u62e9\u6267\u884c\u54ea\u4e00\u6761\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post101 # GPU with CUDA10.1 + TensorRT6\npython3.7 -m pip install paddle-serving-server-gpu==0.7.0.post112 # GPU with CUDA11.2 + TensorRT8\n```\n* \u5982\u679c\u5b89\u88c5\u901f\u5ea6\u592a\u6162\uff0c\u53ef\u4ee5\u901a\u8fc7 `-i https://pypi.tuna.tsinghua.edu.cn/simple` \u66f4\u6362\u6e90\uff0c\u52a0\u901f\u5b89\u88c5\u8fc7\u7a0b\n* \u66f4\u591a\u73af\u5883\u548c\u5bf9\u5e94\u7684\u5b89\u88c5\u5305\u8be6\u89c1\uff1ahttps://github.com/PaddlePaddle/Serving/blob/v0.9.0/doc/Install_Linux_Env_CN.md\n## \u884c\u4e3a\u8bc6\u522b\u670d\u52a1\u90e8\u7f72\n### \u6a21\u578b\u8f6c\u6362\n\u4f7f\u7528 PaddleServing \u505a\u670d\u52a1\u5316\u90e8\u7f72\u65f6\uff0c\u9700\u8981\u5c06\u4fdd\u5b58\u7684 inference \u6a21\u578b\u8f6c\u6362\u4e3a Serving \u6a21\u578b\u3002\u4e0b\u9762\u4ee5 PP-TSM \u6a21\u578b\u4e3a\u4f8b\uff0c\u4ecb\u7ecd\u5982\u4f55\u90e8\u7f72\u884c\u4e3a\u8bc6\u522b\u670d\u52a1\u3002\n- \u4e0b\u8f7d\u8bad\u7ec3\u597d\u7684 PP-TSM \u7684\u6a21\u578b\uff0c\u5e76\u8f6c\u5316\u4e3a\u63a8\u7406\u6a21\u578b\uff1a\n ```bash\n # \u8fdb\u5165PaddleVideo\u76ee\u5f55\n cd PaddleVideo\n wget -P data/ https://videotag.bj.bcebos.com/PaddleVideo-release2.1/PPTSM/ppTSM_k400_uniform.pdparams\n python3.7 tools/export_model.py \\\n -c configs/recognition/pptsm/pptsm_k400_frames_uniform.yaml \\"
+ },
+ {
+ "comment": "This code is converting a pre-trained PaddlePaddle model to a format suitable for serving on the server. It downloads and unzips the pre-trained model, then uses paddle_serving_client to convert it into the correct format for deployment with specified directories for serving server and client. The `dirname` specifies where the pre-trained model files are stored, while `model_filename` names the Inference Program structure file, defaulting to \"__model__\" if not specified.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme.md\":58-82",
+ "content": " -p data/ppTSM_k400_uniform.pdparams \\\n -o inference/ppTSM\n ```\n- \u6211\u4eec\u4e5f\u63d0\u4f9b\u4e86\u8f6c\u6362\u597d\u7684\u63a8\u7406\u6a21\u578b\uff0c\u6309\u4ee5\u4e0b\u547d\u4ee4\u4e0b\u8f7d\u5e76\u89e3\u538b\n ```bash\n mkdir ./inference\n wget -nc -P ./inference https://videotag.bj.bcebos.com/PaddleVideo-release2.3/ppTSM.zip --no-check-certificate\n pushd ./inference\n unzip ppTSM.zip\n popd\n ```\n- \u7528 paddle_serving_client \u628a\u8f6c\u6362\u597d\u7684\u63a8\u7406\u6a21\u578b\u518d\u8f6c\u6362\u6210\u6613\u4e8e Server \u90e8\u7f72\u7684\u6a21\u578b\u683c\u5f0f\uff1a\n ```bash\n python3.7 -m paddle_serving_client.convert \\\n --dirname inference/ppTSM \\\n --model_filename ppTSM.pdmodel \\\n --params_filename ppTSM.pdiparams \\\n --serving_server ./deploy/python_serving/ppTSM_serving_server/ \\\n --serving_client ./deploy/python_serving/ppTSM_serving_client/\n ```\n | \u53c2\u6570 | \u7c7b\u578b | \u9ed8\u8ba4\u503c | \u63cf\u8ff0 |\n | ----------------- | ---- | ------------------ | ------------------------------------------------------------ |\n | `dirname` | str | - | \u9700\u8981\u8f6c\u6362\u7684\u6a21\u578b\u6587\u4ef6\u5b58\u50a8\u8def\u5f84\uff0cProgram\u7ed3\u6784\u6587\u4ef6\u548c\u53c2\u6570\u6587\u4ef6\u5747\u4fdd\u5b58\u5728\u6b64\u76ee\u5f55\u3002 |\n | `model_filename` | str | None | \u5b58\u50a8\u9700\u8981\u8f6c\u6362\u7684\u6a21\u578bInference Program\u7ed3\u6784\u7684\u6587\u4ef6\u540d\u79f0\u3002\u5982\u679c\u8bbe\u7f6e\u4e3aNone\uff0c\u5219\u4f7f\u7528 `__model__` \u4f5c\u4e3a\u9ed8\u8ba4\u7684\u6587\u4ef6\u540d |"
+ },
+ {
+ "comment": "The code provides parameters for the PP-TSM model transformation, including a parameter file name (params_filename), and paths to store the converted model files (serving_server) and client configuration files (serving_client). The resulting files will be organized in separate folders (ppTSM_serving_server and ppTSM_serving_client), with specific formats. The alias names 'outputs' must be set for both fetch_var in serving_server_conf.prototxt to ensure compatibility and easy deployment of different models without modifying the code.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme.md\":83-102",
+ "content": " | `params_filename` | str | None | \u5b58\u50a8\u9700\u8981\u8f6c\u6362\u7684\u6a21\u578b\u6240\u6709\u53c2\u6570\u7684\u6587\u4ef6\u540d\u79f0\u3002\u5f53\u4e14\u4ec5\u5f53\u6240\u6709\u6a21\u578b\u53c2\u6570\u88ab\u4fdd>\u5b58\u5728\u4e00\u4e2a\u5355\u72ec\u7684\u4e8c\u8fdb\u5236\u6587\u4ef6\u4e2d\uff0c\u5b83\u624d\u9700\u8981\u88ab\u6307\u5b9a\u3002\u5982\u679c\u6a21\u578b\u53c2\u6570\u662f\u5b58\u50a8\u5728\u5404\u81ea\u5206\u79bb\u7684\u6587\u4ef6\u4e2d\uff0c\u8bbe\u7f6e\u5b83\u7684\u503c\u4e3aNone |\n | `serving_server` | str | `\"serving_server\"` | \u8f6c\u6362\u540e\u7684\u6a21\u578b\u6587\u4ef6\u548c\u914d\u7f6e\u6587\u4ef6\u7684\u5b58\u50a8\u8def\u5f84\u3002\u9ed8\u8ba4\u503c\u4e3aserving_server |\n | `serving_client` | str | `\"serving_client\"` | \u8f6c\u6362\u540e\u7684\u5ba2\u6237\u7aef\u914d\u7f6e\u6587\u4ef6\u5b58\u50a8\u8def\u5f84\u3002\u9ed8\u8ba4\u503c\u4e3aserving_client |\nPP-TSM \u63a8\u7406\u6a21\u578b\u8f6c\u6362\u5b8c\u6210\u540e\uff0c\u4f1a\u5728\u5f53\u524d\u6587\u4ef6\u5939\u591a\u51fa `ppTSM_serving_server` \u548c `ppTSM_serving_client` \u7684\u6587\u4ef6\u5939\uff0c\u5177\u5907\u5982\u4e0b\u683c\u5f0f\uff1a\n ```bash\n PaddleVideo/deploy/python_serving\n \u251c\u2500\u2500 ppTSM_serving_server\n \u251c\u2500\u2500 ppTSM.pdiparams\n \u251c\u2500\u2500 ppTSM.pdmodel\n \u251c\u2500\u2500 serving_server_conf.prototxt\n \u2514\u2500\u2500 serving_server_conf.stream.prototxt\n \u251c\u2500\u2500 ppTSM_serving_client\n \u251c\u2500\u2500 serving_client_conf.prototxt\n \u2514\u2500\u2500 serving_client_conf.stream.prototxt\n ```\n\u5f97\u5230\u6a21\u578b\u6587\u4ef6\u4e4b\u540e\uff0c\u9700\u8981\u5206\u522b\u4fee\u6539 `ppTSM_serving_server` \u548c `ppTSM_serving_client` \u4e0b\u7684\u6587\u4ef6 `serving_server_conf.prototxt`\uff0c\u5c06 \u4e24\u4efd\u6587\u4ef6\u4e2d`fetch_var` \u4e0b\u7684 `alias_name` \u5747\u6539\u4e3a `outputs`\n**\u5907\u6ce8**: Serving \u4e3a\u4e86\u517c\u5bb9\u4e0d\u540c\u6a21\u578b\u7684\u90e8\u7f72\uff0c\u63d0\u4f9b\u4e86\u8f93\u5165\u8f93\u51fa\u91cd\u547d\u540d\u7684\u529f\u80fd\u3002\u8fd9\u6837\uff0c\u4e0d\u540c\u7684\u6a21\u578b\u5728\u63a8\u7406\u90e8\u7f72\u65f6\uff0c\u53ea\u9700\u8981\u4fee\u6539\u914d\u7f6e\u6587\u4ef6\u7684`alias_name`\u5373\u53ef\uff0c\u65e0\u9700\u4fee\u6539\u4ee3\u7801\u5373\u53ef\u5b8c\u6210\u63a8\u7406\u90e8\u7f72\u3002\n\u4fee\u6539\u540e\u7684`serving_server_conf.prototxt`\u5982\u4e0b\u6240\u793a:"
+ },
+ {
+ "comment": "The code represents the configuration for input (\"feed_var\") and output (\"fetch_var\") variables in the PaddleVideo deployment's Python serving. The input variable has a shape of 8,3,224,224 and the output variable has a shape of 400. This code is part of the setup process for sending prediction requests to the PaddleVideo pipeline service using either HTTP or RPC methods.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme.md\":104-151",
+ "content": "```yaml\nfeed_var {\n name: \"data_batch_0\"\n alias_name: \"data_batch_0\"\n is_lod_tensor: false\n feed_type: 1\n shape: 8\n shape: 3\n shape: 224\n shape: 224\n}\nfetch_var {\n name: \"linear_2.tmp_1\"\n alias_name: \"outputs\"\n is_lod_tensor: false\n fetch_type: 1\n shape: 400\n}\n```\n### \u670d\u52a1\u90e8\u7f72\u548c\u8bf7\u6c42\n`python_serving` \u76ee\u5f55\u5305\u542b\u4e86\u542f\u52a8 pipeline \u670d\u52a1\u3001C++ serving\u670d\u52a1(TODO)\u548c\u53d1\u9001\u9884\u6d4b\u8bf7\u6c42\u7684\u4ee3\u7801\uff0c\u5177\u4f53\u5305\u62ec\uff1a\n```bash\n__init__.py\nconfigs/xxx.yaml # \u542f\u52a8pipeline\u670d\u52a1\u7684\u914d\u7f6e\u6587\u4ef6\npipeline_http_client.py # http\u65b9\u5f0f\u53d1\u9001pipeline\u9884\u6d4b\u8bf7\u6c42\u7684python\u811a\u672c\npipeline_rpc_client.py # rpc\u65b9\u5f0f\u53d1\u9001pipeline\u9884\u6d4b\u8bf7\u6c42\u7684python\u811a\u672c\nrecognition_web_service.py # \u542f\u52a8pipeline\u670d\u52a1\u7aef\u7684python\u811a\u672c\nutils.py # \u50a8\u5b58\u9884\u6d4b\u8fc7\u7a0b\u4e2d\u5e38\u7528\u7684\u51fd\u6570\uff0c\u5982parse_file_paths, numpy_to_base64, video_to_numpy\n```\n#### Python Serving\n- \u8fdb\u5165\u5de5\u4f5c\u76ee\u5f55\uff1a\n```bash\ncd deploy/python_serving\n```\n- \u542f\u52a8\u670d\u52a1\uff1a\n```bash\n# \u5728\u5f53\u524d\u547d\u4ee4\u884c\u7a97\u53e3\u542f\u52a8\u5e76\u4fdd\u6301\u5728\u524d\u7aef\npython3.7 recognition_web_service.py -n PPTSM -c configs/PP-TSM.yaml\n# \u5728\u540e\u53f0\u542f\u52a8\uff0c\u8fc7\u7a0b\u4e2d\u6253\u5370\u8f93\u51fa\u7684\u65e5\u5fd7\u4f1a\u91cd\u5b9a\u5411\u4fdd\u5b58\u5230log.txt\u4e2d\npython3.7 recognition_web_service.py -n PPTSM -c configs/PP-TSM.yaml &>log.txt &\n```\n- \u53d1\u9001\u8bf7\u6c42\uff1a\n```bash\n# \u4ee5http\u65b9\u5f0f\u7684\u53d1\u9001\u9884\u6d4b\u8bf7\u6c42\u5e76\u63a5\u53d7\u7ed3\u679c\npython3.7 pipeline_http_client.py -i ../../data/example.avi"
+ },
+ {
+ "comment": "This code demonstrates running the PaddleVideo model for prediction using the RPC (Remote Procedure Call) method. The command \"python3.7 pipeline_rpc_client.py -i ../../data/example.avi\" is used to execute the prediction, and the results are printed in the cmd window.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/readme.md\":153-184",
+ "content": "# \u4ee5rpc\u65b9\u5f0f\u7684\u53d1\u9001\u9884\u6d4b\u8bf7\u6c42\u5e76\u63a5\u53d7\u7ed3\u679c\npython3.7 pipeline_rpc_client.py -i ../../data/example.avi\n```\n\u6210\u529f\u8fd0\u884c\u540e\uff0c\u6a21\u578b\u9884\u6d4b\u7684\u7ed3\u679c\u4f1a\u6253\u5370\u5728 cmd \u7a97\u53e3\u4e2d\uff0c\u7ed3\u679c\u5982\u4e0b\uff1a\n```bash\n# http\u65b9\u5f0f\u6253\u5370\u7684\u7ed3\u679c\n{'err_no': 0, 'err_msg': '', 'key': ['label', 'prob'], 'value': [\"['archery']\", '[0.9907388687133789]'], 'tensors': []}\n# rpc\u65b9\u5f0f\u6253\u5370\u7684\u7ed3\u679c\nPipelineClient::predict pack_data time:1645631086.764019\nPipelineClient::predict before time:1645631086.8485317\nkey: \"label\"\nkey: \"prob\"\nvalue: \"[\\'archery\\']\"\nvalue: \"[0.9907388687133789]\"\n```\n## FAQ\n**Q1**\uff1a \u53d1\u9001\u8bf7\u6c42\u540e\u6ca1\u6709\u7ed3\u679c\u8fd4\u56de\u6216\u8005\u63d0\u793a\u8f93\u51fa\u89e3\u7801\u62a5\u9519\n**A1**\uff1a \u542f\u52a8\u670d\u52a1\u548c\u53d1\u9001\u8bf7\u6c42\u65f6\u4e0d\u8981\u8bbe\u7f6e\u4ee3\u7406\uff0c\u53ef\u4ee5\u5728\u542f\u52a8\u670d\u52a1\u524d\u548c\u53d1\u9001\u8bf7\u6c42\u524d\u5173\u95ed\u4ee3\u7406\uff0c\u5173\u95ed\u4ee3\u7406\u7684\u547d\u4ee4\u662f\uff1a\n```\nunset https_proxy\nunset http_proxy\n```\n**Q2**\uff1a \u670d\u52a1\u7aef\u542f\u52a8\u540e\u6ca1\u6709\u53cd\u5e94\uff0c\u4e00\u76f4\u505c\u5728`start proxy service`\u4e0d\u52a8\n**A2**\uff1a \u5f88\u53ef\u80fd\u662f\u542f\u52a8\u8fc7\u7a0b\u4e2d\u9047\u5230\u4e86\u95ee\u9898\uff0c\u53ef\u4ee5\u5728`./deploy/python_serving/PipelineServingLogs/pipeline.log`\u65e5\u5fd7\u6587\u4ef6\u4e2d\u67e5\u770b\u8be6\u7ec6\u62a5\u9519\u4fe1\u606f\n\u66f4\u591a\u7684\u670d\u52a1\u90e8\u7f72\u7c7b\u578b\uff0c\u5982 `RPC \u9884\u6d4b\u670d\u52a1` \u7b49\uff0c\u53ef\u4ee5\u53c2\u8003 Serving \u7684[github \u5b98\u7f51](https://github.com/PaddlePaddle/Serving/tree/v0.7.0/examples)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5797c0a4-1a31-4b45-bb26-5a35b4cf2ce2.json b/docs/doc/5797c0a4-1a31-4b45-bb26-5a35b4cf2ce2.json
new file mode 100644
index 000000000..709d7f6d6
--- /dev/null
+++ b/docs/doc/5797c0a4-1a31-4b45-bb26-5a35b4cf2ce2.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The CTRGCNHead class is a neural network head for the CTR-GCN model in PaddleVideo library, containing layers initialization, weight initialization, and forward pass function definition. The ctrgcn_head class returns the result of passing input x through a fully connected layer (fc) for feature processing and prediction.",
+ "details": [
+ {
+ "comment": "This code snippet is a part of the PaddleVideo library, specifically the CTRGCNHead class. It is a neural network head for the CTR-GCN model that takes in input feature channels and outputs the number of classes, with an optional dropout ratio. The code imports necessary libraries, registers the class under the HEADS registry, and defines the class itself as part of the BaseHead class.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/ctrgcn_head.py\":0-31",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport math\nimport paddle\nimport paddle.nn as nn\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass CTRGCNHead(BaseHead):\n \"\"\"\n Head for CTR-GCN model.\n Args:\n in_channels: int, input feature channels. Default: 64.\n num_classes: int, output the number of classes.\n drop_out: float, dropout ratio of layer. Default: 0.\n \"\"\""
+ },
+ {
+ "comment": "Class constructor for a neural network head with optional dropout. Initializes layers, applies weight initialization, and defines the forward pass function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/ctrgcn_head.py\":33-62",
+ "content": " def __init__(self, in_channels=64, num_classes=10, drop_out=0, **kwargs):\n super().__init__(num_classes, in_channels, **kwargs)\n self.in_channels = in_channels\n self.drop_out = drop_out\n self.fc = nn.Linear(self.in_channels * 4, self.num_classes)\n if drop_out:\n self.drop_out = nn.Dropout(self.drop_out)\n else:\n self.drop_out = lambda x: x\n def init_weights(self):\n \"\"\"Initiate the parameters.\n \"\"\"\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n weight_init_(layer.weight,\n 'Normal',\n mean=0.0,\n std=math.sqrt(2. / self.num_classes))\n def forward(self, output_patch):\n \"\"\"Define how the head is going to run.\n \"\"\"\n x, N, M = output_patch\n # N*M,C,T,V\n _, c_new, T, V = x.shape\n x = paddle.reshape(x, shape=[N, M, c_new, T * V])\n x = x.mean(3).mean(1)\n x = self.drop_out(x)"
+ },
+ {
+ "comment": "This code snippet is from the ctrgcn_head class, and it returns the result of passing the input x through a fully connected layer (fc). The purpose might be to process the features extracted by the previous layers in the model for making predictions or generating output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/ctrgcn_head.py\":64-64",
+ "content": " return self.fc(x)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/57d1a9e8-3e06-4b6c-be05-05a9659ff66b.json b/docs/doc/57d1a9e8-3e06-4b6c-be05-05a9659ff66b.json
new file mode 100644
index 000000000..805f367cd
--- /dev/null
+++ b/docs/doc/57d1a9e8-3e06-4b6c-be05-05a9659ff66b.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code is a PaddlePaddle base class for loss functions, requiring subclasses to override `_forward()` and supports optional weight scaling. It initializes the loss class and defines forward pass computation.",
+ "details": [
+ {
+ "comment": "This code is the base class for a loss function in PaddlePaddle. It requires subclasses to override the `_forward()` method, which returns the normal loss without weights. The `loss_weight` parameter is optional and defaults to 1.0, which can be used to scale the final loss value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/base.py\":0-30",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\n#XXX use _forward?? or forward??\nclass BaseWeightedLoss(nn.Layer):\n \"\"\"Base class for loss.\n All subclass should overwrite the ``_forward()`` method which returns the\n normal loss without loss weights.\n Args:\n loss_weight (float): Factor scalar multiplied on the loss.\n Default: 1.0.\n \"\"\"\n def __init__(self, loss_weight=1.0):"
+ },
+ {
+ "comment": "Initializes the loss class with a weight and defines forward pass for computation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/base.py\":31-48",
+ "content": " super().__init__()\n self.loss_weight = loss_weight\n @abstractmethod\n def _forward(self, *args, **kwargs):\n pass\n def forward(self, *args, **kwargs):\n \"\"\"Defines the computation performed at every call.\n Args:\n *args: The positional arguments for the corresponding\n loss.\n **kwargs: The keyword arguments for the corresponding\n loss.\n Returns:\n paddle.Tensor: The calculated loss.\n \"\"\"\n return self._forward(*args, **kwargs) * self.loss_weight"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/57e1d0db-97fa-4149-8697-27535f8ae236.json b/docs/doc/57e1d0db-97fa-4149-8697-27535f8ae236.json
new file mode 100644
index 000000000..e3170ec90
--- /dev/null
+++ b/docs/doc/57e1d0db-97fa-4149-8697-27535f8ae236.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The code imports modules, defines Kaiming uniform initialization and SingleStageModel class. It initializes MSTCN backbone with DilatedResidualLayer stages and applies softmax to previous outputs, concatenating them together while initializing weights for convolutional layers with KaimingUniform_like_torch.",
+ "details": [
+ {
+ "comment": "This code snippet appears to be part of a larger file and sets up some initial definitions, imports, and checks for necessary conditions. It includes license information, imports various modules, and defines a function to calculate fan-in and fan-out for tensor dimensions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ms_tcn.py\":0-31",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nimport copy\nimport random\nimport math\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\ndef _calculate_fan_in_and_fan_out(tensor):\n dimensions = len(tensor.shape)\n if dimensions < 2:\n raise ValueError(\"Fan in and fan out can not be computed \\\n for tensor with fewer than 2 dimensions\")"
+ },
+ {
+ "comment": "This code defines three functions: `_calculate_fan_in_and_fan_out`, `calculate_gain`, and `KaimingUniform_like_torch`. The first function calculates the fan-in and fan-out values based on the input tensor's dimensions. The second function determines the gain value depending on the nonlinearity used. The third function applies the Kaiming uniform initialization to the weight_npy parameter, utilizing the previous two functions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ms_tcn.py\":33-67",
+ "content": " if dimensions == 2: # Linear\n fan_in = tensor.shape[1]\n fan_out = tensor.shape[0]\n else:\n num_input_fmaps = tensor.shape[1]\n num_output_fmaps = tensor.shape[0]\n receptive_field_size = 1\n if tensor.dim() > 2:\n receptive_field_size = tensor[0][0].numel()\n fan_in = num_input_fmaps * receptive_field_size\n fan_out = num_output_fmaps * receptive_field_size\n return fan_in, fan_out\ndef calculate_gain(nonlinearity=None, a=None):\n if nonlinearity == 'tanh':\n return 5.0 / 3\n elif nonlinearity == 'relu':\n return math.sqrt(2.0)\n elif nonlinearity == 'leaky_relu':\n if a != None:\n return math.sqrt(2.0 / (1 + a**2))\n else:\n return math.sqrt(2.0 / (1 + 0.01**2))\n elif nonlinearity == 'selu':\n return 3.0 / 4\n else:\n return 1\ndef KaimingUniform_like_torch(weight_npy,\n mode='fan_in',\n nonlinearity='leaky_relu'):\n fan_in, fan_out = _calculate_fan_in_and_fan_out(weight_npy)"
+ },
+ {
+ "comment": "This code defines a SingleStageModel class that inherits from nn.Layer and consists of a convolutional layer, multiple DilatedResidualLayers, and another convolutional layer. The model is initialized with specified parameters: number of layers, number of feature maps, input dimension, and number of output classes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ms_tcn.py\":68-99",
+ "content": " if mode == 'fan_in':\n fan_mode = fan_in\n else:\n fan_mode = fan_out\n a = math.sqrt(5.0)\n gain = calculate_gain(nonlinearity=nonlinearity, a=a)\n std = gain / math.sqrt(fan_mode)\n bound = math.sqrt(3.0) * std\n return np.random.uniform(-bound, bound, weight_npy.shape)\ndef init_bias(weight_npy, bias_npy):\n # attention this weight is not bias\n fan_in, fan_out = _calculate_fan_in_and_fan_out(weight_npy)\n bound = 1.0 / math.sqrt(fan_in)\n return np.random.uniform(-bound, bound, bias_npy.shape)\nclass SingleStageModel(nn.Layer):\n def __init__(self, num_layers, num_f_maps, dim, num_classes):\n super(SingleStageModel, self).__init__()\n self.conv_in = nn.Conv1D(dim, num_f_maps, 1)\n self.layers = nn.LayerList([\n copy.deepcopy(DilatedResidualLayer(2**i, num_f_maps, num_f_maps))\n for i in range(num_layers)\n ])\n self.conv_out = nn.Conv1D(num_f_maps, num_classes, 1)\n def forward(self, x):\n out = self.conv_in(x)\n for layer in self.layers:"
+ },
+ {
+ "comment": "The code defines a DilatedResidualLayer, which is a type of residual layer used in the MSTCN backbone. The MSTCN class initializes a SingleStageModel and a list of stages using the provided parameters. Each stage within the model is an instance of the DilatedResidualLayer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ms_tcn.py\":100-131",
+ "content": " out = layer(out)\n out = self.conv_out(out)\n return out\nclass DilatedResidualLayer(nn.Layer):\n def __init__(self, dilation, in_channels, out_channels):\n super(DilatedResidualLayer, self).__init__()\n self.conv_dilated = nn.Conv1D(in_channels,\n out_channels,\n 3,\n padding=dilation,\n dilation=dilation)\n self.conv_in = nn.Conv1D(out_channels, out_channels, 1)\n self.dropout = nn.Dropout()\n def forward(self, x):\n out = F.relu(self.conv_dilated(x))\n out = self.conv_in(out)\n out = self.dropout(out)\n return (x + out)\n@BACKBONES.register()\nclass MSTCN(nn.Layer):\n def __init__(self, num_stages, num_layers, num_f_maps, dim, num_classes):\n super().__init__()\n self.stage1 = SingleStageModel(num_layers, num_f_maps, dim, num_classes)\n self.stages = nn.LayerList([\n copy.deepcopy("
+ },
+ {
+ "comment": "The code defines a forward function for MSTCN model and initializes the weights for convolutional layers. It iterates over stages, applying softmax to previous output and concatenating it to previous outputs. Weights are initialized with KaimingUniform_like_torch for conv1D layers and bias is set according to the layer's weight.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/ms_tcn.py\":132-153",
+ "content": " SingleStageModel(num_layers, num_f_maps, num_classes,\n num_classes)) for s in range(num_stages - 1)\n ])\n def forward(self, x):\n \"\"\" MSTCN forward\n \"\"\"\n out = self.stage1(x)\n outputs = out.unsqueeze(0)\n for s in self.stages:\n out = s(F.softmax(out, axis=1))\n outputs = paddle.concat((outputs, out.unsqueeze(0)), axis=0)\n return outputs\n def init_weights(self):\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv1D):\n layer.weight.set_value(\n KaimingUniform_like_torch(layer.weight).astype('float32'))\n if layer.bias is not None:\n layer.bias.set_value(\n init_bias(layer.weight, layer.bias).astype('float32'))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/57ebb0bb-5f73-4cec-b336-49525daca8bd.json b/docs/doc/57ebb0bb-5f73-4cec-b336-49525daca8bd.json
new file mode 100644
index 000000000..73dabc71f
--- /dev/null
+++ b/docs/doc/57ebb0bb-5f73-4cec-b336-49525daca8bd.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This script downloads and extracts four tar files (audio, pptsm, bmn, lstm) related to the FootballAction application within PaddleVideo. The tar files are then deleted after extraction.",
+ "details": [
+ {
+ "comment": "This script downloads and extracts four tar files (audio, pptsm, bmn, lstm) related to the FootballAction application within PaddleVideo. The tar files are then deleted after extraction.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/FootballAction/checkpoints/download.sh\":0-17",
+ "content": "# audio\nwget https://videotag.bj.bcebos.com/PaddleVideo-release2.1/FootballAction/audio.tar\n# pptsm\nwget https://videotag.bj.bcebos.com/PaddleVideo-release2.1/FootballAction/pptsm.tar\n# bmn\nwget https://videotag.bj.bcebos.com/PaddleVideo-release2.1/FootballAction/bmn.tar\n# lstm\nwget https://videotag.bj.bcebos.com/PaddleVideo-release2.1/FootballAction/lstm.tar\ntar -xvf audio.tar\ntar -xvf pptsm.tar\ntar -xvf bmn.tar\ntar -xvf lstm.tar\nrm -f audio.tar\nrm -f pptsm.tar\nrm -f bmn.tar\nrm -f lstm.tar"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5845a5da-55cb-4023-80e7-a48cbe9a0448.json b/docs/doc/5845a5da-55cb-4023-80e7-a48cbe9a0448.json
new file mode 100644
index 000000000..71568d88b
--- /dev/null
+++ b/docs/doc/5845a5da-55cb-4023-80e7-a48cbe9a0448.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The MetricsCalculator class calculates accuracy, average loss, and mean loss for multimodal video tagging models with various top-k values. It compares predictions to actual labels in a multilabel classification and logs the metrics using a logger.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and defines a class for calculating metrics. The MetricsCalculator class initializes with name, mode ('train', 'val', or 'test'), and metrics_args.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/accuracy_metrics.py\":0-34",
+ "content": "# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import absolute_import\nfrom __future__ import unicode_literals\nfrom __future__ import print_function\nfrom __future__ import division\nimport numpy as np\nimport logging\nlogger = logging.getLogger(__name__)\nclass MetricsCalculator():\n \"\"\"\n MetricsCalculator\n \"\"\"\n def __init__(self, name, mode, metrics_args):\n \"\"\"\n init\n \"\"\"\n self.name = name\n self.mode = mode # 'train', 'val', 'test'"
+ },
+ {
+ "comment": "This code initializes an AccuracyMetrics class with metrics arguments, resets the metrics values, finalizes and calculates the accuracy and average loss for each top-n value, and returns the computed metrics. The purpose is to measure the performance of a multimodal video tagging model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/accuracy_metrics.py\":35-67",
+ "content": " self.acc_dict = {}\n self.top_n_list = metrics_args.MODEL.top_n\n self.num_classes = metrics_args.MODEL.num_classes\n self.reset()\n def reset(self):\n \"\"\"\n reset\n \"\"\"\n logger.info('Resetting {} metrics...'.format(self.mode))\n for topk in self.top_n_list:\n self.acc_dict['avg_acc%d' % (topk)] = 0.0\n self.aggr_loss = 0.0\n self.aggr_batch_size = 0\n def finalize_metrics(self):\n \"\"\"finalize_metrics\n \"\"\"\n for key, value in self.acc_dict.items():\n self.acc_dict[key] = value / self.aggr_batch_size\n self.aggr_loss = self.aggr_loss / self.aggr_batch_size\n def get_computed_metrics(self):\n \"\"\"get_computed_metrics\n \"\"\"\n acc_dict = {}\n for key, value in self.acc_dict.items():\n acc_dict[key] = value / self.aggr_batch_size\n aggr_loss = self.aggr_loss / self.aggr_batch_size\n return acc_dict, aggr_loss\n def accumulate(self, loss, softmax, labels):"
+ },
+ {
+ "comment": "This code snippet is part of a class that accumulates metrics for video tagging. It computes the mean loss, average accuracy for different top k values, and then logs these metrics in an informative format using a logger.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/accuracy_metrics.py\":68-94",
+ "content": " \"\"\"accumulate\n \"\"\"\n cur_batch_size = softmax.shape[0]\n # if returned loss is None for e.g. test, just set loss to be 0.\n if loss is None:\n cur_loss = 0.\n else:\n cur_loss = np.mean(np.array(loss)) #\n self.aggr_batch_size += cur_batch_size\n self.aggr_loss += cur_loss * cur_batch_size\n for top_k in self.top_n_list:\n self.acc_dict['avg_acc%d' %\n (top_k)] += cur_batch_size * compute_topk_accuracy(\n softmax, labels, top_k=top_k) * 100.\n return\n def finalize_and_log_out(self, info=''):\n \"\"\"finalize_and_log_out\n \"\"\"\n metrics_dict, loss = self.get_computed_metrics()\n acc_str = []\n for name, value in metrics_dict.items():\n acc_str.append('{}:{},'.format('%s' % name, '%.2f' % value))\n acc_str = '\\t'.join(acc_str)\n logger.info(info +\n '\\tLoss: {},\\t{}'.format('%.6f' % loss, '%s' % acc_str))"
+ },
+ {
+ "comment": "This code computes the number of correct hits for a given top_k in multilabel classification, where it calculates the top_k predictions and checks if any of them match with the actual labels. It returns the total number of correct hits across all samples in the batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/accuracy_metrics.py\":95-124",
+ "content": " return\ndef compute_topk_correct_hits_multilabel(top_k, preds, labels):\n '''Compute the number of corret hits'''\n batch_size = preds.shape[0]\n top_k_preds = np.zeros((batch_size, 10), dtype=np.float32)\n for i in range(batch_size):\n top_k_preds[i, :] = np.argsort(-preds[i, :])[:10]\n correctness = np.zeros(batch_size, dtype=np.float32)\n for i in range(batch_size):\n correc_sum = 0\n for label_id in range(len(labels[i])):\n label_hit = labels[i][label_id]\n if label_hit == 0 or label_hit < 0.1:\n continue\n if label_id in top_k_preds[i, :top_k].astype(np.int32).tolist():\n # correc_sum += 1\n correc_sum = 1\n break\n correctness[i] = correc_sum\n correct_hits = sum(correctness)\n return correct_hits\ndef compute_topk_correct_hits(top_k, preds, labels):\n '''Compute the number of corret hits'''\n batch_size = preds.shape[0]\n top_k_preds = np.zeros((batch_size, top_k), dtype=np.float32)"
+ },
+ {
+ "comment": "This code calculates the top-k accuracy for a batch of predictions and labels. It first computes the top-k predictions and then checks if the ground truth label is within the top-k predictions. The function returns the number of correct hits divided by the batch size to obtain the accuracy. The main section demonstrates usage with example data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/accuracy_metrics.py\":125-157",
+ "content": " for i in range(batch_size):\n top_k_preds[i, :] = np.argsort(-preds[i, :])[:top_k]\n correctness = np.zeros(batch_size, dtype=np.int32)\n for i in range(batch_size):\n if labels[i] in top_k_preds[i, :].astype(np.int32).tolist():\n correctness[i] = 1\n correct_hits = sum(correctness)\n return correct_hits\ndef compute_topk_accuracy(softmax, labels, top_k):\n \"\"\"compute_topk_accuracy\n \"\"\"\n computed_metrics = {}\n assert labels.shape[0] == softmax.shape[0], \"Batch size mismatch.\"\n aggr_batch_size = labels.shape[0]\n # aggr_top_k_correct_hits = compute_topk_correct_hits(top_k, softmax, labels)\n aggr_top_k_correct_hits = compute_topk_correct_hits_multilabel(\n top_k, softmax, labels)\n # normalize results\n computed_metrics = \\\n float(aggr_top_k_correct_hits) / aggr_batch_size\n return computed_metrics\nif __name__ == \"__main__\":\n pred = np.array([[0.5, 0.2, 0.3, 0, 0]])\n label = np.array([[0.5, 0.5, 0, 0, 0]])\n print('pred: ', pred)\n print('label: ', label)"
+ },
+ {
+ "comment": "Computing top-1 and top-5 hits for multilabel prediction using compute_topk_correct_hits_multilabel function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/accuracy_metrics.py\":158-159",
+ "content": " print('Top 1 hits', compute_topk_correct_hits_multilabel(1, pred, label))\n print('Top 5 hits', compute_topk_correct_hits_multilabel(5, pred, label))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/584f8aff-900a-4892-a9a9-3f8ed867f6b8.json b/docs/doc/584f8aff-900a-4892-a9a9-3f8ed867f6b8.json
new file mode 100644
index 000000000..593697cb5
--- /dev/null
+++ b/docs/doc/584f8aff-900a-4892-a9a9-3f8ed867f6b8.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code imports required modules, handles web serving for PaddleVideo models and includes a client to make predictions by passing encoded frames and shape to the predict method. It outputs labels and probabilities as results.",
+ "details": [
+ {
+ "comment": "This code is importing necessary modules, defining a function to parse command line arguments, and setting up a parser for those arguments. The main purpose of this file seems to be handling the execution of the PaddleVideo model in a web serving environment.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/pipeline_rpc_client.py\":0-28",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\ntry:\n from paddle_serving_server_gpu.pipeline import PipelineClient\nexcept ImportError:\n from paddle_serving_server.pipeline import PipelineClient\nimport argparse\nfrom utils import numpy_to_base64, parse_file_paths, video_to_numpy\ndef parse_args():\n # general params\n parser = argparse.ArgumentParser(\"PaddleVideo Web Serving model script\")\n parser.add_argument('-c',\n '--config',"
+ },
+ {
+ "comment": "This code sets up command line arguments for the serving config file path, RPC port number, and input file/directory path. It then connects to a server at the specified port, processes the input files (decoding videos and converting ndarrays to base64 strings), and transports the data to the server for processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/pipeline_rpc_client.py\":29-59",
+ "content": " type=str,\n default='configs/PP-TSM.yaml',\n help='serving config file path')\n parser.add_argument('-ptn',\n '--port_number',\n type=int,\n default=9993,\n help='rpc port number')\n parser.add_argument('-i',\n '--input_file',\n type=str,\n help='input file path or directory path')\n return parser.parse_args()\nif __name__ == \"__main__\":\n args = parse_args()\n client = PipelineClient()\n client.connect([f'127.0.0.1:{args.port_number}'])\n files_list = parse_file_paths(args.input_file)\n for file_path in files_list:\n # decoding video and get stacked frames as ndarray\n decoded_frames = video_to_numpy(file_path=file_path)\n # encode ndarray to base64 string for transportation.\n decoded_frames_base64 = numpy_to_base64(decoded_frames)\n # transport to server & get get results."
+ },
+ {
+ "comment": "This code snippet uses a PaddleVideo client to make a prediction. It passes base64 encoded frames and their shape to the client's predict method, fetching both labels and probabilities as results. The print statement outputs these results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/python_serving/pipeline_rpc_client.py\":60-67",
+ "content": " ret = client.predict(feed_dict={\n \"frames\": decoded_frames_base64,\n \"frames_shape\": str(decoded_frames.shape)\n },\n fetch=[\"label\", \"prob\"])\n # print result\n print(ret)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/5a13773a-b59f-4c5a-98b2-946896833581.json b/docs/doc/5a13773a-b59f-4c5a-98b2-946896833581.json
new file mode 100644
index 000000000..fb763050c
--- /dev/null
+++ b/docs/doc/5a13773a-b59f-4c5a-98b2-946896833581.json
@@ -0,0 +1,20 @@
+{
+ "summary": "ActBERT is a multimodal pretrain task using global action info and TaNgled Transformer block (TNT) for text-object interactions. It outperforms state-of-the-art in video-language tasks and can be trained on HowTo100M dataset with AMP for faster training, evaluated on MSR-VTT, and found at the provided link.",
+ "details": [
+ {
+ "comment": "This is an introduction to ActBERT, a multimodal pretrain task proposed by Baidu in CVPR2020. It uses global action information to analyze mutual interactions between linguistic texts and local regional objects. The method introduces TaNgled Transformer block (TNT) to encode three sources of information. ActBERT outperforms state-of-the-art in five video-and-language tasks, including text-video clip retrieval, video captioning, and action segmentation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/multimodal/actbert.md\":0-24",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/multimodal/actbert.md) | English\n# ActBERT\n---\n## Contents\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Reference](#Reference)\nBefore getting started, you need to install additional dependencies as follows:\n```bash\npython -m pip install paddlenlp\npython -m pip install lmdb\n```\n## Introduction\nActbert is proposed by Baidu in CVPR2020 for multimodal pretrain task. It leverage global action information to cat- alyze mutual interactions between linguistic texts and local regional objects. This method introduce a TaNgled Transformer block (TNT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. ActBERT significantly outperforms the state- of-the-art in five downstream video-and-language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization.\n\n
![](\"../../../images/actbert.png\")
"
+ },
+ {
+ "comment": "This code describes how to train ActBERT on HowTo100M dataset. It first requires downloading the pretrain-model \"bert-base-uncased\" from a specified URL and adding its path to the config file. Then, it provides the command to start training using the provided script with specific configuration and GPU allocation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/multimodal/actbert.md\":25-64",
+ "content": "
\n
![](\"./imgs/t2vlad.png\")
\n
\n
![](\"docs/images/home.gif\")
\n
\n
![](\"docs/images/user_group.png\")
\n
![](\"./docs/images/features.png\")
\n
\n \n | Action Recognition | \n
\n \n | Kinetics-400 (Homepage) (CVPR'2017) | \n UCF101 (Homepage) (CRCV-IR-12-01) | \n ActivityNet (Homepage) (CVPR'2015) | \n YouTube-8M (Homepage) (CVPR'2017) | \n
\n \n | Action Localization | \n
\n \n | ActivityNet (Homepage) (CVPR'2015) | \n | \n | \n | \n
\n \n | Spatio-Temporal Action Detection | "
+ },
+ {
+ "comment": "This code snippet is a table of datasets. It mentions dataset names, their corresponding homepages, and the year they were published in. The categories include Skeleton-based Action Recognition, Depth Estimation, and Text-Video Retrieval.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/README.md\":28-57",
+ "content": "
\n \n | AVA (Homepage) (CVPR'2018) | \n | \n | \n | \n
\n \n | Skeleton-based Action Recognition | \n
\n \n | NTURGB+D (Homepage) (IEEE CS'2016) | \n FSD (Homepage) | \n | \n | \n
\n \n | Depth Estimation | \n
\n \n | Oxford-RobotCar (Homepage) (IJRR'2017) | \n | \n | \n | \n
\n \n | Text-Video Retrieval | \n
\n \n | MSR-VTT (<"
+ },
+ {
+ "comment": "This code appears to be part of a table within an HTML file. The table lists different datasets related to video and text, with their respective names, descriptions, and links to their homepages or documentation. It also provides information on the year of publication for each dataset, which seems to be from various conferences such as CVPR and ICCV.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/README.md\":57-72",
+ "content": "a href=\"https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/\" rel=\"nofollow\">Homepage) (CVPR'2016) | \n | \n | \n | \n
\n \n | Text-Video Pretrained Model | \n
\n \n | HowTo100M (Homepage) (ICCV'2019) | \n | \n | \n | \n
\n\n
"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/67f9dbca-b1f4-4ef2-93a4-4af3f80cadfa.json b/docs/doc/67f9dbca-b1f4-4ef2-93a4-4af3f80cadfa.json
new file mode 100644
index 000000000..cc2830448
--- /dev/null
+++ b/docs/doc/67f9dbca-b1f4-4ef2-93a4-4af3f80cadfa.json
@@ -0,0 +1,80 @@
+{
+ "summary": "This code constructs a PaddlePaddle DeepLab network with convolution layers, batch normalization and activation functions in Bottleneck and ResNet classes. It includes additional layers for better performance, initializes ASPP modules in the DeepLab model for feature extraction, defines a segmentation model with ResNet backbone, adaptive pooling, and Decoder modules, and performs inference using forward function.",
+ "details": [
+ {
+ "comment": "This code defines a class `FrozenBatchNorm2D` which is a type of batch normalization layer where the batch statistics and affine parameters are fixed. It inherits from `nn.Layer` and initializes `paddle.ones` and `paddle.zeros` tensors as its parameters, representing fixed batch statistics and affine transformation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":0-32",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport copy\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..registry import BACKBONES\nclass FrozenBatchNorm2D(nn.Layer):\n \"\"\"\n BatchNorm2D where the batch statistics and the affine parameters\n are fixed\n \"\"\"\n def __init__(self, n, epsilon=1e-5):\n super(FrozenBatchNorm2D, self).__init__()\n x1 = paddle.ones([n])\n x2 = paddle.zeros([n])\n weight = self.create_parameter("
+ },
+ {
+ "comment": "The code defines a DeepLab class, initializes its parameters, and creates a Bottleneck layer. The DeepLab class contains a weight parameter for the convolution operation, a bias parameter to adjust output, and running_mean and running_var parameters used in normalization. The Bottleneck layer has an expansion factor of 4, implying it will increase the number of channels by this factor. This code is part of a neural network backbone implementation using PaddlePaddle framework.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":33-58",
+ "content": " shape=x1.shape, default_initializer=nn.initializer.Assign(x1))\n bias = self.create_parameter(\n shape=x2.shape, default_initializer=nn.initializer.Assign(x2))\n running_mean = self.create_parameter(\n shape=x2.shape, default_initializer=nn.initializer.Assign(x2))\n running_var = self.create_parameter(\n shape=x1.shape, default_initializer=nn.initializer.Assign(x1))\n self.add_parameter('weight', weight)\n self.add_parameter('bias', bias)\n self.add_parameter('running_mean', running_mean)\n self.add_parameter('running_var', running_var)\n self.epsilon = epsilon\n def forward(self, x):\n scale = self.weight * paddle.rsqrt((self.running_var + self.epsilon))\n bias = self.bias - self.running_mean * scale\n scale = paddle.reshape(scale, [1, -1, 1, 1])\n bias = paddle.reshape(bias, [1, -1, 1, 1])\n return x * scale + bias\nclass Bottleneck(nn.Layer):\n expansion = 4\n def __init__(self,\n inplanes,"
+ },
+ {
+ "comment": "Bottleneck class is a convolution neural network layer with batch normalization, designed for DeepLab model. It consists of 3 consecutive convolutions with varying kernel sizes and stride. BatchNorm layers are used after each convolution to normalize the activations, followed by ReLU activation function. The output channels are scaled by 4 in the final convolution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":59-85",
+ "content": " planes,\n stride=1,\n dilation=1,\n downsample=None,\n BatchNorm=None):\n super(Bottleneck, self).__init__()\n self.conv1 = nn.Conv2D(inplanes, planes, kernel_size=1, bias_attr=False)\n self.bn1 = BatchNorm(planes)\n self.conv2 = nn.Conv2D(planes,\n planes,\n kernel_size=3,\n stride=stride,\n dilation=dilation,\n padding=dilation,\n bias_attr=False)\n self.bn2 = BatchNorm(planes)\n self.conv3 = nn.Conv2D(planes,\n planes * 4,\n kernel_size=1,\n bias_attr=False)\n self.bn3 = BatchNorm(planes * 4)\n self.relu = nn.ReLU()\n self.downsample = downsample\n self.stride = stride\n self.dilation = dilation\n def forward(self, x):"
+ },
+ {
+ "comment": "Code snippet performs residual block operations using convolutional layers and batch normalization with ReLU activation. It also includes downsampling if specified, and returns the output after applying final ReLU. The class ResNet initializes a ResNet network with given number of blocks, output stride, BatchNorm type, and pretrained option.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":86-129",
+ "content": " residual = x\n out = self.conv1(x)\n out = self.bn1(out)\n out = self.relu(out)\n out = self.conv2(out)\n out = self.bn2(out)\n out = self.relu(out)\n out = self.conv3(out)\n out = self.bn3(out)\n if self.downsample is not None:\n residual = self.downsample(x)\n out += residual\n out = self.relu(out)\n return out\nclass ResNet(nn.Layer):\n def __init__(self,\n block,\n layers,\n output_stride,\n BatchNorm,\n pretrained=False):\n self.inplanes = 64\n super(ResNet, self).__init__()\n blocks = [1, 2, 4]\n if output_stride == 16:\n strides = [1, 2, 2, 1]\n dilations = [1, 1, 1, 2]\n elif output_stride == 8:\n strides = [1, 2, 1, 1]\n dilations = [1, 1, 2, 4]\n else:\n raise NotImplementedError\n # Modules\n self.conv1 = nn.Conv2D(3,\n 64,"
+ },
+ {
+ "comment": "This code is defining a deep learning model with convolutional layers, batch normalization, and activation functions. It uses the DeepLab backbone architecture and specifies parameters such as kernel sizes, strides, padding, and dilation rates for each layer. The BatchNorm parameter allows for optional batch normalization between layers, improving model performance by reducing internal covariate shift.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":130-151",
+ "content": " kernel_size=7,\n stride=2,\n padding=3,\n bias_attr=False)\n self.bn1 = BatchNorm(64)\n self.relu = nn.ReLU()\n self.maxpool = nn.MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.layer1 = self._make_layer(block,\n 64,\n layers[0],\n stride=strides[0],\n dilation=dilations[0],\n BatchNorm=BatchNorm)\n self.layer2 = self._make_layer(block,\n 128,\n layers[1],\n stride=strides[1],\n dilation=dilations[1],\n BatchNorm=BatchNorm)\n self.layer3 = self._make_layer(block,\n 256,"
+ },
+ {
+ "comment": "This code defines a class with two layers, layer3 and layer4. Layer3 is created using the _make_MG_unit function with specific parameters like block, planes, blocks, stride, dilation, and BatchNorm. Layer4 is also created by calling _make_layer function. Downsampling is done if stride is not 1 or inplanes are not equal to planes*block.expansion.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":152-175",
+ "content": " layers[2],\n stride=strides[2],\n dilation=dilations[2],\n BatchNorm=BatchNorm)\n self.layer4 = self._make_MG_unit(block,\n 512,\n blocks=blocks,\n stride=strides[3],\n dilation=dilations[3],\n BatchNorm=BatchNorm)\n self._init_weight()\n def _make_layer(self,\n block,\n planes,\n blocks,\n stride=1,\n dilation=1,\n BatchNorm=None):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n downsample = nn.Sequential(\n nn.Conv2D(self.inplanes,\n planes * block.expansion,"
+ },
+ {
+ "comment": "This code defines a function _make_MG_unit that creates a module for the DeepLab model, which includes multiple layers of a specified block. It takes in parameters such as block, planes, blocks, stride, dilation, and BatchNorm (optional). The function first checks if downsampling is needed based on stride and inplanes. If so, it creates a Conv2D layer for downsampling. Then, it appends the initial layer with the specified parameters and expands the number of layers as required. Finally, it returns the created sequence of layers as a nn.Sequential module.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":176-206",
+ "content": " kernel_size=1,\n stride=stride,\n bias_attr=False),\n BatchNorm(planes * block.expansion),\n )\n layers = []\n layers.append(\n block(self.inplanes, planes, stride, dilation, downsample,\n BatchNorm))\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(\n block(self.inplanes,\n planes,\n dilation=dilation,\n BatchNorm=BatchNorm))\n return nn.Sequential(*layers)\n def _make_MG_unit(self,\n block,\n planes,\n blocks,\n stride=1,\n dilation=1,\n BatchNorm=None):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n downsample = nn.Sequential(\n nn.Conv2D(self.inplanes,"
+ },
+ {
+ "comment": "This code defines a function that creates a convolutional neural network for the DeepLab model. It takes input, creates layers with specified parameters, and returns a Sequential object representing the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":207-239",
+ "content": " planes * block.expansion,\n kernel_size=1,\n stride=stride,\n bias_attr=False),\n BatchNorm(planes * block.expansion),\n )\n layers = []\n layers.append(\n block(self.inplanes,\n planes,\n stride,\n dilation=blocks[0] * dilation,\n downsample=downsample,\n BatchNorm=BatchNorm))\n self.inplanes = planes * block.expansion\n for i in range(1, len(blocks)):\n layers.append(\n block(self.inplanes,\n planes,\n stride=1,\n dilation=blocks[i] * dilation,\n BatchNorm=BatchNorm))\n return nn.Sequential(*layers)\n def forward(self, input, return_mid_level=False):\n x = self.conv1(input)\n x = self.bn1(x)\n x = self.relu(x)\n x = self.maxpool(x)\n x = self.layer1(x)"
+ },
+ {
+ "comment": "This code defines a DeepLab model that utilizes an ASPP module. It extracts low and mid-level features from the input, has multiple layers of convolutions, and initializes weights using specific initializers. The ASPP module applies atrous convolutions with different dilation rates for feature extraction.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":240-268",
+ "content": " low_level_feat = x\n x = self.layer2(x)\n mid_level_feat = x\n x = self.layer3(x)\n x = self.layer4(x)\n if return_mid_level:\n return x, low_level_feat, mid_level_feat\n else:\n return x, low_level_feat\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass _ASPPModule(nn.Layer):\n def __init__(self, inplanes, planes, kernel_size, padding, dilation,\n BatchNorm):\n super(_ASPPModule, self).__init__()\n self.atrous_conv = nn.Conv2D(inplanes,\n planes,\n kernel_size=kernel_size,\n stride=1,\n padding=padding,\n dilation=dilation,"
+ },
+ {
+ "comment": "The code defines a DeepLab class with an ASPP module for feature extraction. It initializes the layers and sets their weights using Kaiming normal initialization or fills BatchNorm2D weight with 1 and biases with 0. The ASPP class accepts backbone and output_stride as parameters to determine dilations for the ASPP modules.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":269-306",
+ "content": " bias_attr=False)\n self.bn = BatchNorm(planes)\n self.relu = nn.ReLU()\n self._init_weight()\n def forward(self, x):\n x = self.atrous_conv(x)\n x = self.bn(x)\n return self.relu(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n m.weight_attr = nn.initializer.KaimingNormal()\n elif isinstance(m, nn.BatchNorm2D):\n m.weight.data.fill_(1)\n m.bias.data.zero_()\nclass ASPP(nn.Layer):\n def __init__(self, backbone, output_stride, BatchNorm):\n super(ASPP, self).__init__()\n if backbone == 'drn':\n inplanes = 512\n elif backbone == 'mobilenet':\n inplanes = 320\n else:\n inplanes = 2048\n if output_stride == 16:\n dilations = [1, 6, 12, 18]\n elif output_stride == 8:\n dilations = [1, 12, 24, 36]\n else:\n raise NotImplementedError\n self.aspp1 = _ASPPModule(inplanes,"
+ },
+ {
+ "comment": "This code initializes four instances of the _ASPPModule class, each with different dilation rates and padding values for the DeepLab model's ASPP feature extraction module. The inplanes parameter is consistent across all four modules, indicating the number of input feature planes. BatchNorm specifies whether to apply batch normalization or not.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":307-329",
+ "content": " 256,\n 1,\n padding=0,\n dilation=dilations[0],\n BatchNorm=BatchNorm)\n self.aspp2 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[1],\n dilation=dilations[1],\n BatchNorm=BatchNorm)\n self.aspp3 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[2],\n dilation=dilations[2],\n BatchNorm=BatchNorm)\n self.aspp4 = _ASPPModule(inplanes,\n 256,\n 3,\n padding=dilations[3],\n dilation=dilations[3],\n BatchNorm=BatchNorm)"
+ },
+ {
+ "comment": "This code defines a DeepLab backbone model for image segmentation. It has adaptive global average pooling, multiple ASPP modules, and convolutional layers with batch normalization, ReLU activation, and dropout regularization. The constructor initializes the model's sublayers with Kaiming Normal initialization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":331-362",
+ "content": " self.global_avg_pool = nn.Sequential(\n nn.AdaptiveAvgPool2D((1, 1)),\n nn.Conv2D(inplanes, 256, 1, stride=1, bias_attr=False),\n BatchNorm(256), nn.ReLU())\n self.conv1 = nn.Conv2D(1280, 256, 1, bias_attr=False)\n self.bn1 = BatchNorm(256)\n self.relu = nn.ReLU()\n self.dropout = nn.Dropout(0.1)\n self._init_weight()\n def forward(self, x):\n x1 = self.aspp1(x)\n x2 = self.aspp2(x)\n x3 = self.aspp3(x)\n x4 = self.aspp4(x)\n x5 = self.global_avg_pool(x)\n x5 = F.interpolate(x5,\n size=x4.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat(x=[x1, x2, x3, x4, x5], axis=1)\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n return self.dropout(x)\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()"
+ },
+ {
+ "comment": "This code is defining a Decoder class that takes in a backbone and BatchNorm as arguments. It initializes a convolution layer, batch normalization layer, and ReLU activation function. The last convolution sequence includes two convolutional layers with BatchNorm between them, followed by an optional second sequence of layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":363-394",
+ "content": " elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass Decoder(nn.Layer):\n def __init__(self, backbone, BatchNorm):\n super(Decoder, self).__init__()\n if backbone == 'resnet':\n low_level_inplanes = 256\n elif backbone == 'mobilenet':\n raise NotImplementedError\n else:\n raise NotImplementedError\n self.conv1 = nn.Conv2D(low_level_inplanes, 48, 1, bias_attr=False)\n self.bn1 = BatchNorm(48)\n self.relu = nn.ReLU()\n self.last_conv = nn.Sequential(\n nn.Conv2D(304,\n 256,\n kernel_size=3,\n stride=1,\n padding=1,\n bias_attr=False), BatchNorm(256), nn.ReLU(),\n nn.Sequential(),\n nn.Conv2D(256,\n 256,\n kernel_size=3,\n stride=1,\n padding=1,"
+ },
+ {
+ "comment": "The provided code defines a DeepLab model for segmentation. It includes a convolution layer, batch normalization, ReLU activation function, and interpolation operation. The forward method processes input features and returns output features. The _init_weight method initializes the weight of each sublayer. The DeepLab class takes parameters like backbone type, output stride, and freeze batch normalization flag for model initialization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":395-425",
+ "content": " bias_attr=False), BatchNorm(256), nn.ReLU(),\n nn.Sequential())\n self._init_weight()\n def forward(self, x, low_level_feat):\n low_level_feat = self.conv1(low_level_feat)\n low_level_feat = self.bn1(low_level_feat)\n low_level_feat = self.relu(low_level_feat)\n x = F.interpolate(x,\n size=low_level_feat.shape[2:],\n mode='bilinear',\n align_corners=True)\n x = paddle.concat(x=[x, low_level_feat], axis=1)\n x = self.last_conv(x)\n return x\n def _init_weight(self):\n for m in self.sublayers():\n if isinstance(m, nn.Conv2D):\n nn.initializer.KaimingNormal()\n elif isinstance(m, nn.GroupNorm):\n m.weight.data = nn.initializer.Constant(1)\n m.bias.data = nn.initializer.Constant(0)\nclass DeepLab(nn.Layer):\n \"\"\"DeepLab model for segmentation\"\"\"\n def __init__(self, backbone='resnet', output_stride=16, freeze_bn=True):"
+ },
+ {
+ "comment": "The code defines a DeepLab class with an optional frozen Batch Normalization layer. It initializes the backbone network (ResNet) and adds ASPP and Decoder modules. The forward function performs inference, returning either the final output or additional intermediate features depending on the return_aspp flag.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/deeplab.py\":426-453",
+ "content": " super(DeepLab, self).__init__()\n if freeze_bn == True:\n print(\"Use frozen BN in DeepLab!\")\n BatchNorm = FrozenBatchNorm2D\n else:\n BatchNorm = nn.BatchNorm2D\n self.backbone = ResNet(Bottleneck, [3, 4, 23, 3],\n output_stride,\n BatchNorm,\n pretrained=True)\n self.aspp = ASPP(backbone, output_stride, BatchNorm)\n self.decoder = Decoder(backbone, BatchNorm)\n def forward(self, input, return_aspp=False):\n \"\"\"forward function\"\"\"\n if return_aspp:\n x, low_level_feat, mid_level_feat = self.backbone(input, True)\n else:\n x, low_level_feat = self.backbone(input)\n aspp_x = self.aspp(x)\n x = self.decoder(aspp_x, low_level_feat)\n if return_aspp:\n return x, aspp_x, low_level_feat, mid_level_feat\n else:\n return x, low_level_feat"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/683b79f6-e8d9-41ea-8f71-b0ddc8cdb9cf.json b/docs/doc/683b79f6-e8d9-41ea-8f71-b0ddc8cdb9cf.json
new file mode 100644
index 000000000..073c792c9
--- /dev/null
+++ b/docs/doc/683b79f6-e8d9-41ea-8f71-b0ddc8cdb9cf.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This Python package, \"paddlevideo\", utilizes PaddlePaddle toolkits for video understanding and supports multiple Python versions. It is set up using setuptools and includes dependencies and documentation.",
+ "details": [
+ {
+ "comment": "This code is setting up a Python package using setuptools for the PaddleVideo library, specifying its name as \"paddlevideo\". It includes the necessary dependencies listed in the \"requirements.txt\" file and provides a README file located at 'docs/en/whl_en.md' for documentation purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/setup.py\":0-33",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom setuptools import setup\nfrom io import open\nwith open('requirements.txt', encoding=\"utf-8-sig\") as f:\n requirements = f.readlines()\ndef readme():\n \"\"\"readme\"\"\"\n with open('docs/en/whl_en.md', encoding=\"utf-8-sig\") as f:\n README = f.read()\n return README\nsetup(\n name='paddlevideo', #name of .whl file\n packages=['ppvideo'], #install package name\n package_dir={'ppvideo': ''},\n include_package_data=True, #Accept all data files and directories matched by MANIFEST.in"
+ },
+ {
+ "comment": "This code is a setup file for a Python package named \"ppvideo\" that utilizes PaddlePaddle toolkits for video understanding. It specifies installation requirements, entry points, version, license, description, URL, download link, keywords, and classifiers. The package supports multiple versions of Python and is categorized under the Utilities topic.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/setup.py\":34-55",
+ "content": " install_requires=requirements,\n entry_points={\"console_scripts\": [\"ppvideo= ppvideo.tools.paddlevideo_clas:main\"]},\n version='0.0.1',\n license='Apache License 2.0',\n description='Awesome Video toolkits based on PaddlePaddle ',\n long_description=readme(),\n long_description_content_type='text/markdown',\n url='https://github.com/PaddlePaddle/PaddleVideo',\n download_url='https://github.com/PaddlePaddle/PaddleVideo.git',\n keywords=[\n 'A treasure chest for video understanding powered by PaddlePaddle.'\n ],\n classifiers=[\n 'Intended Audience :: Developers', 'Operating System :: OS Independent',\n 'Natural Language :: Chinese (Simplified)',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.2',\n 'Programming Language :: Python :: 3.3',\n 'Programming Language :: Python :: 3.4',\n 'Programming Language :: Python :: 3.5',\n 'Programming Language :: Python :: 3.6',\n 'Programming Language :: Python :: 3.7', 'Topic :: Utilities'"
+ },
+ {
+ "comment": "This code is creating a tuple with empty elements. The specific purpose or usage of this tuple in the context of the setup.py file might require more information to provide an accurate and relevant comment.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/setup.py\":56-56",
+ "content": " ],)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/68a58ce5-b0c3-46e1-b9a4-31e20350a75a.json b/docs/doc/68a58ce5-b0c3-46e1-b9a4-31e20350a75a.json
new file mode 100644
index 000000000..1b2aee9ed
--- /dev/null
+++ b/docs/doc/68a58ce5-b0c3-46e1-b9a4-31e20350a75a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "The code reads files from a directory, retrieves their names without extensions, checks if the names exist in another set of labels, and deletes any mismatched labels. It then writes the fixed label file with updated sizes.",
+ "details": [
+ {
+ "comment": "The code reads files from a directory, retrieves their names without extensions, checks if the names exist in another set of labels, and deletes any mismatched labels. It then writes the fixed label file with updated sizes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/fix_bad_label.py\":0-36",
+ "content": "import copy\nimport json\nimport re\nimport os\nurl = '/home/aistudio/work/BMN/Input_for_bmn/feature/'\ndirectory = os.fsencode(url)\ncount = 0\ntarget_set = []\nfor file in os.listdir(directory):\n filename = os.fsdecode(file)\n target_name = filename.split('.npy')[0]\n target_set.append(target_name)\n count += 1\nprint('Feature size:', len(target_set))\nwith open('/home/aistudio/work/BMN/Input_for_bmn/label.json') as f:\n data = json.load(f)\ndelet_set = []\nfor key in data.keys():\n if not key in target_set:\n delet_set.append(key)\nprint('(Label) Original size:', len(data))\nprint('(Label) Deleted size:', len(delet_set))\nfor item in delet_set:\n data.pop(item, None)\nprint('(Label) Fixed size:', len(data))\njsonString = json.dumps(data, indent=4, ensure_ascii=False)\njsonFile = open('/home/aistudio/work/BMN/Input_for_bmn/label_fixed.json', 'w')\njsonFile.write(jsonString)\njsonFile.close()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6963a95f-f75e-4530-869d-4afbc5a040d6.json b/docs/doc/6963a95f-f75e-4530-869d-4afbc5a040d6.json
new file mode 100644
index 000000000..8f8765a8f
--- /dev/null
+++ b/docs/doc/6963a95f-f75e-4530-869d-4afbc5a040d6.json
@@ -0,0 +1,50 @@
+{
+ "summary": "This function ensures state dict consistency by comparing optimizer and model parameters, saving/loading checkpoints, and converting sub-bn to normal bn. It checks if certain layers are set to load and prints a message for unloaded weights before loading pre-trained weights and setting the optimizer's state dictionary.",
+ "details": [
+ {
+ "comment": "This function converts Sub-BN parameters to normal BN parameters in a state dict. It renames `bn.bn` to `bn`, and modifies `_mean` and `_variance` accordingly. This is done before saving or evaluation to maintain consistency with normal BN layers. The modifications are made by iterating through the dictionary and checking if the key ends with the appropriate string, then updating it accordingly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":0-30",
+ "content": "import os\nimport numpy as np\nimport paddle\nimport copy\ndef sub_to_normal_bn(sd):\n \"\"\"\n When save, Convert the Sub-BN paprameters to normal BN parameters in a state dict.\n There are two copies of BN layers in a Sub-BN implementation: `bn.bn` and\n `bn.split_bn`. `bn.split_bn` is used during training and\n \"compute_precise_bn\". Before saving or evaluation, its stats are copied to\n `bn.bn`. We rename `bn.bn` to `bn` and store it to be consistent with normal\n BN layers.\n Args:\n sd (OrderedDict): a dict of parameters which might contain Sub-BN\n parameters.\n Returns:\n new_sd (OrderedDict): a dict with Sub-BN parameters reshaped to\n normal parameters.\n \"\"\"\n modifications = [\n (\"bn.bn._mean\", \"bn._mean\"),\n (\"bn.bn._variance\", \"bn._variance\"),\n ]\n to_remove = [\"bn.bn.\", \".split_bn.\"]\n key_list = list(sd.keys()) #odict_keys to list\n for key in key_list:\n for before, after in modifications:\n if key.endswith(before):\n new_key = key.split(before)[0] + after"
+ },
+ {
+ "comment": "This function converts BN parameters to Sub-BN parameters when loading a checkpoint into a model containing Sub-BNs. It loops through the model's parameters, if a parameter has the \"bn.split_bn.\" prefix and is not the weight or bias of BN, it renames and moves the corresponding value from the checkpoint dict to the bn.bn key in the same subdict. Finally, it adjusts the shape of the Sub-BN parameters to match the original BN parameters' shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":31-57",
+ "content": " sd[new_key] = sd.pop(key)\n for rm in to_remove:\n if rm in key and key in sd:\n del sd[key]\ndef normal_to_sub_bn(checkpoint_sd, model_sd):\n \"\"\"\n When load, Convert BN parameters to Sub-BN parameters if model contains Sub-BNs.\n Args:\n checkpoint_sd (OrderedDict): source dict of parameters.\n model_sd (OrderedDict): target dict of parameters.\n Returns:\n new_sd (OrderedDict): converted dict of parameters.\n \"\"\"\n for key in model_sd:\n if key not in checkpoint_sd:\n # not to replace bn.weight and bn.bias\n if \"bn.split_bn.\" in key and \"bn.weight\" not in key and \"bn.bias\" not in key:\n load_key = key.replace(\"bn.split_bn.\", \"bn.\")\n bn_key = key.replace(\"bn.split_bn.\", \"bn.bn.\")\n checkpoint_sd[key] = checkpoint_sd.pop(load_key)\n checkpoint_sd[bn_key] = checkpoint_sd[key]\n # match the shape of bn.split_bn._xx\n # model_sd: split_bn.rm.shape = num_feature*num_split"
+ },
+ {
+ "comment": "This code is comparing the shape of certain keys in the model and checkpoint dictionaries. If they match certain criteria, it will concatenate the checkpoint key to expand its size based on the model's shape. This is done for specific keys in the dictionary, except 'split_bn'. The function prints out the before and after shapes of the affected keys.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":58-80",
+ "content": " # checkpoint_sd: split_bn.rm.shape = bn.rm.shape = num_feature\n for key in model_sd:\n if key in checkpoint_sd:\n model_blob_shape = model_sd[key].shape #bn.split_bn\n c2_blob_shape = checkpoint_sd[key].shape #bn.bn\n if (len(model_blob_shape) == 1 and len(c2_blob_shape) == 1\n and model_blob_shape[0] > c2_blob_shape[0]\n and model_blob_shape[0] % c2_blob_shape[0] == 0):\n before_shape = checkpoint_sd[key].shape\n checkpoint_sd[key] = np.concatenate(\n [checkpoint_sd[key]] *\n (model_blob_shape[0] // c2_blob_shape[0]))\n if 'split_bn' not in key: #split_bn is excepted\n print(\"{} {} -> {}\".format(key, before_shape,\n checkpoint_sd[key].shape))\n return checkpoint_sd\ndef mapping_opt_dict(opt_dict, model_key_list):\n \"\"\"\n Paddle Name schedule: conv_1.w -> conv_2.w\n Sometimes: sub_bn -> bn"
+ },
+ {
+ "comment": "This function takes an optimizer state dict and a list of parameter names from a rebuilt model. It aims to modify the keys in the optimizer state dict to match the new parameters' names, while also considering any added index for better compatibility. The function then returns the modified optimizer state dict.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":81-102",
+ "content": " when re-build model, we desire the parameter name to be coincident,\n but the parameters name index will be added, as conv_1 to conv_2, not conv_1.\n It will raise error if we set old saved parameters to new created optimizer.\n as conv_2 cannot find in state_dict(only conv_1).\n Args:\n opt_dict: optimizer state dict, including the name and value of parameters gradient.\n model_key_list: the parameters name list of re-build model.\n Return: optimizer state dict with modified keys\n \"\"\"\n def get_name_info(PNAME, PN_key_list, key_list):\n min_index = float('inf')\n max_index = 0\n for name in PN_key_list[1:]:\n for key in key_list:\n if name in key:\n index = int(key.split('.')[0].split(name)[-1])\n if index < min_index:\n min_index = index\n if index > max_index:\n max_index = index\n num_name = max_index - min_index + 1\n PNAME[name].append((min_index, max_index, num_name))"
+ },
+ {
+ "comment": "This code appears to be a part of a larger program that compares the parameters in an optimizer state dict with those in a re-built model. It calculates and prints information about the number of parameters associated with each prefix, checks if batch normalization layers need their names changed, and potentially removes the \"sub_batch_norm3d_\" prefix from consideration. The code assumes that the \"opt_dict\" and \"model\" variables have already been defined elsewhere.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":103-134",
+ "content": " min_index = float('inf')\n max_index = 0\n PNAME = {\n \"LR_Scheduler\": [],\n \"conv3d_\": [],\n \"linear_\": [],\n \"sub_batch_norm3d_\": [],\n \"batch_norm3d_\": [],\n }\n pd_key_list = list(opt_dict.keys())\n print(\"The number of parameters in saved optimizer state dict = {}\".format(\n len(pd_key_list)))\n print(\"The number of parameters in re-build model list = {}\".format(\n len(model_key_list)))\n # 1 may be LR_Scheduler\n PN_key_list = list(PNAME.keys())\n # get the number of each PNAME\n get_name_info(PNAME, PN_key_list, pd_key_list)\n get_name_info(PNAME, PN_key_list, model_key_list)\n print(\"[Parameters info] prefix: min_index, max_index, number_params: \\n\",\n PNAME)\n # whether to change name of bn layer\n change_name = False\n if PNAME[\"sub_batch_norm3d_\"][0][-1] == -float('inf'):\n PN_key_list.remove(\"sub_batch_norm3d_\")\n if PNAME[\"sub_batch_norm3d_\"][1][-1] != -float('inf'):\n print(\n \"Optimizer state dict saved bn, but Re-build model use sub_bn, changed name!\""
+ },
+ {
+ "comment": "The code checks if the optimizer state dict saved batch normalization (bn) or sub_batch_normalization and updates the key names accordingly. If the state dict saved bn but the model uses sub_bn, it prints a message and changes the name. If the state dict saved sub_bn and the model also uses sub_bn, it prints a separate message. The code then defines a change_dict mapping and iterates over the key list to update the names if required.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":135-162",
+ "content": " )\n change_name = True\n else:\n print(\"Optimizer state dict saved bn, and Re-build model use bn\")\n else:\n PN_key_list.remove(\"batch_norm3d_\")\n if PNAME[\"sub_batch_norm3d_\"][1][-1] == -float('inf'):\n print(\n \"Optimizer state dict saved sub_bn, but Re-build model use bn, changed name!\"\n )\n change_name = True\n else:\n print(\n \"Optimizer state dict saved sub_bn, Re-build model use sub_bn\")\n #update key name\n # sub_bn -> bn name mapping, pre-define dict\n change_dict = {\n \"sub_batch_norm3d_\": \"batch_norm3d_\",\n \"batch_norm3d_\": \"sub_batch_norm3d_\"\n }\n for key in pd_key_list:\n for name in PN_key_list[1:]:\n if key.startswith(name):\n start = change_dict[name] if (\n change_name and \"batch_norm\" in name) else name\n str_index = key.split('.')[0].split(name)[-1]\n index = int(str_index)"
+ },
+ {
+ "comment": "This code defines two functions: \"subn_save\" and \"subn_load\". \"subn_save\" saves a model's state dictionary along with the optimizer's state dictionary to specified directories in a specific format. It also converts sub-bn to normal bn before saving, and prints a message confirming the save operation. \"subn_load\" loads checkpoints from given files into the specified model and optionally an optimizer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":163-189",
+ "content": " new_index = str(index +\n (PNAME[start][1][0] - PNAME[name][0][0]))\n end = key.split('.')[-1]\n update_key = start + new_index + '.' + end\n opt_dict[update_key] = opt_dict.pop(key)\n return opt_dict\ndef subn_save(save_dir, name_prefix, epoch, video_model, optimizer):\n if not os.path.isdir(save_dir):\n os.makedirs(save_dir)\n model_path = os.path.join(save_dir, name_prefix + \"{:05d}\".format(epoch))\n model_dict = video_model.state_dict()\n sub_to_normal_bn(model_dict)\n opti_dict = optimizer.state_dict()\n paddle.save(model_dict, model_path + '.pdparams')\n paddle.save(opti_dict, model_path + '.pdopt')\n print('[Saved Epoch {} parameters and optimizer state ]'.format(epoch))\ndef subn_load(model, ck_path, optimizer=None):\n \"\"\"\n Load the checkpoint from the given file.\n Args:\n model (model): model to load the weights from the checkpoint.\n optimizer (optim, optional): optimizer to load the historical state."
+ },
+ {
+ "comment": "This function loads checkpoints from a specific path and returns the number of training epochs. It ensures that the given directory has .pdparams file, prints the checkpoint loading information, copies model state dictionary, and compares the shapes of pre-trained weights to current model weights for matching. It also identifies layers that are not loaded with pre-trained weights.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":190-215",
+ "content": " ck_path (str): checkpoint path\n Returns:\n (int): the number of training epoch of the checkpoint.\n \"\"\"\n assert os.path.exists(ck_path + \".pdparams\"), \\\n \"Given dir {}.pdparams not exist.\".format(ck_path)\n print(\"load checkpint from {}.pdparams\".format(ck_path))\n model_dict = model.state_dict()\n checkpoint_dict = paddle.load(ck_path + \".pdparams\")\n # checkpoint_dict = copy.deepcopy(checkpoint_dict_orig) #not modify when multi card\n pre_train_dict = normal_to_sub_bn(checkpoint_dict, model_dict)\n # Match pre-trained weights that have same shape as current model.\n pre_train_dict_match = {\n k: v\n for k, v in pre_train_dict.items()\n if k in model_dict and tuple(v.shape) == tuple(model_dict[k].shape)\n }\n # Weights that do not have match from the pre-trained model.\n not_load_layers = [\n k for k in model_dict.keys() if k not in pre_train_dict_match.keys()\n ]\n # Log weights that are not loaded with the pre-trained weights."
+ },
+ {
+ "comment": "This code block checks if certain layers in the model are not set to load, and prints a message if those weights are not loaded. It then loads the pre-trained weights for the model and checks if a specific file exists before loading the optimizer's state dictionary from that file. The function mapping_opt_dict is called to create a new dictionary containing only parameters that require gradient, which is then set as the state dictionary of the optimizer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/save_load_helper.py\":216-236",
+ "content": " if not_load_layers:\n for k in not_load_layers:\n if 'bn.weight' not in k and 'bn.bias' not in k:\n print(\"Network weights {} not loaded.\".format(k))\n # Load pre-trained weights.\n model.set_state_dict(pre_train_dict_match)\n if optimizer:\n assert os.path.exists(ck_path + \".pdopt\"), \\\n \"Given dir {}.pdopt not exist.\".format(ck_path)\n print(\"load checkpint from {}.pdopt\".format(ck_path))\n opt_dict = paddle.load(ck_path + \".pdopt\")\n # get parameters that required gradient from re-build model\n model_key_list = []\n for param in model.parameters():\n if param.stop_gradient == False:\n model_key_list.append(param.name)\n new_opt_dict = mapping_opt_dict(opt_dict, model_key_list)\n optimizer.set_state_dict(new_opt_dict)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/69686180-705e-42c2-bf0c-84e8e04bb6bb.json b/docs/doc/69686180-705e-42c2-bf0c-84e8e04bb6bb.json
new file mode 100644
index 000000000..e02f77f78
--- /dev/null
+++ b/docs/doc/69686180-705e-42c2-bf0c-84e8e04bb6bb.json
@@ -0,0 +1,150 @@
+{
+ "summary": "The PaddleVideo library includes functions for convolutional layers, ResBlock classes, Slow and Fast branches, and a VideoModelStem class to initialize a ResNet SlowFast model with MRI for video analysis.",
+ "details": [
+ {
+ "comment": "This code is part of the PaddleVideo library, licensed under Apache 2.0. It imports necessary modules and defines functions for initializing convolutional layers with KaimingNormal distribution. It also includes a function for batch normalization parameters and registers backbones in PaddlePaddle's registry. The code uses paddle, numpy, and other libraries for various functionalities such as seeding random numbers for reproducibility.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":0-32",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddle.nn.initializer import KaimingNormal\nfrom ..registry import BACKBONES\nfrom paddlevideo.utils.multigrid import get_norm\nimport sys\nimport numpy as np\nimport paddle.distributed as dist\n# seed random seed\npaddle.framework.seed(0)\n# get init parameters for conv layer\ndef get_conv_init(fan_out):\n return KaimingNormal(fan_in=fan_out)\ndef get_bn_param_attr(bn_weight=1.0, coeff=0.0):"
+ },
+ {
+ "comment": "This code defines a BottleneckTransform class, which is a layer for video models. It performs temporal convolutions with 1x1, 1x3x3, and 1x1x1 layers, where T is the size of the temporal kernel. The class takes various parameters such as dimension, stride, and kernel sizes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":33-65",
+ "content": " param_attr = paddle.ParamAttr(\n initializer=paddle.nn.initializer.Constant(bn_weight),\n regularizer=paddle.regularizer.L2Decay(coeff))\n return param_attr\n\"\"\"Video models.\"\"\"\nclass BottleneckTransform(paddle.nn.Layer):\n \"\"\"\n Bottleneck transformation: Tx1x1, 1x3x3, 1x1x1, where T is the size of\n temporal kernel.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck."
+ },
+ {
+ "comment": "The code defines a class 'BottleneckTransform' with parameters for dimensions, stride, inner dimension, number of groups, and other attributes. It inherits from another class and initializes its own instance variables before constructing the model structure. The constructor method takes in various arguments to configure the bottleneck transformation block.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":66-86",
+ "content": " dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): if True, calculate the relu on the original\n input without allocating new memory.\n eps (float): epsilon for batch norm.\n dilation (int): size of dilation.\n \"\"\"\n super(BottleneckTransform, self).__init__()\n self.temp_kernel_size = temp_kernel_size\n self._inplace_relu = inplace_relu\n self._eps = eps\n self._stride_1x1 = stride_1x1\n self.norm_module = norm_module\n self._construct(dim_in, dim_out, stride, dim_inner, num_groups,\n dilation)\n def _construct(self, dim_in, dim_out, stride, dim_inner, num_groups,"
+ },
+ {
+ "comment": "This code initializes two 3D convolutional layers with Batch Normalization and ReLU activation. The first layer has stride 1 for all dimensions, while the second layer has stride 1 for the first dimension and a different value (determined by _stride_1x1) for the remaining two dimensions. Both layers have specified kernel sizes, padding, and use custom initializers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":87-112",
+ "content": " dilation):\n str1x1, str3x3 = (stride, 1) if self._stride_1x1 else (1, stride)\n fan = (dim_inner) * (self.temp_kernel_size * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.a = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_inner,\n kernel_size=[self.temp_kernel_size, 1, 1],\n stride=[1, str1x1, str1x1],\n padding=[int(self.temp_kernel_size // 2), 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.a_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x3x3, BN, ReLU.\n fan = (dim_inner) * (1 * 3 * 3)\n initializer_tmp = get_conv_init(fan)\n self.b = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_inner,"
+ },
+ {
+ "comment": "The code defines a Conv3D layer with 1x3x3 kernel, stride of str3x3, and dilation of dilation. It also includes a batch normalization (BN) module for the output. The BN module is applied to the output of the previous layer and has no bias. Finally, another Conv3D layer with 1x1x1 kernel and no BN is defined followed by another BN without a bias.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":113-138",
+ "content": " kernel_size=[1, 3, 3],\n stride=[1, str3x3, str3x3],\n padding=[0, dilation, dilation],\n groups=num_groups,\n dilation=[1, dilation, dilation],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.b_bn = self.norm_module(num_features=dim_inner,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n # 1x1x1, BN.\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.c = paddle.nn.Conv3D(\n in_channels=dim_inner,\n out_channels=dim_out,\n kernel_size=[1, 1, 1],\n stride=[1, 1, 1],\n padding=[0, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self.c_bn = self.norm_module(\n num_features=dim_out,"
+ },
+ {
+ "comment": "ResNetSlowFastMRI: A residual network model that adds Slow and Fast branches to extract temporal features. ResBlock is a residual block class used in the architecture, which utilizes BatchNorm3D for normalization and applies ReLU activations after each branch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":139-179",
+ "content": " epsilon=self._eps,\n weight_attr=get_bn_param_attr(bn_weight=0.0),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n # Branch2a.\n x = self.a(x)\n x = self.a_bn(x)\n x = F.relu(x)\n # Branch2b.\n x = self.b(x)\n x = self.b_bn(x)\n x = F.relu(x)\n # Branch2c\n x = self.c(x)\n x = self.c_bn(x)\n return x\nclass ResBlock(paddle.nn.Layer):\n \"\"\"\n Residual block.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups=1,\n stride_1x1=False,\n inplace_relu=True,\n eps=1e-5,\n dilation=1,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n ResBlock class constructs redisual blocks. More details can be found in:\n Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\n \"Deep residual learning for image recognition.\""
+ },
+ {
+ "comment": "This function defines a bottleneck for ResNet and ResNeXt-like networks with specified parameters. It takes channel dimensions, temporal kernel sizes, stride, transform function, inner dimension, number of groups, if applying stride to 1x1 conv, inplace relu calculation, and epsilon for batch norm as arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":180-197",
+ "content": " https://arxiv.org/abs/1512.03385\n Args:\n dim_in (int): the channel dimensions of the input.\n dim_out (int): the channel dimension of the output.\n temp_kernel_size (int): the temporal kernel sizes of the middle\n convolution in the bottleneck.\n stride (int): the stride of the bottleneck.\n trans_func (string): transform function to be used to construct the\n bottleneck.\n dim_inner (int): the inner dimension of the block.\n num_groups (int): number of groups for the convolution. num_groups=1\n is for standard ResNet like networks, and num_groups>1 is for\n ResNeXt like networks.\n stride_1x1 (bool): if True, apply stride to 1x1 conv, otherwise\n apply stride to the 3x3 conv.\n inplace_relu (bool): calculate the relu on the original input\n without allocating new memory.\n eps (float): epsilon for batch norm."
+ },
+ {
+ "comment": "This code defines a ResBlock class with skip connection, which performs convolution operations for image processing tasks. The constructor takes various parameters like dimensions, kernel size, stride, etc., and initializes the necessary components based on whether a skip connection is needed or not.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":198-236",
+ "content": " dilation (int): size of dilation.\n \"\"\"\n super(ResBlock, self).__init__()\n self._inplace_relu = inplace_relu\n self._eps = eps\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n # Use skip connection with projection if dim or res change.\n if (dim_in != dim_out) or (stride != 1):\n fan = (dim_out) * (1 * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self.branch1 = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=1,\n stride=[1, stride, stride],"
+ },
+ {
+ "comment": "This code defines a ResNet SlowFast MRI model with batch normalization (BN) for the branch1 and a BottleneckTransform layer for branch2. The forward function checks if \"branch1\" exists, suggesting it may be conditionally defined elsewhere in the codebase.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":237-259",
+ "content": " padding=0,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False,\n dilation=1)\n self.branch1_bn = self.norm_module(\n num_features=dim_out,\n epsilon=self._eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n self.branch2 = BottleneckTransform(dim_in,\n dim_out,\n temp_kernel_size,\n stride,\n dim_inner,\n num_groups,\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation,\n norm_module=self.norm_module)\n def forward(self, x):\n if hasattr(self, \"branch1\"):"
+ },
+ {
+ "comment": "This code defines a ResStage class for 3D ResNet, which can handle multi-pathway cases in SlowFast networks. It consists of one or more tensors as input. The stage includes operations such as adding tensors and applying ReLU activation. It also has a branch1 and branch2 for different paths, with batch normalization applied to the first path.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":260-293",
+ "content": " x1 = self.branch1(x)\n x1 = self.branch1_bn(x1)\n x2 = self.branch2(x)\n x = paddle.add(x=x1, y=x2)\n else:\n x2 = self.branch2(x)\n x = paddle.add(x=x, y=x2)\n x = F.relu(x)\n return x\nclass ResStage(paddle.nn.Layer):\n \"\"\"\n Stage of 3D ResNet. It expects to have one or more tensors as input for\n multi-pathway (SlowFast) cases. More details can be found here:\n Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n stride,\n temp_kernel_sizes,\n num_blocks,\n dim_inner,\n num_groups,\n num_block_temp_kernel,\n dilation,\n stride_1x1=False,\n inplace_relu=True,\n norm_module=paddle.nn.BatchNorm3D):"
+ },
+ {
+ "comment": "The given code is the initialization method of a ResStage class in PaddleVideo. It accepts arguments such as dim_in, dim_out, temp_kernel_sizes, stride, and num_blocks to build p pathways with different channel dimensions, temporal kernel sizes, strides, and numbers of blocks for each pathway.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":294-311",
+ "content": " \"\"\"\n The `__init__` method of any subclass should also contain these arguments.\n ResStage builds p streams, where p can be greater or equal to one.\n Args:\n dim_in (list): list of p the channel dimensions of the input.\n Different channel dimensions control the input dimension of\n different pathways.\n dim_out (list): list of p the channel dimensions of the output.\n Different channel dimensions control the input dimension of\n different pathways.\n temp_kernel_sizes (list): list of the p temporal kernel sizes of the\n convolution in the bottleneck. Different temp_kernel_sizes\n control different pathway.\n stride (list): list of the p strides of the bottleneck. Different\n stride control different pathway.\n num_blocks (list): list of p numbers of blocks for each of the\n pathway.\n dim_inner (list): list of the p inner channel dimensions of the"
+ },
+ {
+ "comment": "This function initializes a ResStage object, which is a layer of a network. It takes in parameters such as the number of blocks and temporal kernel sizes for each pathway, and asserts that the number of block_temp_kernel does not exceed the number of blocks. The temp_kernel_sizes are extended to num_block_temp_kernel blocks with temporal kernel size 1 for the rest of the layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":312-329",
+ "content": " input. Different channel dimensions control the input dimension\n of different pathways.\n num_groups (list): list of number of p groups for the convolution.\n num_groups=1 is for standard ResNet like networks, and\n num_groups>1 is for ResNeXt like networks.\n num_block_temp_kernel (list): extent the temp_kernel_sizes to\n num_block_temp_kernel blocks, then fill temporal kernel size\n of 1 for the rest of the layers.\n dilation (list): size of dilation for each pathway.\n \"\"\"\n super(ResStage, self).__init__()\n assert all((num_block_temp_kernel[i] <= num_blocks[i]\n for i in range(len(temp_kernel_sizes))))\n self.num_blocks = num_blocks\n self.temp_kernel_sizes = [\n (temp_kernel_sizes[i] * num_blocks[i])[:num_block_temp_kernel[i]] +\n [1] * (num_blocks[i] - num_block_temp_kernel[i])\n for i in range(len(temp_kernel_sizes))"
+ },
+ {
+ "comment": "The code creates an instance of a backbone network with adjustable pathways and blocks. It checks the input parameters' length, assigns the number of pathways, initializes a norm module, and calls a private method to construct the network structure. The _construct method loops through each pathway and block, creating ResBlock instances.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":330-371",
+ "content": " ]\n assert (len({\n len(dim_in),\n len(dim_out),\n len(temp_kernel_sizes),\n len(stride),\n len(num_blocks),\n len(dim_inner),\n len(num_groups),\n len(num_block_temp_kernel),\n }) == 1)\n self.num_pathways = len(self.num_blocks)\n self.norm_module = norm_module\n self._construct(\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n )\n def _construct(\n self,\n dim_in,\n dim_out,\n stride,\n dim_inner,\n num_groups,\n stride_1x1,\n inplace_relu,\n dilation,\n ):\n for pathway in range(self.num_pathways):\n for i in range(self.num_blocks[pathway]):\n res_block = ResBlock(\n dim_in[pathway] if i == 0 else dim_out[pathway],\n dim_out[pathway],\n self.temp_kernel_sizes[pathway][i],"
+ },
+ {
+ "comment": "This code defines a ResNet backbone with SlowFast pathways, including stem module and residual blocks. It initializes the layers for each pathway and then defines a forward function to process inputs through the specified number of pathways and blocks. The ResNetBasicStem performs spatiotemporal convolution, batch normalization, and ReLU before pooling in the 3D stem module.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":372-403",
+ "content": " stride[pathway] if i == 0 else 1,\n dim_inner[pathway],\n num_groups[pathway],\n stride_1x1=stride_1x1,\n inplace_relu=inplace_relu,\n dilation=dilation[pathway],\n norm_module=self.norm_module)\n self.add_sublayer(\"pathway{}_res{}\".format(pathway, i),\n res_block)\n def forward(self, inputs):\n output = []\n for pathway in range(self.num_pathways):\n x = inputs[pathway]\n for i in range(self.num_blocks[pathway]):\n m = getattr(self, \"pathway{}_res{}\".format(pathway, i))\n x = m(x)\n output.append(x)\n return output\nclass ResNetBasicStem(paddle.nn.Layer):\n \"\"\"\n ResNe(X)t 3D stem module.\n Performs spatiotemporal Convolution, BN, and Relu following by a\n spatiotemporal pooling.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,"
+ },
+ {
+ "comment": "This code defines a ResNetBasicStem class that initializes the stem of a ResNet network. The constructor takes parameters for kernel size, stride, padding, epsilon value, and norm module. The _construct_stem method creates a 3D convolutional layer with specified dimensions and uses an appropriate initializer for its weights. A batch normalization layer is also created using the provided norm module and epsilon value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":404-431",
+ "content": " kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n super(ResNetBasicStem, self).__init__()\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)\n def _construct_stem(self, dim_in, dim_out):\n fan = (dim_out) * (self.kernel[0] * self.kernel[1] * self.kernel[2])\n initializer_tmp = get_conv_init(fan)\n self._conv = paddle.nn.Conv3D(\n in_channels=dim_in,\n out_channels=dim_out,\n kernel_size=self.kernel,\n stride=self.stride,\n padding=self.padding,\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = self.norm_module(num_features=dim_out,\n epsilon=self.eps,\n weight_attr=get_bn_param_attr(),"
+ },
+ {
+ "comment": "This code defines a video stem module for slow and fast pathways, performing Conv, BN, ReLU, MaxPool operations on input data tensors. The function takes dim_in, dim_out, kernel, stride, padding, eps, and norm_module as arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":432-465",
+ "content": " bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x = self._conv(x)\n x = self._bn(x)\n x = F.relu(x)\n x = F.max_pool3d(x=x,\n kernel_size=[1, 3, 3],\n stride=[1, 2, 2],\n padding=[0, 1, 1],\n data_format=\"NCDHW\")\n return x\nclass VideoModelStem(paddle.nn.Layer):\n \"\"\"\n Video 3D stem module. Provides stem operations of Conv, BN, ReLU, MaxPool\n on input data tensor for slow and fast pathways.\n \"\"\"\n def __init__(self,\n dim_in,\n dim_out,\n kernel,\n stride,\n padding,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (list): the list of channel dimensions of the inputs.\n dim_out (list): the output dimension of the convolution in the stem\n layer.\n kernel (list): the kernels' size of the convolutions in the stem"
+ },
+ {
+ "comment": "The code defines a class VideoModelStem with parameters for dimensions, kernel size, stride, padding, and epsilon for batch normalization. It checks for consistent input pathway dimensions and initializes instance variables before constructing the stem layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":466-491",
+ "content": " layers. Temporal kernel size, height kernel size, width kernel\n size in order.\n stride (list): the stride sizes of the convolutions in the stem\n layer. Temporal kernel stride, height kernel size, width kernel\n size in order.\n padding (list): the paddings' sizes of the convolutions in the stem\n layer. Temporal padding size, height padding size, width padding\n size in order.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(VideoModelStem, self).__init__()\n assert (len({\n len(dim_in),\n len(dim_out),\n len(kernel),\n len(stride),\n len(padding),\n }) == 1), \"Input pathway dimensions are not consistent.\"\n self.num_pathways = len(dim_in)\n self.kernel = kernel\n self.stride = stride\n self.padding = padding\n self.eps = eps\n self.norm_module = norm_module\n self._construct_stem(dim_in, dim_out)"
+ },
+ {
+ "comment": "This code defines a class that constructs and fuses two pathways (slow and fast) in a video processing model. It initializes the stem layers for each pathway and then fuses the information from the fast pathway to the slow pathway. The input tensor should contain the specified number of pathways, and the output is returned as tensors from both pathways in order.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":493-517",
+ "content": " def _construct_stem(self, dim_in, dim_out):\n for pathway in range(len(dim_in)):\n stem = ResNetBasicStem(dim_in[pathway], dim_out[pathway],\n self.kernel[pathway], self.stride[pathway],\n self.padding[pathway], self.eps,\n self.norm_module)\n self.add_sublayer(\"pathway{}_stem\".format(pathway), stem)\n def forward(self, x):\n assert (len(x) == self.num_pathways\n ), \"Input tensor does not contain {} pathway\".format(\n self.num_pathways)\n for pathway in range(len(x)):\n m = getattr(self, \"pathway{}_stem\".format(pathway))\n x[pathway] = m(x[pathway])\n return x\nclass FuseFastToSlow(paddle.nn.Layer):\n \"\"\"\n Fuses the information from the Fast pathway to the Slow pathway. Given the\n tensors from Slow pathway and Fast pathway, fuse information from Fast to\n Slow, then return the fused tensors from Slow and Fast pathway in order."
+ },
+ {
+ "comment": "The code is initializing a class called FuseFastToSlow with parameters for input channel dimension, fusion convolution channel ratio, fusion kernel size, and frame rate ratio. It sets the number of channels in the 3D Convolution layer for fusing frames from Fast to Slow pathways based on given ratios. The fusion operation is performed using a Conv3D layer initialized with the given fan value.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":518-543",
+ "content": " \"\"\"\n def __init__(self,\n dim_in,\n fusion_conv_channel_ratio,\n fusion_kernel,\n alpha,\n fuse_bn_relu=1,\n eps=1e-5,\n norm_module=paddle.nn.BatchNorm3D):\n \"\"\"\n Args:\n dim_in (int): the channel dimension of the input.\n fusion_conv_channel_ratio (int): channel ratio for the convolution\n used to fuse from Fast pathway to Slow pathway.\n fusion_kernel (int): kernel size of the convolution used to fuse\n from Fast pathway to Slow pathway.\n alpha (int): the frame rate ratio between the Fast and Slow pathway.\n eps (float): epsilon for batch norm.\n \"\"\"\n super(FuseFastToSlow, self).__init__()\n self.fuse_bn_relu = fuse_bn_relu\n fan = (dim_in * fusion_conv_channel_ratio) * (fusion_kernel * 1 * 1)\n initializer_tmp = get_conv_init(fan)\n self._conv_f2s = paddle.nn.Conv3D(\n in_channels=dim_in,"
+ },
+ {
+ "comment": "This code defines a ResNetSlowFast_MRI model, which is a type of SlowFast network. It includes a fusion convolution layer and a batch normalization (BN) layer with optional ReLU activation after the fusion convolution. The forward method combines input x_s and x_f using concat before returning both inputs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":544-571",
+ "content": " out_channels=dim_in * fusion_conv_channel_ratio,\n kernel_size=[fusion_kernel, 1, 1],\n stride=[alpha, 1, 1],\n padding=[fusion_kernel // 2, 0, 0],\n weight_attr=paddle.ParamAttr(initializer=initializer_tmp),\n bias_attr=False)\n self._bn = norm_module(num_features=dim_in * fusion_conv_channel_ratio,\n epsilon=eps,\n weight_attr=get_bn_param_attr(),\n bias_attr=get_bn_param_attr(bn_weight=0.0))\n def forward(self, x):\n x_s = x[0]\n x_f = x[1]\n fuse = self._conv_f2s(x_f)\n # TODO: For AVA, set fuse_bn_relu=1, check mAP's improve.\n if self.fuse_bn_relu:\n fuse = self._bn(fuse)\n fuse = F.relu(fuse)\n x_s_fuse = paddle.concat(x=[x_s, fuse], axis=1, name=None)\n return [x_s_fuse, x_f]\n@BACKBONES.register()\nclass ResNetSlowFast_MRI(paddle.nn.Layer):\n \"\"\"\n SlowFast model builder for SlowFast network."
+ },
+ {
+ "comment": "This code initializes a ResNetSlowFast_MRI model with specified parameters, including alpha and beta values for the network architecture. The class extends from an existing superclass and includes properties like the norm module type, number of pathways, depth, group numbers, input channel numbers, width per group, fusion convolution channel ratio, pool size ratios, whether to use a pooling average operation at spatial stride 2, and spatial strides. The class also initializes these specified attributes for the model configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":573-606",
+ "content": " Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.\n \"Slowfast networks for video recognition.\"\n https://arxiv.org/pdf/1812.03982.pdf\n \"\"\"\n def __init__(\n self,\n alpha,\n beta,\n bn_norm_type=\"batchnorm\",\n bn_num_splits=1,\n num_pathways=2,\n depth=50,\n num_groups=1,\n input_channel_num=[1, 1],\n width_per_group=64,\n fusion_conv_channel_ratio=2,\n fusion_kernel_sz=7, #5?\n pool_size_ratio=[[1, 1, 1], [1, 1, 1]],\n fuse_bn_relu=1,\n spatial_strides=[[1, 1], [2, 2], [2, 2], [2, 2]],\n use_pool_af_s2=1,\n ):\n \"\"\"\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n super(ResNetSlowFast_MRI, self).__init__()\n self.alpha = alpha #8\n self.beta = beta #8\n self.norm_module = get_norm(bn_norm_type, bn_num_splits)\n self.num_pathways = num_pathways\n self.depth = depth"
+ },
+ {
+ "comment": "The code defines a SlowFast model with separate slow and fast pathways. The constructor sets parameters like input channel number, group number, fusion convolution channel ratio, and more. It also includes functions to build the network structure. The temporal kernels for each layer of both pathways are defined within the code.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":607-631",
+ "content": " self.num_groups = num_groups\n self.input_channel_num = input_channel_num\n self.width_per_group = width_per_group\n self.fusion_conv_channel_ratio = fusion_conv_channel_ratio\n self.fusion_kernel_sz = fusion_kernel_sz # NOTE: modify to 7 in 8*8, 5 in old implement\n self.pool_size_ratio = pool_size_ratio\n self.fuse_bn_relu = fuse_bn_relu\n self.spatial_strides = spatial_strides\n self.use_pool_af_s2 = use_pool_af_s2\n self._construct_network()\n def _construct_network(self):\n \"\"\"\n Builds a SlowFast model.\n The first pathway is the Slow pathway\n and the second pathway is the Fast pathway.\n Args:\n cfg (CfgNode): model building configs, details are in the\n comments of the config file.\n \"\"\"\n temp_kernel = [\n [[1], [5]], # conv1 temporal kernel for slow and fast pathway.\n [[1], [3]], # res2 temporal kernel for slow and fast pathway.\n [[1], [3]], # res3 temporal kernel for slow and fast pathway."
+ },
+ {
+ "comment": "This code defines a ResNet backbone for the MRI dataset. It sets the temporal kernels for res4 and res5 pathways, initializes the video model stem (s1) with specified dimensions and stride, adds a fuseFastToSlow module for fusion, and defines model stage depth based on the chosen depth of ResNet.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":632-656",
+ "content": " [[3], [3]], # res4 temporal kernel for slow and fast pathway.\n [[3], [3]],\n ] # res5 temporal kernel for slow and fast pathway.\n self.s1 = VideoModelStem(\n dim_in=self.input_channel_num,\n dim_out=[self.width_per_group, self.width_per_group // self.beta],\n kernel=[temp_kernel[0][0] + [7, 7], temp_kernel[0][1] + [7, 7]],\n stride=[[1, 2, 2]] * 2,\n padding=[\n [temp_kernel[0][0][0] // 2, 3, 3],\n [temp_kernel[0][1][0] // 2, 3, 3],\n ],\n norm_module=self.norm_module)\n self.s1_fuse = FuseFastToSlow(\n dim_in=self.width_per_group // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu)\n # ResNet backbone\n MODEL_STAGE_DEPTH = {50: (3, 4, 6, 3)}\n (d2, d3, d4, d5) = MODEL_STAGE_DEPTH[self.depth]"
+ },
+ {
+ "comment": "The code is defining the parameters for a ResStage layer in a residual network model. It sets the input and output dimensions, inner dimensions, temporal kernel sizes, and stride values based on previously defined values from the function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":658-677",
+ "content": " num_block_temp_kernel = [[3, 3], [4, 4], [6, 6], [3, 3]]\n spatial_dilations = [[1, 1], [1, 1], [1, 1], [1, 1]]\n spatial_strides = self.spatial_strides\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [2, 2]]\n #spatial_strides = [[1, 1], [2, 2], [2, 2], [1, 1]] #TODO:check which value is FAIR's impliment\n out_dim_ratio = self.beta // self.fusion_conv_channel_ratio #4\n dim_inner = self.width_per_group * self.num_groups #64\n self.s2 = ResStage(dim_in=[\n self.width_per_group + self.width_per_group // out_dim_ratio,\n self.width_per_group // self.beta,\n ],\n dim_out=[\n self.width_per_group * 4,\n self.width_per_group * 4 // self.beta,\n ],\n dim_inner=[dim_inner, dim_inner // self.beta],\n temp_kernel_sizes=temp_kernel[1],\n stride=spatial_strides[0],"
+ },
+ {
+ "comment": "The code initializes a ResNet SlowFast model with multiple layers and parameters. It creates two branches (slow and fast) for the network, each with its own set of layers and parameters. The slow branch has 2x more blocks than the fast branch, and both branches have identical group numbers and dilation rates. The code also initializes a fusing layer that combines features from the fast and slow branches and a stage for the third level of the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":678-703",
+ "content": " num_blocks=[d2] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[0],\n dilation=spatial_dilations[0],\n norm_module=self.norm_module)\n self.s2_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 4 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s3 = ResStage(\n dim_in=[\n self.width_per_group * 4 +\n self.width_per_group * 4 // out_dim_ratio,\n self.width_per_group * 4 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 8,\n self.width_per_group * 8 // self.beta,\n ],\n dim_inner=[dim_inner * 2, dim_inner * 2 // self.beta],"
+ },
+ {
+ "comment": "This code is initializing a ResNet SlowFast model for MRI. It creates an instance of the class, sets parameters like kernel sizes, strides, block numbers, group numbers, dilation rates, and normalization module. The model consists of multiple stages, including s1, s2, s3_fuse, and s4, each with different dimensions, inner dimensions, and configurations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":704-732",
+ "content": " temp_kernel_sizes=temp_kernel[2],\n stride=spatial_strides[1],\n num_blocks=[d3] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[1],\n dilation=spatial_dilations[1],\n norm_module=self.norm_module,\n )\n self.s3_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 8 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s4 = ResStage(\n dim_in=[\n self.width_per_group * 8 +\n self.width_per_group * 8 // out_dim_ratio,\n self.width_per_group * 8 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 16,\n self.width_per_group * 16 // self.beta,\n ],\n dim_inner=[dim_inner * 4, dim_inner * 4 // self.beta],"
+ },
+ {
+ "comment": "This code defines a ResNet SlowFast model with MRI for video analysis. It includes creating layers for stage 4, fusing fast and slow features, and defining the stage 5 layer with specific dimensions for input, output, and inner dimensions. The model utilizes group width, out_dim_ratio, beta, and dim_inner parameters to control its behavior.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":733-761",
+ "content": " temp_kernel_sizes=temp_kernel[3],\n stride=spatial_strides[2],\n num_blocks=[d4] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[2],\n dilation=spatial_dilations[2],\n norm_module=self.norm_module,\n )\n self.s4_fuse = FuseFastToSlow(\n dim_in=self.width_per_group * 16 // self.beta,\n fusion_conv_channel_ratio=self.fusion_conv_channel_ratio,\n fusion_kernel=self.fusion_kernel_sz,\n alpha=self.alpha,\n norm_module=self.norm_module,\n fuse_bn_relu=self.fuse_bn_relu,\n )\n self.s5 = ResStage(\n dim_in=[\n self.width_per_group * 16 +\n self.width_per_group * 16 // out_dim_ratio,\n self.width_per_group * 16 // self.beta,\n ],\n dim_out=[\n self.width_per_group * 32,\n self.width_per_group * 32 // self.beta,\n ],\n dim_inner=[dim_inner * 8, dim_inner * 8 // self.beta],"
+ },
+ {
+ "comment": "This code initializes a 3D ResNet SlowFast model, sets the weights, and applies several stages of convolutions and fusions to process video input. The forward function sequentially passes the input through multiple stages, potentially applying max pooling for AVA if use_pool_af_s2 is True.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":762-792",
+ "content": " temp_kernel_sizes=temp_kernel[4],\n stride=spatial_strides[3],\n num_blocks=[d5] * 2,\n num_groups=[self.num_groups] * 2,\n num_block_temp_kernel=num_block_temp_kernel[3],\n dilation=spatial_dilations[3],\n norm_module=self.norm_module,\n )\n def init_weights(self):\n pass\n def forward(self, x):\n x = self.s1(x) #VideoModelStem\n x = self.s1_fuse(x) #FuseFastToSlow\n x = self.s2(x) #ResStage\n x = self.s2_fuse(x)\n # TODO: For AVA, set use_pool_af_s2=1, check mAP's improve.\n if self.use_pool_af_s2:\n for pathway in range(self.num_pathways):\n x[pathway] = F.max_pool3d(\n x=x[pathway],\n kernel_size=self.pool_size_ratio[pathway],\n stride=self.pool_size_ratio[pathway],\n padding=[0, 0, 0],\n data_format=\"NCDHW\")\n x = self.s3(x)\n x = self.s3_fuse(x)\n x = self.s4(x)"
+ },
+ {
+ "comment": "This code snippet is part of a ResNet SlowFast model. It fuses the outputs from previous stages (s4), passes them through another stage (s5) and returns the result as output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_slowfast_MRI.py\":793-795",
+ "content": " x = self.s4_fuse(x)\n x = self.s5(x)\n return x"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/697c842e-2194-4f41-9c5e-8c5b11a49a12.json b/docs/doc/697c842e-2194-4f41-9c5e-8c5b11a49a12.json
new file mode 100644
index 000000000..7bfe9e679
--- /dev/null
+++ b/docs/doc/697c842e-2194-4f41-9c5e-8c5b11a49a12.json
@@ -0,0 +1,70 @@
+{
+ "summary": "The code initializes a ResNet-TSM backbone with convolutional layers and bottleneck blocks, but may be deprecated and needs data preparation changes for better compatibility.",
+ "details": [
+ {
+ "comment": "This code snippet defines a ConvBNLayer class which combines a Conv2D and BatchNorm2D layer. It is imported from paddle.nn and will be used for creating convolutional neural network layers in the PaddlePaddle framework. The layer can be utilized to process image data or other types of spatial input data by applying convolution operations followed by batch normalization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":0-29",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nimport paddle.nn as nn\nfrom paddle.nn import (Conv2D, BatchNorm2D, Linear, Dropout, MaxPool2D,\n AvgPool2D)\nfrom paddle import ParamAttr\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt\nclass ConvBNLayer(nn.Layer):\n \"\"\"Conv2D and BatchNorm2D layer.\n Args:"
+ },
+ {
+ "comment": "This code defines a class called ConvBNLayer which inherits from an unspecified parent class. It takes in parameters such as number of input and output channels, kernel size, stride, groups, activation function, name, and data format for the Conv2D layer. The class initializes a Conv2D layer using these parameters and adds BatchNorm2D and ReLU layers after it. Weight and bias initialization values are named in the restore parameters, and they are explicitly declared in the init_weights method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":30-52",
+ "content": " in_channels (int): Number of channels for the input.\n out_channels (int): Number of channels for the output.\n kernel_size (int): Kernel size.\n stride (int): Stride in the Conv2D layer. Default: 1.\n groups (int): Groups in the Conv2D, Default: 1.\n act (str): Indicate activation after BatchNorm2D layer.\n name (str): the name of an instance of ConvBNLayer.\n Note: weight and bias initialization include initialize values and name the restored parameters, values initialization are explicit declared in the ```init_weights``` method.\n \"\"\"\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n act=None,\n name=None,\n data_format=\"NCHW\"):\n super(ConvBNLayer, self).__init__()\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,"
+ },
+ {
+ "comment": "This code defines a BottleneckBlock class with a convolution layer, batch normalization, and optional activation function. The forward pass applies the convolution, batch normalization, and activation if present.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":53-83",
+ "content": " stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\"),\n bias_attr=False,\n data_format=data_format)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._act = act\n self._batch_norm = BatchNorm2D(\n out_channels,\n weight_attr=ParamAttr(name=bn_name + \"_scale\",\n regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(name=bn_name + \"_offset\",\n regularizer=L2Decay(0.0)),\n data_format=data_format)\n def forward(self, inputs):\n y = self._conv(inputs)\n y = self._batch_norm(y)\n if self._act:\n y = getattr(paddle.nn.functional, self._act)(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,"
+ },
+ {
+ "comment": "This code defines a class called BottleneckBlock, which is a part of a neural network model. It contains several ConvBNLayer objects for processing input data, with different parameters such as in_channels, out_channels, kernel_size, stride, and act (activation function). The class also has an attribute for data_format and initializes the ConvBNLayer objects with specific names.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":84-107",
+ "content": " in_channels,\n out_channels,\n stride,\n shortcut=True,\n num_seg=8,\n name=None,\n data_format=\"NCHW\"):\n super(BottleneckBlock, self).__init__()\n self.data_format = data_format\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act=\"relu\",\n name=name + \"_branch2a\",\n data_format=data_format)\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act=\"relu\",\n name=name + \"_branch2b\",\n data_format=data_format)\n self.conv2 = ConvBNLayer(in_channels=out_channels,"
+ },
+ {
+ "comment": "This code is initializing a ConvBNLayer and a shortcut connection for the TSM backbone. The layers have specific out_channels, kernel_size, stride, name, and data_format configurations. If a shortcut is not provided, it initializes another ConvBNLayer. The forward function reshapes input to have a shape of [N, T, C, H, W] for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":108-133",
+ "content": " out_channels=out_channels * 4,\n kernel_size=1,\n act=None,\n name=name + \"_branch2c\",\n data_format=data_format)\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=stride,\n name=name + \"_branch1\",\n data_format=data_format)\n self.shortcut = shortcut\n self.num_seg = num_seg\n def forward(self, inputs):\n if paddle.is_compiled_with_custom_device('npu'):\n x = inputs\n seg_num = self.num_seg\n shift_ratio = 1.0 / self.num_seg\n shape = x.shape #[N*T, C, H, W]\n reshape_x = x.reshape(\n (-1, seg_num, shape[1], shape[2], shape[3])) #[N, T, C, H, W]"
+ },
+ {
+ "comment": "This code performs temporal shift operation on the input tensor. If a certain condition is met, it pads and concatenates slices of the input tensor before performing the reshape and temporal shift operations. The resulting output is then passed through several convolutional layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":134-163",
+ "content": " pad_x = F.pad(reshape_x, [\n 0,\n 0,\n 1,\n 1,\n 0,\n 0,\n 0,\n 0,\n 0,\n 0,\n ]) #[N, T+2, C, H, W]\n c1 = int(shape[1] * shift_ratio)\n c2 = int(shape[1] * 2 * shift_ratio)\n slice1 = pad_x[:, :seg_num, :c1, :, :]\n slice2 = pad_x[:, 2:seg_num + 2, c1:c2, :, :]\n slice3 = pad_x[:, 1:seg_num + 1, c2:, :, :]\n concat_x = paddle.concat([slice1, slice2, slice3],\n axis=2) #[N, T, C, H, W]\n shifts = concat_x.reshape(shape)\n else:\n shifts = F.temporal_shift(inputs,\n self.num_seg,\n 1.0 / self.num_seg,\n data_format=self.data_format)\n y = self.conv0(shifts)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:"
+ },
+ {
+ "comment": "The code defines a BasicBlock class which is a residual block. It contains two 3x3 convolutional layers followed by BN and ReLU activations. If the shortcut connection is not used, it also includes an additional convolution layer for the shortcut path. The purpose of this residual block is to alleviate the problem of vanishing gradients in deeper networks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":164-201",
+ "content": " short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n return F.relu(y)\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,\n name=None,\n data_format=\"NCHW\"):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(\n in_channels=in_channels,\n out_channels=out_channels,\n filter_size=3,\n stride=stride,\n act=\"relu\",\n name=name + \"_branch2a\",\n data_format=data_format,\n )\n self.conv1 = ConvBNLayer(\n in_channels=out_channels,\n out_channels=out_channels,\n filter_size=3,\n act=None,\n name=name + \"_branch2b\",\n data_format=data_format,\n )\n if not shortcut:\n self.short = ConvBNLayer(\n in_channels=in_channels,"
+ },
+ {
+ "comment": "The code defines a ResNet TSM backbone model with specified depth and data format. It consists of an initialization, a forward function for processing inputs, and the ability to be registered at BACKBONES. It also supports different layers like 18, 34, 50, 101, and 152.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":202-240",
+ "content": " out_channels=out_channels,\n filter_size=1,\n stride=stride,\n name=name + \"_branch1\",\n data_format=data_format,\n )\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(short, conv1)\n y = F.relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTSM(nn.Layer):\n \"\"\"ResNet TSM backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self, depth, num_seg=8, data_format=\"NCHW\", pretrained=None):\n super(ResNetTSM, self).__init__()\n self.pretrained = pretrained\n self.layers = depth\n self.num_seg = num_seg\n self.data_format = data_format\n supported_layers = [18, 34, 50, 101, 152]\n assert self.layers in supported_layers, \\"
+ },
+ {
+ "comment": "This code initializes a ResNet-TSM backbone with different depth configurations based on the input layers. It includes a convolution layer, max pooling 2D layer, and a block list for deeper networks. The code checks if the layers are supported (18, 34, 50, 101, or 152) and assigns corresponding depth and number of channels accordingly.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":241-272",
+ "content": " \"supported layers are {} but input layer is {}\".format(\n supported_layers, self.layers)\n if self.layers == 18:\n depth = [2, 2, 2, 2]\n elif self.layers == 34 or self.layers == 50:\n depth = [3, 4, 6, 3]\n elif self.layers == 101:\n depth = [3, 4, 23, 3]\n elif self.layers == 152:\n depth = [3, 8, 36, 3]\n in_channels = 64\n out_channels = [64, 128, 256, 512]\n self.conv = ConvBNLayer(in_channels=3,\n out_channels=64,\n kernel_size=7,\n stride=2,\n act=\"relu\",\n name=\"conv1\",\n data_format=self.data_format)\n self.pool2D_max = MaxPool2D(\n kernel_size=3,\n stride=2,\n padding=1,\n data_format=self.data_format,\n )\n self.block_list = []\n if self.layers >= 50:\n for block in range(len(depth)):"
+ },
+ {
+ "comment": "Code creates bottleneck blocks for ResNet TSM architecture, varying the number of input channels based on layer index and configuration. It adds sublayers with specified parameters including number of segments and stride for each block.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":273-292",
+ "content": " shortcut = False\n for i in range(depth[block]):\n if self.layers in [101, 152] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n conv_name,\n BottleneckBlock(\n in_channels=in_channels\n if i == 0 else out_channels[block] * 4,\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n num_seg=self.num_seg,\n shortcut=shortcut,\n name=conv_name,\n data_format=self.data_format))"
+ },
+ {
+ "comment": "Code initializes a ResNet TSM model backbone, with block-specific in_channels and adds either bottleneck or basic blocks depending on the depth configuration. Init_weights function is also defined to initialize weights for the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":293-315",
+ "content": " in_channels = out_channels[block] * 4\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n conv_name,\n BasicBlock(\n in_channels=in_channels[block]\n if i == 0 else out_channels[block],\n out_channels=out_channels[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n name=conv_name,\n data_format=self.data_format,\n ))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):"
+ },
+ {
+ "comment": "This code initializes parameters for a ResNet TSM backbone. If a pretrained loading path is provided, it loads the weights from that path; otherwise, it uses specific initialization functions for Conv2D and BatchNorm2d layers. No bias is used in Conv2D layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":316-331",
+ "content": " \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n #XXX: check bias!!! check pretrained!!!\n if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n #XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):"
+ },
+ {
+ "comment": "This code defines a forward function for a backbone model. It uses convolution and pooling layers to extract features from input data. The comments indicate that this implementation may be deprecated, and the data preparation should be modified according to recognizer2d.py for better compatibility with paddlepaddle's to_static method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsm.py\":332-352",
+ "content": " weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n \"\"\"Define how the backbone is going to run.\n \"\"\"\n #NOTE: (deprecated design) Already merge axis 0(batches) and axis 1(clips) before extracting feature phase,\n # please refer to paddlevideo/modeling/framework/recognizers/recognizer2d.py#L27\n #y = paddle.reshape(\n # inputs, [-1, inputs.shape[2], inputs.shape[3], inputs.shape[4]])\n #NOTE: As paddlepaddle to_static method need a \"pure\" model to trim. It means from\n # 1. the phase of generating data[images, label] from dataloader\n # to\n # 2. last layer of a model, always is FC layer\n y = self.conv(inputs)\n y = self.pool2D_max(y)\n for block in self.block_list:\n y = block(y)\n return y"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/69888b77-5f29-424b-a6fd-2c258d265b45.json b/docs/doc/69888b77-5f29-424b-a6fd-2c258d265b45.json
new file mode 100644
index 000000000..49ec046fd
--- /dev/null
+++ b/docs/doc/69888b77-5f29-424b-a6fd-2c258d265b45.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code defines a SkeletonDataset class for action recognition, loading skeleton features and applying normalization operations. It imports libraries, registers the dataset, includes a logger, and has a class for loading skeleton data with optional label path and test mode parameter. The class loads and returns data for training or testing, preparing features based on training/testing needs and considering labels if available.",
+ "details": [
+ {
+ "comment": "This code defines a SkeletonDataset class for action recognition. It loads skeleton features and applies normalization operations. It also imports necessary libraries, registers the dataset with DATASETS, and includes a logger for logging purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/skeleton.py\":0-33",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport pickle\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass SkeletonDataset(BaseDataset):\n \"\"\"\n Skeleton dataset for action recognition.\n The dataset loads skeleton feature, and apply norm operatations.\n Args:\n file_path (str): Path to the index file."
+ },
+ {
+ "comment": "This code defines a class for loading skeleton data. It takes file path, pipeline, and label path (optional) as input parameters. It also has an optional test mode parameter. The `__init__` method initializes the class with provided parameters. The `load_file` method loads feature files to get skeleton information and handles different file types for labels. If a label path is given and it ends with 'npy' or 'pkl', it will load the label; otherwise, it just outputs predictions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/skeleton.py\":34-54",
+ "content": " pipeline(obj): Define the pipeline of data preprocessing.\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n \"\"\"\n def __init__(self, file_path, pipeline, label_path=None, test_mode=False):\n self.label_path = label_path\n super().__init__(file_path, pipeline, test_mode=test_mode)\n def load_file(self):\n \"\"\"Load feature file to get skeleton information.\"\"\"\n logger.info(\"Loading data, it will take some moment...\")\n self.data = np.load(self.file_path)\n if self.label_path:\n if self.label_path.endswith('npy'):\n self.label = np.load(self.label_path)\n elif self.label_path.endswith('pkl'):\n with open(self.label_path, 'rb') as f:\n sample_name, self.label = pickle.load(f)\n else:\n logger.info(\n \"Label path not provided when test_mode={}, here just output predictions.\""
+ },
+ {
+ "comment": "The code defines a class for loading, preparing, and returning data for training or testing. The `__getitem__` method loads the data and returns it when accessed by index. The `prepare_train` method prepares the feature for training/validation given an index. The `prepare_test` method prepares the feature for testing given an index, considering label if available.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/skeleton.py\":55-77",
+ "content": " .format(self.test_mode))\n logger.info(\"Data Loaded!\")\n return self.data # used for __len__\n def prepare_train(self, idx):\n \"\"\"Prepare the feature for training/valid given index. \"\"\"\n results = dict()\n results['data'] = copy.deepcopy(self.data[idx])\n results['label'] = copy.deepcopy(self.label[idx])\n results = self.pipeline(results)\n return results['data'], results['label']\n def prepare_test(self, idx):\n \"\"\"Prepare the feature for test given index. \"\"\"\n results = dict()\n results['data'] = copy.deepcopy(self.data[idx])\n if self.label_path:\n results['label'] = copy.deepcopy(self.label[idx])\n results = self.pipeline(results)\n return results['data'], results['label']\n else:\n results = self.pipeline(results)\n return [results['data']]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/69960b55-1729-4511-930a-f4f153b795a5.json b/docs/doc/69960b55-1729-4511-930a-f4f153b795a5.json
new file mode 100644
index 000000000..8b369b9a9
--- /dev/null
+++ b/docs/doc/69960b55-1729-4511-930a-f4f153b795a5.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is a module for the PaddleVideo package that includes various utility functions and classes, such as Registry, build_utils, config, logger, record, dist_utils, save_load, and precise_bn. It also defines __all__ to include Registry and build.",
+ "details": [
+ {
+ "comment": "This code is a module for the PaddleVideo package that includes various utility functions and classes, such as Registry, build_utils, config, logger, record, dist_utils, save_load, and precise_bn. It also defines __all__ to include Registry and build.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/utils/__init__.py\":0-24",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .registry import Registry\nfrom .build_utils import build\nfrom .config import *\nfrom .logger import setup_logger, coloring, get_logger\nfrom .record import AverageMeter, build_record, build_rec_record, log_batch, log_epoch\nfrom .dist_utils import get_dist_info, main_only\nfrom .save_load import save, load, load_ckpt, mkdir\nfrom .precise_bn import do_preciseBN\n__all__ = ['Registry', 'build']"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/69a563ed-fcd4-4c89-b365-43dad88d4cc6.json b/docs/doc/69a563ed-fcd4-4c89-b365-43dad88d4cc6.json
new file mode 100644
index 000000000..a00a4cd63
--- /dev/null
+++ b/docs/doc/69a563ed-fcd4-4c89-b365-43dad88d4cc6.json
@@ -0,0 +1,10 @@
+{
+ "summary": "Code file \"preprocess.py\" contains four functions: \n1. ffmpeg_frames extracts frames from a video using ffmpeg and saves them as jpg files in a specified folder, at the specified frame rate.\n2. ffmpeg_pcm extracts audio from a video and saves it as a PCM file.\n3. ffmpeg_mp4 downloads a video file from a URL to the local machine.\n4. get_images retrieves all image files in a directory, sorts them, and stores their paths in a list.",
+ "details": [
+ {
+ "comment": "Code file \"preprocess.py\" contains four functions: \n1. ffmpeg_frames extracts frames from a video using ffmpeg and saves them as jpg files in a specified folder, at the specified frame rate.\n2. ffmpeg_pcm extracts audio from a video and saves it as a PCM file.\n3. ffmpeg_mp4 downloads a video file from a URL to the local machine.\n4. get_images retrieves all image files in a directory, sorts them, and stores their paths in a list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/utils/preprocess.py\":0-35",
+ "content": "\"\"\" extract frames and pcm\"\"\"\nimport os\nimport sys\nimport shutil\ndef ffmpeg_frames(mp4_addr, frame_out_folder, fps=5):\n \"\"\"ffmpeg_frames\"\"\"\n if os.path.exists(frame_out_folder):\n shutil.rmtree(frame_out_folder)\n os.makedirs(frame_out_folder)\n cmd = './src/utils/ffmpeg -v 0 -i %s -r %d -q 0 %s/%s.jpg' % (\n mp4_addr, fps, frame_out_folder, '%08d')\n os.system(cmd)\ndef ffmpeg_pcm(mp4_addr, save_file_name):\n \"\"\"ffmpeg_pcm\"\"\"\n cmd = './src/utils/ffmpeg -y -i %s -acodec pcm_s16le -f s16le -ac 1 -ar 16000 %s -v 0' \\\n % (mp4_addr, save_file_name)\n os.system(cmd)\ndef ffmpeg_mp4(mp4_url, mp4_addr):\n \"\"\"ffmpeg_mp4\"\"\"\n cmd = \"wget %s -O %s -q\" % (mp4_url, mp4_addr)\n print(\"cmd = \", cmd)\n os.system(cmd)\ndef get_images(image_path):\n \"\"\"get_images\"\"\"\n images = sorted(os.listdir(image_path))\n images = images\n images_path_list = [image_path + '/' + im for im in images]\n return images_path_list"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6a1dd251-9bc5-473b-b8bc-e9eca8392b41.json b/docs/doc/6a1dd251-9bc5-473b-b8bc-e9eca8392b41.json
new file mode 100644
index 000000000..0c95dc832
--- /dev/null
+++ b/docs/doc/6a1dd251-9bc5-473b-b8bc-e9eca8392b41.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The AttentionLSTM model is presented, using LSTMs and an Attention layer to weigh frame features. The code trains and tests on YouTube-8M with PaddleVideo, exporting the model for classification. It accurately predicts top-1 class 11 with 0.9841 confidence.",
+ "details": [
+ {
+ "comment": "This code introduces the AttentionLSTM model, which utilizes two-way LSTMs to encode all video frame features and adds an Attention layer for adaptive weighting. This improves upon traditional methods by linearly weighing final feature vectors based on hidden state outputs at each moment.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/attention_lstm.md\":0-18",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/recognition/attention_lstm.md) | English\n# AttentionLSTM\n## content\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Inference](#Inference)\n- [Reference](#Reference)\n## Introduction\nRecurrent Neural Networks (RNN) are often used in the processing of sequence data, which can model the sequence information of multiple consecutive frames of video, and are commonly used methods in the field of video classification.\nThis model uses a two-way long and short-term memory network (LSTM) to encode all the frame features of the video in sequence. Unlike the traditional method that directly uses the output of the last moment of LSTM, this model adds an Attention layer, and the hidden state output at each moment has an adaptive weight, and then linearly weights the final feature vector. The reference paper implements a two-layer LSTM structure, while **this model implements a two-way LSTM with Attention**.\nThe Attention layer can refer to the paper [AttentionCluster](https://arxiv.org/abs/1711.09550)"
+ },
+ {
+ "comment": "This code provides instructions on how to train and test a model using PaddleVideo's attention LSTM on the Youtube-8M dataset. It mentions the required commands for training and testing, and also states that 8 GPUs are used during the process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/attention_lstm.md\":20-44",
+ "content": "## Data\nPaddleVide provides training and testing scripts on the Youtube-8M dataset. Youtube-8M data download and preparation please refer to [YouTube-8M data preparation](../../dataset/youtube8m.md)\n## Train\n### Youtube-8M data set training\n#### Start training\n- The Youtube-8M data set uses 8 cards for training. In the feature format, video and audio features will be used as input. The training start command of the data is as follows\n ```bash\n python3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_attetion_lstm main.py --validate -c configs/recognition/attention_lstm/attention_lstm_youtube8m.yaml\n ```\n## Test\nThe command is as follows:\n```bash\npython3.7 -B -m paddle.distributed.launch --gpus=\"0,1,2,3,4,5,6,7\" --log_dir=log_attetion_lstm main.py --test -c configs/recognition/attention_lstm/attention_lstm_youtube8m.yaml -w \"output/AttentionLSTM/AttentionLSTM_best.pdparams\"\n```\nWhen the test configuration uses the following parameters, the test indicators on the validation data set of Youtube-8M are as follows:"
+ },
+ {
+ "comment": "This code provides instructions to export an inference model and use the prediction engine for it. The exported model will be stored as AttentionLSTM.pdmodel and AttentionLSTM.pdiparams files, which are necessary for making predictions. Users can use the tools/predict.py script with the input file data/example.pkl and the configuration file configs/recognition/attention_lstm/attention_lstm_youtube8m.yaml to perform inference using the prediction engine.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/attention_lstm.md\":46-67",
+ "content": "| Hit@1 | PERR | GAP | checkpoints |\n| :-----: | :---------: | :---: | ----- |\n| 89.05 | 80.49 | 86.30 | [AttentionLSTM_yt8.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/AttentionLSTM_yt8.pdparams) |\n## Inference\n### Export inference model\n```bash\npython3.7 tools/export_model.py -c configs/recognition/attention_lstm/attention_lstm_youtube8m.yaml \\\n -p data/AttentionLSTM_yt8.pdparams \\\n -o inference/AttentionLSTM\n```\nThe above command will generate the model structure file `AttentionLSTM.pdmodel` and the model weight file `AttentionLSTM.pdiparams` required for prediction.\nFor the meaning of each parameter, please refer to [Model Reasoning Method](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0.0/docs/en/start.md#2-infer)\n### Use prediction engine inference\n```bash\npython3.7 tools/predict.py --input_file data/example.pkl \\\n --config configs/recognition/attention_lstm/attention_lstm_youtube8m.yaml \\"
+ },
+ {
+ "comment": "This code executes the AttentionLSTM model for video classification on a specific file (data/example.pkl). The predicted top-1 class is 11, and the confidence is 0.9841002225875854. This result utilizes the model trained on YouTube-8M dataset, indicating its accuracy in video classification tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/attention_lstm.md\":68-83",
+ "content": " --model_file inference/AttentionLSTM/AttentionLSTM.pdmodel \\\n --params_file inference/AttentionLSTM/AttentionLSTM.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nAn example of the output is as follows:\n```bash\nCurrent video file: data/example.pkl\n top-1 class: 11\n top-1 score: 0.9841002225875854\n```\nIt can be seen that using the AttentionLSTM model trained on Youtube-8M to predict data/example.pkl, the output top1 category id is 11, and the confidence is 0.98.\n## Reference paper\n- [Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification](https://arxiv.org/abs/1711.09550), Xiang Long, Chuang Gan, Gerard de Melo, Jiajun Wu, Xiao Liu, Shilei Wen\n- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://arxiv.org/abs/1609.08675), Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6b1c905d-49db-4909-8ae6-bf47e973e703.json b/docs/doc/6b1c905d-49db-4909-8ae6-bf47e973e703.json
new file mode 100644
index 000000000..aa604bb76
--- /dev/null
+++ b/docs/doc/6b1c905d-49db-4909-8ae6-bf47e973e703.json
@@ -0,0 +1,40 @@
+{
+ "summary": "This code adapts ViT model parameters, modifies pos_embed and time_embed for compatibility, and includes functions for loading/saving PaddlePaddle models with parallel/non-parallel handling and progress bar.",
+ "details": [
+ {
+ "comment": "The code is a function that converts pre-trained ViT model parameters to match the existing model. It takes in the model, state_dicts, num_patches, seg_num, and attention_type as arguments. The function adapts the ViT's pre-trained model parameters for better compatibility with the existing model structure.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":0-27",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport os.path as osp\nimport time\nfrom tqdm import tqdm\nimport paddle\nimport paddle.nn.functional as F\nfrom EIVideo.paddlevideo.utils import get_logger\nfrom EIVideo.paddlevideo.utils import main_only\ndef pretrain_vit_param_trans(model, state_dicts, num_patches, seg_num,\n attention_type):\n \"\"\"\n Convert ViT's pre-trained model parameters to a parameter dictionary that matches the existing model"
+ },
+ {
+ "comment": "This code modifies the 'pos_embed' tensor in state_dicts if its shape doesn't match the expected shape. It interpolates the other_pos_embed to fit the desired size, then concatenates the cls_pos_embed and new_pos_embed and updates the state_dicts['pos_embed']. This allows the code to maintain consistency in the 'pos_embed' tensor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":28-48",
+ "content": " \"\"\"\n if 'head' + '.weight' in state_dicts:\n del state_dicts['head' + '.weight']\n if 'head' + '.bias' in state_dicts:\n del state_dicts['head' + '.bias']\n total_len = len(model.state_dict())\n if num_patches + 1 != state_dicts['pos_embed'].shape[1]:\n pos_embed = state_dicts['pos_embed']\n cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)\n other_pos_embed = pos_embed[0,\n 1:, :].unsqueeze(0).unsqueeze(1).transpose(\n (0, 1, 3, 2))\n new_pos_embed = F.interpolate(other_pos_embed,\n size=(other_pos_embed.shape[-2],\n num_patches),\n mode='nearest')\n new_pos_embed = new_pos_embed.squeeze(0).transpose((0, 2, 1))\n new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed), axis=1)\n state_dicts['pos_embed'] = new_pos_embed\n time.sleep(0.01)"
+ },
+ {
+ "comment": "The code checks if a specific key 'time_embed' exists in the state_dicts and adjusts its shape accordingly. It then interpolates the time_embed using nearest mode and transposes it to fit into the new shape. After that, it creates a progress bar \"Loading weights\" using tqdm for the total length of data and sets the description as the current key being processed. If 'attn' is present in the key and 'blocks', it replaces 'attn' with 'temporal_attn' if not already present in state_dicts and adds it to new_state_dicts.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":50-70",
+ "content": " if 'time_embed' in state_dicts and seg_num != state_dicts[\n 'time_embed'].shape[1]:\n time_embed = state_dicts['time_embed'].transpose((0, 2, 1)).unsqueeze(0)\n new_time_embed = F.interpolate(time_embed,\n size=(time_embed.shape[-2], seg_num),\n mode='nearest')\n state_dicts['time_embed'] = new_time_embed.squeeze(0).transpose(\n (0, 2, 1))\n time.sleep(0.01)\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n if attention_type == 'divided_space_time':\n new_state_dicts = state_dicts.copy()\n for key in tqdm(state_dicts):\n if 'blocks' in key and 'attn' in key:\n desc.set_description(\"Loading %s\" % key)\n new_key = key.replace('attn', 'temporal_attn')\n if not new_key in state_dicts:\n new_state_dicts[new_key] = state_dicts[key]"
+ },
+ {
+ "comment": "This code is checking if a certain key exists in the state_dict and creating a new key with 'temporal' added to it. It is also updating the description for loading weights and returning the updated state_dicts. The function pretrain_resnet18_param_trans compares loaded dict with encoder and pose_encoder dictionaries in the model, possibly for parameter transfer learning.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":71-95",
+ "content": " else:\n new_state_dicts[new_key] = state_dicts[new_key]\n if 'blocks' in key and 'norm1' in key:\n desc.set_description(\"Loading %s\" % key)\n new_key = key.replace('norm1', 'temporal_norm1')\n if not new_key in state_dicts:\n new_state_dicts[new_key] = state_dicts[key]\n else:\n new_state_dicts[new_key] = state_dicts[new_key]\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n return new_state_dicts\ndef pretrain_resnet18_param_trans(model, loaded_dict):\n encoder_dict = model.encoder.state_dict()\n pose_encoder_dict = model.pose_encoder.state_dict()\n names = ['encoder.', 'encoder_day.', 'encoder_night.']\n for name in names:\n for key, value in loaded_dict.items():\n key = str(name + key)\n if key in encoder_dict:"
+ },
+ {
+ "comment": "The code is defining a function to load pre-trained model parameters, which requires converting the parameters of the pre-trained model into the parameters needed for the current model. The function first checks if the weight path exists and raises an IOError if it does not. Then, it loads the state_dicts from the given weight_path using paddle.load().",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":96-126",
+ "content": " encoder_dict[key] = value\n num_input_images = 2\n loaded_dict['conv1.weight'] = paddle.concat(\n [loaded_dict['conv1.weight']] * num_input_images, 1) / num_input_images\n for name, value in loaded_dict.items():\n name = str('encoder.' + name)\n if name in pose_encoder_dict:\n pose_encoder_dict[name] = value\n return encoder_dict, pose_encoder_dict\n#XXX(shipping): maybe need load N times because of different cards have different params.\n@main_only\ndef load_ckpt(model, weight_path, **kargs):\n \"\"\"\n 1. Load pre-trained model parameters\n 2. Extract and convert from the pre-trained model to the parameters\n required by the existing model\n 3. Load the converted parameters of the existing model\n \"\"\"\n #model.set_state_dict(state_dict)\n if not osp.isfile(weight_path):\n raise IOError(f'{weight_path} is not a checkpoint file')\n #state_dicts = load(weight_path)\n logger = get_logger(\"paddlevideo\")\n state_dicts = paddle.load(weight_path)"
+ },
+ {
+ "comment": "This code is used to load weights for a model, specifically handling Resnet Encoder and Vision Transformer cases. For Resnet Encoder, it updates the state dictionary with separate dictionaries for encoder and pose_encoder. For Vision Transformer (TimeSformer), it uses pretrain_vit_param_trans function. If the model is neither of these types, it initializes an empty dictionary. The code also includes a tqdm progress bar to show the loading progress.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":127-151",
+ "content": " if 'ResnetEncoder' in str(model):\n encoder_dict, pose_encoder_dict = pretrain_resnet18_param_trans(\n model, state_dicts)\n tmp = model.state_dict()\n tmp.update(\n {'backbone.encoder.' + k: v\n for (k, v) in encoder_dict.items()})\n tmp.update({\n 'backbone.pose_encoder.' + k: v\n for (k, v) in pose_encoder_dict.items()\n })\n elif \"VisionTransformer\" in str(model): # For TimeSformer case\n tmp = pretrain_vit_param_trans(model, state_dicts, kargs['num_patches'],\n kargs['seg_num'],\n kargs['attention_type'])\n else:\n tmp = {}\n total_len = len(model.state_dict())\n with tqdm(total=total_len,\n position=1,\n bar_format='{desc}',\n desc=\"Loading weights\") as desc:\n for item in tqdm(model.state_dict(), total=total_len, position=0):\n name = item\n desc.set_description('Loading %s' % name)"
+ },
+ {
+ "comment": "This code snippet defines functions for loading and saving PaddlePaddle models. It checks if the model is parallel or non-parallel, converts the state dictionaries accordingly, and updates the model's state dictionary. The `mkdir` function creates a directory if it doesn't exist already, and there are separate save and load functions defined for ease of use.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/utils/save_load.py\":152-181",
+ "content": " if name not in state_dicts: # Convert from non-parallel model\n if str('backbone.' + name) in state_dicts:\n tmp[name] = state_dicts['backbone.' + name]\n else: # Convert from parallel model\n tmp[name] = state_dicts[name]\n time.sleep(0.01)\n ret_str = \"loading {:<20d} weights completed.\".format(\n len(model.state_dict()))\n desc.set_description(ret_str)\n model.set_state_dict(tmp)\ndef mkdir(dir):\n if not os.path.exists(dir):\n # avoid error when train with multiple gpus\n try:\n os.makedirs(dir)\n except:\n pass\n@main_only\ndef save(obj, path):\n paddle.save(obj, path)\ndef load(file_name):\n if not osp.isfile(file_name):\n raise IOError(f'{file_name} not exist')\n return paddle.load(file_name)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6b644553-9c32-4362-acb9-3a20852badfc.json b/docs/doc/6b644553-9c32-4362-acb9-3a20852badfc.json
new file mode 100644
index 000000000..95eb92b2e
--- /dev/null
+++ b/docs/doc/6b644553-9c32-4362-acb9-3a20852badfc.json
@@ -0,0 +1,25 @@
+{
+ "summary": "This code imports modules and registers functions for building a video object detection model, as well as dynamically constructing components based on a configuration file.",
+ "details": [
+ {
+ "comment": "Imports necessary modules and registers various components for video object detection model building.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/builder.py\":0-18",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .registry import BACKBONES, HEADS, LOSSES, RECOGNIZERS, LOCALIZERS, ROI_EXTRACTORS, DETECTORS, BBOX_ASSIGNERS, BBOX_SAMPLERS, BBOX_CODERS, PARTITIONERS, MULTIMODAL, SEGMENT, SEGMENTERS\nfrom ..utils import build\nfrom .registry import (BACKBONES, BBOX_ASSIGNERS, BBOX_CODERS, BBOX_SAMPLERS,\n DETECTORS, ESTIMATORS, HEADS, LOCALIZERS, LOSSES,\n MULTIMODAL, PARTITIONERS, RECOGNIZERS, ROI_EXTRACTORS)"
+ },
+ {
+ "comment": "This code defines various building functions for different parts of a model. The \"build_backbone\" function builds the backbone of the model, while \"build_roi_extractor\", \"build_assigner\", and \"build_sampler\" build the region of interest extractor, box assigner, and box sampler respectively. \"build_head\" builds the head of the model, \"build_loss\" builds the loss function, and \"build_recognizer\" and \"build_segmenter\" build recognizers and segmenters with different frameworks or keys. The functions use the \"build\" method from an unspecified source (\"*\") to perform the actual building process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/builder.py\":21-72",
+ "content": "def build_backbone(cfg):\n \"\"\"Build backbone.\"\"\"\n return build(cfg, BACKBONES)\ndef build_roi_extractor(cfg):\n \"\"\"Build roi extractor.\"\"\"\n return build(cfg, ROI_EXTRACTORS)\ndef build_assigner(cfg, **default_args):\n \"\"\"Builder of box assigner.\"\"\"\n return build(cfg, BBOX_ASSIGNERS)\ndef build_sampler(cfg, **default_args):\n \"\"\"Builder of box sampler.\"\"\"\n return build(cfg, BBOX_SAMPLERS)\ndef build_roi_extractor(cfg):\n \"\"\"Build roi extractor.\"\"\"\n return build(cfg, ROI_EXTRACTORS)\ndef build_assigner(cfg, **default_args):\n \"\"\"Builder of box assigner.\"\"\"\n return build(cfg, BBOX_ASSIGNERS)\ndef build_sampler(cfg, **default_args):\n \"\"\"Builder of box sampler.\"\"\"\n return build(cfg, BBOX_SAMPLERS)\ndef build_head(cfg):\n \"\"\"Build head.\"\"\"\n return build(cfg, HEADS)\ndef build_loss(cfg):\n \"\"\"Build loss.\"\"\"\n return build(cfg, LOSSES)\ndef build_recognizer(cfg):\n \"\"\"Build recognizer.\"\"\"\n return build(cfg, RECOGNIZERS, key='framework')\ndef build_segmenter(cfg):\n \"\"\"Build segmenter.\"\"\""
+ },
+ {
+ "comment": "The code is a builder function that dynamically builds various components (recognizers, localizers, detectors, partitioners, estimators, and multimodal) based on the specified framework type in the configuration file. It utilizes the 'build' function to return the appropriate component for further processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/builder.py\":73-115",
+ "content": " return build(cfg, SEGMENTERS, key='framework')\ndef build_localizer(cfg):\n \"\"\"Build localizer.\"\"\"\n return build(cfg, LOCALIZERS, key='framework')\ndef build_detector(cfg, train_cfg=None, test_cfg=None):\n \"\"\"Build detector.\"\"\"\n return build(cfg, DETECTORS, key='framework')\ndef build_partitioner(cfg):\n \"\"\"Build partitioner.\"\"\"\n return build(cfg, PARTITIONERS, key='framework')\ndef build_estimator(cfg):\n \"\"\"Build estimator.\"\"\"\n return build(cfg, ESTIMATORS, key='framework')\ndef build_multimodal(cfg):\n \"\"\"Build multimodal.\"\"\"\n return build(cfg, MULTIMODAL, key='framework')\ndef build_segment(cfg):\n \"\"\"Build segment.\"\"\"\n return build(cfg, SEGMENT, key='framework')\ndef build_model(cfg):\n cfg_copy = cfg.copy()\n framework_type = cfg_copy.get('framework')\n if framework_type in RECOGNIZERS:\n return build_recognizer(cfg)\n elif framework_type in LOCALIZERS:\n return build_localizer(cfg)\n elif framework_type in PARTITIONERS:\n return build_partitioner(cfg)\n elif framework_type in DETECTORS:"
+ },
+ {
+ "comment": "This code selects a specific function to build based on the framework type. It checks if the framework type is in defined lists of detectors, estimators, multimodal models, segmenters, or segments. If none match, it raises NotImplementedError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/builder.py\":116-126",
+ "content": " return build_detector(cfg)\n elif framework_type in ESTIMATORS:\n return build_estimator(cfg)\n elif framework_type in MULTIMODAL:\n return build_multimodal(cfg)\n elif framework_type in SEGMENTERS:\n return build_segmenter(cfg)\n elif framework_type in SEGMENT:\n return build_segment(cfg)\n else:\n raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6b83d6d1-3f2f-49f7-88b0-8b6ffdd861c6.json b/docs/doc/6b83d6d1-3f2f-49f7-88b0-8b6ffdd861c6.json
new file mode 100644
index 000000000..2b4abee05
--- /dev/null
+++ b/docs/doc/6b83d6d1-3f2f-49f7-88b0-8b6ffdd861c6.json
@@ -0,0 +1,30 @@
+{
+ "summary": "This code trains and evaluates the MS-TCN video action segmentation model using provided datasets, compares performance with PaddleVideo's MSTCN, exports inference models, uses metrics like accuracy and F1 score, and runs with GPU usage enabled.",
+ "details": [
+ {
+ "comment": "Introduction: MS-TCN model for video motion segmentation was published in 2019 and optimized for higher precision results in PaddleVideo.\nData: Choose from 50salads, breakfast, gtea datasets for training. Refer to Video Action Segmentation dataset download and preparation doc.\nTrain: After preparing the dataset, run scripts with provided command example.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/mstcn.md\":0-34",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/segmentation/mstcn.md) | English\n# MS-TCN : Video Action Segmentation Model\n---\n## Contents\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Inference](#Inference)\n- [Reference](#Reference)\n## Introduction\nMs-tcn model is a classic model of video motion segmentation model, which was published on CVPR in 2019. We optimized the officially implemented pytorch code and obtained higher precision results in paddlevideo.\n\n![](\"../../../images/mstcn.PNG\")
\nMS-TCN Overview\n
\n## Data\nMS-TCN can choose 50salads, breakfast, gtea as trianing set. Please refer to Video Action Segmentation dataset download and preparation doc [Video Action Segmentation dataset](../../dataset/SegmentationDataset.md)\n## Train\nAfter prepare dataset, we can run sprits.\n```bash\n# gtea dataset\nexport CUDA_VISIBLE_DEVICES=3\npython3.7 main.py --validate -c configs/segmentation/ms_tcn/ms_tcn_gtea.yaml --seed 1538574472"
+ },
+ {
+ "comment": "The code snippet provides instructions for training and testing the video action segmentation model, specifically MSTCN. It mentions that single sample training is supported, and demonstrates how to test MSTCN on a dataset using the provided command line or script program. Additionally, it explains the evaluation method used for datasets and refers to the author's provided evaluation script.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/mstcn.md\":35-51",
+ "content": "```\n- Start the training by using the above command line or script program. There is no need to use the pre training model. The video action segmentation model is usually a full convolution network. Due to the different lengths of videos, the `DATASET.batch_size` of the video action segmentation model is usually set to `1`, that is, batch training is not required. At present, only **single sample** training is supported.\n## Test\nTest MS-TCN on dataset scripts:\n```bash\npython main.py --test -c configs/segmentation/ms_tcn/ms_tcn_gtea.yaml --weights=./output/MSTCN/MSTCN_split_1.pdparams\n```\n- The specific implementation of the index is to calculate ACC, edit and F1 scores by referring to the test script[evel.py](https://github.com/yabufarha/ms-tcn/blob/master/eval.py) provided by the author of ms-tcn.\n- The evaluation method of data set adopts the folding verification method in ms-tcn paper, and the division method of folding is the same as that in ms-tcn paper.\nAccuracy on Breakfast dataset(4 folding verification):"
+ },
+ {
+ "comment": "This table compares the performance of a paper model and PaddleVideo's MSTCN model on different datasets. The metrics include accuracy (Acc), edit distance, and F1 score (F1@0.1, F1@0.25, F1@0.5). The models are validated with 5-fold cross-validation on the 50salads dataset and 4-fold on the gtea dataset. The provided checkpoints are for gtea dataset splits.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/mstcn.md\":53-77",
+ "content": "| Model | Acc | Edit | F1@0.1 | F1@0.25 | F1@0.5 |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| paper | 66.3% | 61.7% | 48.1% | 48.1% | 37.9% |\n| paddle | 65.2% | 61.5% | 53.7% | 49.2% | 38.8% |\nAccuracy on 50salads dataset(5 folding verification):\n| Model | Acc | Edit | F1@0.1 | F1@0.25 | F1@0.5 |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| paper | 80.7% | 67.9% | 76.3% | 74.0% | 64.5% |\n| paddle | 81.1% | 71.5% | 77.9% | 75.5% | 66.5% |\nAccuracy on gtea dataset(4 folding verification):\n| Model | Acc | Edit | F1@0.1 | F1@0.25 | F1@0.5 |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| paper | 79.2% | 81.4% | 87.5% | 85.4% | 74.6% |\n| paddle | 76.9% | 81.8% | 86.4% | 84.7% | 74.8% |\nModel weight for gtea\nTest_Data| F1@0.5 | checkpoints |\n| :----: | :----: | :---- |\n| gtea_split1 | 70.2509 | [MSTCN_gtea_split_1.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/MSTCN_gtea_split_1.pdparams) |\n| gtea_split2 | 70.7224 | [MSTCN_gtea_split_2.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/MSTCN_gtea_split_2.pdparams) |"
+ },
+ {
+ "comment": "This code provides instructions for exporting and using an inference model. The `export_model.py` script is used to create the architecture file (`MSTCN.pdmodel`) and parameters file (`MSTCN.pdiparams`). These files can be obtained by running the script with the given configuration file, pre-trained parameters file path, and output directory. The inference process involves providing a list of input files in the format `S1_- _C1.npy`. To execute the inference, run the `predict.py` script with the input file list as an argument.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/mstcn.md\":78-107",
+ "content": "| gtea_split3 | 80.0 | [MSTCN_gtea_split_3.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/MSTCN_gtea_split_3.pdparams) |\n| gtea_split4 | 78.1609 | [MSTCN_gtea_split_4.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/MSTCN_gtea_split_4.pdparams) |\n## Infer\n### export inference model\n```bash\npython3.7 tools/export_model.py -c configs/segmentation/ms_tcn/ms_tcn_gtea.yaml \\\n -p data/MSTCN_gtea_split_1.pdparams \\\n -o inference/MSTCN\n```\nTo get model architecture file `MSTCN.pdmodel` and parameters file `MSTCN.pdiparams`, use:\n- Args usage please refer to [Model Inference](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0/docs/zh-CN/start.md#2-%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86).\n### infer\nInput file are the file list for infering, for example:\n```\nS1_Cheese_C1.npy\nS1_CofHoney_C1.npy\nS1_Coffee_C1.npy\nS1_Hotdog_C1.npy\n...\n```\n```bash\npython3.7 tools/predict.py --input_file data/gtea/splits/test.split1.bundle \\"
+ },
+ {
+ "comment": "The code is specifying the configuration file, model file, and parameter file for running the MSTCN (Multi-Stage Temporal Convolutional Network) segmentation model. It also sets the GPU usage to True and TensorRT to False.\nExample logs show the results being written into respective text files in the inference/infer_results folder.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/mstcn.md\":108-129",
+ "content": " --config configs/segmentation/ms_tcn/ms_tcn_gtea.yaml \\\n --model_file inference/MSTCN/MSTCN.pdmodel \\\n --params_file inference/MSTCN/MSTCN.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nexample of logs:\n```bash\nresult write in : ./inference/infer_results/S1_Cheese_C1.txt\nresult write in : ./inference/infer_results/S1_CofHoney_C1.txt\nresult write in : ./inference/infer_results/S1_Coffee_C1.txt\nresult write in : ./inference/infer_results/S1_Hotdog_C1.txt\nresult write in : ./inference/infer_results/S1_Pealate_C1.txt\nresult write in : ./inference/infer_results/S1_Peanut_C1.txt\nresult write in : ./inference/infer_results/S1_Tea_C1.txt\n```\n## Reference\n- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://arxiv.org/pdf/1903.01945.pdf), Y. Abu Farha and J. Gall."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6cc7ca89-cbfe-416d-b637-65338839e3f4.json b/docs/doc/6cc7ca89-cbfe-416d-b637-65338839e3f4.json
new file mode 100644
index 000000000..7b13017ba
--- /dev/null
+++ b/docs/doc/6cc7ca89-cbfe-416d-b637-65338839e3f4.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code is a part of PaddleVideo's TableTennis application, containing an AttrDict class and parse_config function for parsing YAML configuration files using yaml and ast libraries. It also imports the logger module for logging purposes, and logs a separator string to indicate context changes.",
+ "details": [
+ {
+ "comment": "The code is part of the PaddleVideo TableTennis application and contains a class called AttrDict that extends the Python dictionary functionality. The file also includes the parse_config function, which likely reads and parses configuration files. The code uses the yaml and ast libraries for processing configuration data in a format-agnostic manner. Additionally, it defines a list of configuration types (CONFIG_SECS) and utilizes the logger module for logging purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/utils/config_utils.py\":0-46",
+ "content": "\"\"\"\nconfig_utils\n\"\"\"\n# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport yaml\nimport ast\nimport logger\nlogger = logger.Logger()\nCONFIG_SECS = [\n 'train',\n 'valid',\n 'test',\n 'infer',\n]\nclass AttrDict(dict):\n \"\"\"\n AttrDict\n \"\"\"\n def __getattr__(self, key):\n return self[key]\n def __setattr__(self, key, value):\n if key in self.__dict__:\n self.__dict__[key] = value\n else:\n self[key] = value\ndef parse_config(cfg_file):"
+ },
+ {
+ "comment": "This code imports the yaml library and loads a configuration file into an AttrDict object, allowing for easier manipulation of nested dictionary data. It also includes functions to create an AttrDict from a string and print the configurations in a formatted manner.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/utils/config_utils.py\":47-79",
+ "content": " \"\"\"Load a config file into AttrDict\"\"\"\n import yaml\n with open(cfg_file, 'r') as fopen:\n yaml_config = AttrDict(yaml.load(fopen, Loader=yaml.Loader))\n create_attr_dict(yaml_config)\n return yaml_config\ndef create_attr_dict(yaml_config):\n \"\"\"create_attr_dict\"\"\"\n for key, value in yaml_config.items():\n if isinstance(value, dict):\n yaml_config[key] = value = AttrDict(value)\n if isinstance(value, str):\n try:\n value = ast.literal_eval(value)\n except BaseException:\n pass\n if isinstance(value, AttrDict):\n create_attr_dict(yaml_config[key])\n else:\n yaml_config[key] = value\n return\ndef print_configs(cfg, mode):\n \"\"\"print_configs\"\"\"\n logger.info(\n \"---------------- {:>5} Arguments ----------------\".format(mode))\n for sec, sec_items in cfg.items():\n logger.info(\"{}:\".format(sec))\n for k, v in sec_items.items():\n logger.info(\" {}:{}\".format(k, v))"
+ },
+ {
+ "comment": "This code snippet logs a separator string to the logger, indicating a change in context or section within the program.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/utils/config_utils.py\":80-80",
+ "content": " logger.info(\"-------------------------------------------------\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6d049a5b-64d9-43c3-9132-f86a22f43e35.json b/docs/doc/6d049a5b-64d9-43c3-9132-f86a22f43e35.json
new file mode 100644
index 000000000..1cc3685ae
--- /dev/null
+++ b/docs/doc/6d049a5b-64d9-43c3-9132-f86a22f43e35.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The PaddleVideo library's FrameDataset and FrameDataset_Sport classes load, transform, and process video data with error handling for missing or corrupted files under Apache License 2.0.",
+ "details": [
+ {
+ "comment": "This code is a Python class for a FrameDataset, which loads raw frames from frame files and applies specified transform operations. It is part of the PaddleVideo library and follows Apache License 2.0. The dataset index file is used to organize the loaded data. This class inherits from BaseDataset, suggesting it has some common functionalities.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":0-30",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@DATASETS.register()\nclass FrameDataset(BaseDataset):\n \"\"\"Rawframe dataset for action recognition.\n The dataset loads raw frames from frame files, and apply specified transform operatation them.\n The indecx file "
+ },
+ {
+ "comment": "This code initializes a class for loading video information from an index file. The index file contains the directory of frames, total frames, and label for each video. It supports pipeline and data_prefix, and has options for test mode and suffix format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":30-60",
+ "content": "is a text file with multiple lines, and each line indicates the directory of frames of a video, toatl frames of the video, and its label, which split with a whitespace.\n Example of an index file:\n .. code-block:: txt\n file_path-1 150 1\n file_path-2 160 1\n file_path-3 170 2\n file_path-4 180 2\n Args:\n file_path (str): Path to the index file.\n pipeline(XXX):\n data_prefix (str): directory path of the data. Default: None.\n test_mode (bool): Whether to bulid the test dataset. Default: False.\n suffix (str): suffix of file. Default: 'img_{:05}.jpg'.\n \"\"\"\n def __init__(self,\n file_path,\n pipeline,\n num_retries=5,\n data_prefix=None,\n test_mode=False,\n suffix='img_{:05}.jpg'):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\""
+ },
+ {
+ "comment": "This code reads data from a file and returns information related to frames, such as the frame directory, suffix, number of frames, and labels. It also includes a try-catch block that attempts to prepare the frames for training or validation multiple times if an exception occurs while reading the frames files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":61-85",
+ "content": " info = []\n with open(self.file_path, 'r') as fin:\n for line in fin:\n line_split = line.strip().split()\n frame_dir, frames_len, labels = line_split\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n info.append(\n dict(frame_dir=frame_dir,\n suffix=self.suffix,\n frames_len=frames_len,\n labels=int(labels)))\n return info\n def prepare_train(self, idx):\n \"\"\"Prepare the frames for training/valid given index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info("
+ },
+ {
+ "comment": "The code handles exceptions for loading missing frames in the dataset. It tries to load frames multiple times within a specified range and logs errors when needed. If an exception occurs, it randomly selects another index and continues the process until successful.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":86-107",
+ "content": " \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"Prepare the frames for test given index. \"\"\"\n #Try to catch Exception caused by reading missing frames files\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['frame_dir'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])"
+ },
+ {
+ "comment": "The code defines a FrameDataset_Sport class for loading raw videos and applying specified transforms. It uses an index file containing video file paths and labels, and takes arguments for file path, data transforms pipeline, retry attempts, and other BaseDataset kwargs. The load_file function reads the index file to obtain video information.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":110-135",
+ "content": "@DATASETS.register()\nclass FrameDataset_Sport(BaseDataset):\n \"\"\"Video dataset for action recognition\n The dataset loads raw videos and apply specified transforms on them.\n The index file is a file with multiple lines, and each line indicates\n a sample video with the filepath and label, which are split with a whitesapce.\n Example of a inde file:\n .. code-block:: txt\n path/000.mp4 1\n path/001.mp4 1\n path/002.mp4 2\n path/003.mp4 2\n Args:\n file_path(str): Path to the index file.\n pipeline(XXX): A sequence of data transforms.\n **kwargs: Keyword arguments for ```BaseDataset```.\n \"\"\"\n def __init__(self, file_path, pipeline, num_retries=5, suffix='', **kwargs):\n self.num_retries = num_retries\n self.suffix = suffix\n super().__init__(file_path, pipeline, **kwargs)\n def load_file(self):\n \"\"\"Load index file to get video information.\"\"\"\n info = []\n with open(self.file_path, 'r') as fin:"
+ },
+ {
+ "comment": "This code reads lines from a file, each representing a frame directory and associated information. It then splits the line into components and appends the frame directory and suffix to the info list. The prepare_train function attempts to process data for training or validation, handling exceptions by retrying up to a specified number of times before selecting another index at random.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":136-157",
+ "content": " for line in fin:\n line_split = line.strip().split()\n frame_dir = line_split[0]\n if self.data_prefix is not None:\n frame_dir = osp.join(self.data_prefix, frame_dir)\n info.append(dict(frame_dir=frame_dir, suffix=self.suffix))\n return info\n def prepare_train(self, idx):\n \"\"\"TRAIN & VALID. Prepare the data for training/valid given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)"
+ },
+ {
+ "comment": "The function `prepare_test` is attempting to prepare data for testing by iterating through a certain number of retries in case of exceptions caused by corrupted video files. If an exception occurs, it logs the error and tries again with a different random index from the list of info. Once successful, it returns the images and labels as arrays.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/frame.py\":158-176",
+ "content": " continue\n return results['imgs'], np.array([results['labels']])\n def prepare_test(self, idx):\n \"\"\"TEST. Prepare the data for test given the index.\"\"\"\n #Try to catch Exception caused by reading corrupted video file\n for ir in range(self.num_retries):\n try:\n results = copy.deepcopy(self.info[idx])\n results = self.pipeline(results)\n except Exception as e:\n #logger.info(e)\n if ir < self.num_retries - 1:\n logger.info(\n \"Error when loading {}, have {} trys, will try again\".\n format(results['filename'], ir))\n idx = random.randint(0, len(self.info) - 1)\n continue\n return results['imgs'], np.array([results['labels']])"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6d190f3b-28cc-4448-a3fc-ef0b99294dd5.json b/docs/doc/6d190f3b-28cc-4448-a3fc-ef0b99294dd5.json
new file mode 100644
index 000000000..e0484fc63
--- /dev/null
+++ b/docs/doc/6d190f3b-28cc-4448-a3fc-ef0b99294dd5.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is importing the recognizers module from the paddlevideo.modeling.framework package, and defining the BaseRecognizer and Recognizer2D classes as part of its API. The __all__ variable lists these two classes as the public API elements of this package.",
+ "details": [
+ {
+ "comment": "This code is importing the recognizers module from the paddlevideo.modeling.framework package, and defining the BaseRecognizer and Recognizer2D classes as part of its API. The __all__ variable lists these two classes as the public API elements of this package.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/framework/__init__.py\":0-21",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .recognizers import BaseRecognizer, Recognizer2D\n__all__ = [\n 'BaseRecognizer',\n 'Recognizer2D',\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6de2121f-318f-4bea-a0d9-5d3a9b77d550.json b/docs/doc/6de2121f-318f-4bea-a0d9-5d3a9b77d550.json
new file mode 100644
index 000000000..45eb797bb
--- /dev/null
+++ b/docs/doc/6de2121f-318f-4bea-a0d9-5d3a9b77d550.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The Python class VOS_Test extends BaseDataset for video object segmentation tasks and supports pipeline mode, color type options, and resizing. The Davis 2017 dataset is initialized in PaddleVideo and returns a sequence dataset with images, labels, and fixed resolution of 480 pixels.",
+ "details": [
+ {
+ "comment": "This code snippet is from PaddleVideo's davis_dataset.py file and it appears to be a Python class named VOS_Test, which extends the BaseDataset class from the same module. The class is used for processing frames in each video of a dataset. It takes image_root and label_root as input parameters for accessing the required data. The logger is imported from paddle.utils to log any relevant information during execution. This dataset seems to be designed for video object segmentation (VOS) tasks, commonly used in computer vision applications.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":0-36",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport os.path as osp\nimport copy\nimport random\nimport numpy as np\nimport shutil\nfrom PIL import Image\nimport cv2\nfrom paddle.io import Dataset\nfrom ..registry import DATASETS\nfrom .base import BaseDataset\nfrom ...utils import get_logger\nlogger = get_logger(\"paddlevideo\")\nclass VOS_Test(Dataset):\n \"\"\"process frames in each video\n \"\"\"\n def __init__(self,\n image_root,\n label_root,"
+ },
+ {
+ "comment": "This code initializes the dataset with image and label file lists, image root and label root paths. It sets object number, total frames, pipeline mode, color type, resolution, creates an object numbers list, and assigns object numbers from labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":37-65",
+ "content": " seq_name,\n images,\n labels,\n pipeline=None,\n rgb=False,\n resolution=None):\n self.image_root = image_root\n self.label_root = label_root\n self.seq_name = seq_name\n self.images = images # image file list\n self.labels = labels\n self.obj_num = 1\n self.num_frame = len(self.images)\n self.pipeline = pipeline\n self.rgb = rgb\n self.resolution = resolution\n self.obj_nums = []\n temp_obj_num = 0\n for img_name in self.images:\n self.obj_nums.append(temp_obj_num)\n current_label_name = img_name.split('.')[0] + '.png'\n if current_label_name in self.labels:\n current_label = self.read_label(current_label_name)\n if temp_obj_num < np.unique(\n current_label)[-1]: #get object number from label_id\n temp_obj_num = np.unique(current_label)[-1]\n def __len__(self):"
+ },
+ {
+ "comment": "This code defines a class that loads data from the DAVIS dataset. It first returns the number of images in the dataset, then reads an image at a given index, and finally reads a corresponding label for the image. The class also allows resizing the images to a specified resolution if needed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":66-93",
+ "content": " return len(self.images)\n def read_image(self, idx):\n img_name = self.images[idx]\n img_path = os.path.join(self.image_root, self.seq_name, img_name)\n img = cv2.imread(img_path)\n img = np.array(img, dtype=np.float32)\n if self.rgb:\n img = img[:, :, [2, 1, 0]]\n return img\n def read_label(self, label_name):\n label_path = os.path.join(self.label_root, self.seq_name, label_name)\n label = Image.open(label_path)\n label = np.array(label, dtype=np.uint8)\n return label\n def __getitem__(self, idx):\n img_name = self.images[idx]\n current_img = self.read_image(idx)\n current_img = np.array(current_img)\n height, width, channels = current_img.shape\n if self.resolution is not None:\n width = int(np.ceil(float(width) * self.resolution / float(height)))\n height = int(self.resolution)\n current_label_name = img_name.split('.')[0] + '.png'\n obj_num = self.obj_nums[idx]"
+ },
+ {
+ "comment": "The function generates a sample for a dataset, including image and label data. It checks if the current_label_name is in labels, reads the label if present, creates a sample dictionary, adds metadata to the sample, applies a pipeline if one is specified, and converts 'current_img' to numpy array format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":95-126",
+ "content": " if current_label_name in self.labels:\n current_label = self.read_label(current_label_name)\n current_label = np.array(current_label)\n sample = {\n 'current_img': current_img,\n 'current_label': current_label\n }\n else:\n sample = {\n 'current_img': current_img\n } #only the first frame contains label\n sample['meta'] = {\n 'seq_name': self.seq_name,\n 'frame_num': self.num_frame,\n 'obj_num': obj_num,\n 'current_name': img_name,\n 'height': height,\n 'width': width,\n 'flip': False\n }\n if self.pipeline is not None:\n sample = self.pipeline(sample)\n for s in sample:\n s['current_img'] = np.array(s['current_img'])\n if 'current_label' in s.keys():\n s['current_label'] = s['current_label']\n return sample\n@DATASETS.register()\nclass DavisDataset(BaseDataset):"
+ },
+ {
+ "comment": "The code represents the initialization and file loading process for the Davis 2017 dataset in PaddleVideo. The constructor takes various parameters like file path, result root, pipeline, data prefix, test mode, year, rgb, and resolution to initialize the class attributes. The load_file() method sets image and label roots based on the provided resolution and reads the sequence names from a specified file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":127-157",
+ "content": " \"\"\"Davis 2017 dataset.\n \"\"\"\n def __init__(\n self,\n file_path,\n result_root,\n pipeline,\n data_prefix=None,\n test_mode=False,\n year=2017,\n rgb=False,\n resolution='480p',\n ):\n self.rgb = rgb\n self.result_root = result_root\n self.resolution = resolution\n self.year = year\n self.spt = 'val' if test_mode else 'train'\n super().__init__(file_path, pipeline, data_prefix, test_mode)\n def load_file(self):\n self.image_root = os.path.join(self.file_path, 'JPEGImages',\n self.resolution)\n self.label_root = os.path.join(self.file_path, 'Annotations',\n self.resolution)\n seq_names = []\n with open(\n os.path.join(self.file_path, 'ImageSets', str(self.year),\n self.spt + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))"
+ },
+ {
+ "comment": "This function prepares a test dataset for the VOS task. It retrieves the video name from the info list, then lists all image files in the corresponding directory and adds the first frame as the target label. If the target label does not exist in the result directory, it creates the necessary directories and copies the target label file to the correct location.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":158-181",
+ "content": " seq_names.extend(seqs_tmp)\n self.info = list(np.unique(seq_names))\n return self.info\n def prepare_test(self, idx):\n seq_name = self.info[idx] #video name\n images = list(\n np.sort(os.listdir(os.path.join(self.image_root, seq_name))))\n labels = [images[0].replace('jpg', 'png')] #we have first frame target\n # copy first frame target\n if not os.path.isfile(\n os.path.join(self.result_root, seq_name, labels[0])):\n if not os.path.exists(os.path.join(self.result_root, seq_name)):\n os.makedirs(os.path.join(self.result_root, seq_name))\n source_label_path = os.path.join(self.label_root, seq_name,\n labels[0])\n result_label_path = os.path.join(self.result_root, seq_name,\n labels[0])\n shutil.copy(source_label_path, result_label_path)\n seq_dataset = VOS_Test(self.image_root,\n self.label_root,"
+ },
+ {
+ "comment": "This code is returning a sequence dataset named seq_name with associated images and labels, processed by the pipeline function specified, potentially using RGB format if self.rgb is True, and a fixed resolution of 480 pixels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/dataset/davis_dataset.py\":182-188",
+ "content": " seq_name,\n images,\n labels,\n self.pipeline,\n rgb=self.rgb,\n resolution=480)\n return seq_dataset"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6e1d2d2e-6ddf-4b67-9f49-231e705fd17f.json b/docs/doc/6e1d2d2e-6ddf-4b67-9f49-231e705fd17f.json
new file mode 100644
index 000000000..a15bdf0aa
--- /dev/null
+++ b/docs/doc/6e1d2d2e-6ddf-4b67-9f49-231e705fd17f.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This is a Chinese comment for an interactive video annotation tool's Command Line Interface (CLI). It mentions installing the \"scikit-image\" package, running the program in inference mode using a specific configuration and model file, and provides a reference document link.",
+ "details": [
+ {
+ "comment": "This is a Chinese comment for an interactive video annotation tool's Command Line Interface (CLI). It mentions installing the \"scikit-image\" package, running the program in inference mode using a specific configuration and model file, and provides a reference document link.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/README.MD\":0-14",
+ "content": "# \u4ea4\u4e92\u5f0f\u89c6\u9891\u667a\u80fd\u6807\u6ce8\u5de5\u5177 - CLI(Command Line Interface)\n\u5728\u5f00\u59cb\u4f7f\u7528\u4e4b\u524d\uff0c\u60a8\u9700\u8981\u6309\u7167\u4ee5\u4e0b\u547d\u4ee4\u5b89\u88c5\u989d\u5916\u7684\u4f9d\u8d56\u5305\uff1a\n```bash\npython -m pip install scikit-image\n```\n## \u63a8\u7406\u8fd0\u884c\u65b9\u5f0f\n```shell\nC:\\Python\\Python37\\python.exe main.py --test -c E:/PaddlePaddle_Project/EIVideo/resources/backend/configs/manet.yaml -w E:/PaddlePaddle_Project/EIVideo/resources/backend/model/save_step_80000.pdparams\nC:\\Python\\Python37\\python.exe resources/backend/main.py --test -c E:/PaddlePaddle_Project/EIVideo/resources/backend/configs/manet.yaml -w E:/PaddlePaddle_Project/EIVideo/resources/backend/model/save_step_80000.pdparams\n```\n## \u53c2\u8003\u6587\u6863\n[manet](docs/zh-CN/manet.md)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6e9bbbd2-a926-421a-bbb0-ba8383f3362a.json b/docs/doc/6e9bbbd2-a926-421a-bbb0-ba8383f3362a.json
new file mode 100644
index 000000000..ebd886a2c
--- /dev/null
+++ b/docs/doc/6e9bbbd2-a926-421a-bbb0-ba8383f3362a.json
@@ -0,0 +1,50 @@
+{
+ "summary": "The code introduces a RecognizerDistillation class for recognizer distillation in PaddleVideo's framework, and includes model selection, modes like training and validation, loss functions, accuracy functions, and forward pass capabilities.",
+ "details": [
+ {
+ "comment": "This code defines a RecognizerDistillation class that inherits from nn.Layer in PaddleVideo's framework. It implements recognizer distillation, which is a machine learning framework for object recognition tasks. The class takes optional arguments such as freeze_params_list (a list to set models trainable/not), models, and loss. It is registered under RECOGNIZERS and uses logger from paddlevideo's utils.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":0-33",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nimport paddle\nimport paddle.nn as nn\nfrom ...registry import RECOGNIZERS\nfrom ... import builder\nfrom paddlevideo.utils import get_logger, get_dist_info\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerDistillation(nn.Layer):\n \"\"\"recognizer Distillation framework.\"\"\"\n def __init__(self,\n freeze_params_list=None,\n models=None,\n loss=None,\n **kargs):\n \"\"\"\n Args:\n freeze_params_list: list, set each model is trainable or not"
+ },
+ {
+ "comment": "This code initializes an instance of a distillation model. It takes in a list of models and loss configurations, as well as a freeze_params_list (optional). It checks the lengths of the input lists, builds teacher and student models, and initializes backbone and head if they exist in the configurations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":34-59",
+ "content": " models: config of distillaciton model.\n loss: config of loss list\n \"\"\"\n super().__init__()\n self.model_list = []\n self.model_name_list = []\n self.loss_cfgs = loss\n if freeze_params_list is None:\n freeze_params_list = [False] * len(models)\n assert len(freeze_params_list) == len(models)\n # build Teacher and Student model\n for idx, model_config in enumerate(models):\n assert len(model_config) == 1\n key = list(model_config.keys())[0] #Teacher or Student\n model_config = model_config[key]\n model_name = model_config['backbone']['name']\n backbone, head = None, None\n if model_config.get('backbone'):\n backbone = builder.build_backbone(model_config['backbone'])\n if hasattr(backbone, 'init_weights'):\n backbone.init_weights()\n if model_config.get('head'):\n head = builder.build_head(model_config['head'])"
+ },
+ {
+ "comment": "Builds a distillation model by appending a head to the backbone, initializes weights for the head if possible, and sets trainable parameters based on freeze_params_list. Constructs loss functions using builder.build_loss().",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":60-84",
+ "content": " if hasattr(head, 'init_weights'):\n head.init_weights()\n model = nn.Sequential(backbone, head)\n logger.info('build distillation {} model done'.format(key))\n # for add all parameters in nn.Layer class\n self.model_list.append(self.add_sublayer(key, model))\n self.model_name_list.append({model_name: key})\n # set model trainable or not\n if freeze_params_list[idx]:\n for param in model.parameters():\n param.trainable = False\n # build loss: support for loss list\n self.loss_func_list = []\n mode_keys = list(loss.keys())\n for mode in mode_keys:\n loss_cfgs = loss[mode]\n for loss_cfg in loss_cfgs:\n loss_func_dict = {}\n model_name_pairs = loss_cfg.pop('model_name_pairs')\n loss_func = builder.build_loss(loss_cfg)\n loss_func_dict['mode'] = mode\n loss_func_dict['loss_func'] = loss_func"
+ },
+ {
+ "comment": "This code defines a class for handling different modes of operation (train, valid, test, infer) and includes methods to handle each mode. It also contains a method to calculate the loss based on output and labels in 'Train' or 'Val' mode. The code is likely used in a model framework and the class might be used to control the flow and operations of the model depending on the mode it runs in.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":85-113",
+ "content": " loss_func_dict['model_name_pairs'] = model_name_pairs\n self.loss_func_list.append(loss_func_dict)\n def forward(self, data_batch, mode='infer'):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step\n 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch)\n elif mode == 'valid':\n return self.val_step(data_batch)\n elif mode == 'test':\n return self.test_step(data_batch)\n elif mode == 'infer':\n return self.infer_step(data_batch)\n else:\n raise NotImplementedError\n def get_loss(self, output, labels, mode):\n \"\"\"\n Args:\n output: dict, output name and its value\n labels: label of data\n mode: str, 'Train' or 'Val'\n \"\"\"\n output['GroundTruth'] = labels\n loss_list = []"
+ },
+ {
+ "comment": "This code is iterating over a list of loss function dictionaries to find the appropriate loss function based on the input mode. It then calculates the loss value and appends it to a list. Finally, it adds up all the loss values to get the total loss. In the `get_acc` method, it defines an inner function that calculates top-1 and top-5 accuracy scores using PaddlePaddle's `metric.accuracy` function. It also handles multi-card validation by reducing the sum of top-1 and top-5 accuracy scores across multiple cards.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":115-135",
+ "content": " for loss_func_dict in self.loss_func_list:\n if mode == loss_func_dict['mode']:\n model_name_pairs = loss_func_dict['model_name_pairs']\n loss_func = loss_func_dict['loss_func']\n loss_val = loss_func(output[model_name_pairs[0]],\n output[model_name_pairs[1]])\n loss_list.append(loss_val)\n total_loss = paddle.add_n(loss_list)\n return total_loss\n def get_acc(self, scores, labels, mode='Train'):\n def _get_acc(score, label, mode='Train'):\n top1 = paddle.metric.accuracy(input=score, label=label, k=1)\n top5 = paddle.metric.accuracy(input=score, label=label, k=5)\n _, world_size = get_dist_info()\n # Deal with multi cards validate\n if world_size > 1 and mode == 'Val': #reduce sum when valid\n top1 = paddle.distributed.all_reduce(\n top1, op=paddle.distributed.ReduceOp.SUM) / world_size\n top5 = paddle.distributed.all_reduce("
+ },
+ {
+ "comment": "The code snippet contains a recognizerDistillation function, which calculates accuracy based on given scores and labels. It also includes a forward_model function for reshaping images and applying model operations. The train_step function defines the training process from input to output, including loss metrics calculation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":136-164",
+ "content": " top5, op=paddle.distributed.ReduceOp.SUM) / world_size\n return top1, top5\n if len(labels) == 1:\n label = labels[0]\n return _get_acc(scores, label)\n # Deal with VideoMix\n elif len(labels) == 3:\n label_a, label_b, lam = labels\n top1a, top5a = _get_acc(scores, label_a, mode)\n top1b, top5b = _get_acc(scores, label_b, mode)\n top1 = lam * top1a + (1 - lam) * top1b\n top5 = lam * top5a + (1 - lam) * top5b\n return top1, top5\n def forward_model(self, imgs, model_name, model):\n if model_name in ['PPTSM_v2', 'ResNetTweaksTSM']:\n # [N,T,C,H,W] -> [N*T,C,H,W]\n imgs = paddle.reshape(imgs, [-1] + list(imgs.shape[2:]))\n return model(imgs)\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n out = {}\n loss_metrics = {}\n imgs = data_batch[0]\n labels = data_batch[1:]"
+ },
+ {
+ "comment": "This code defines a class that implements a recognizer for distillation. The model takes in an image and a model name, and returns the output from both the student and teacher models. It calculates loss using the student and teacher outputs, as well as top-1 and top-5 accuracy metrics from the student's output only. This is used for both training (train_step) and validation (val_step). The class utilizes a list of model names and corresponding models for both types, student and teacher, and iterates over them to apply the forward pass and calculate loss and metrics.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":166-192",
+ "content": " for idx, item in enumerate(self.model_name_list):\n model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n out[model_type] = self.forward_model(imgs, model_name, model)\n # out_student, out_teacher\n loss = self.get_loss(out, labels, 'Train')\n loss_metrics['loss'] = loss\n # calculate acc with student output\n top1, top5 = self.get_acc(out['Student'], labels)\n loss_metrics['top1'] = top1\n loss_metrics['top5'] = top5\n return loss_metrics\n def val_step(self, data_batch):\n out = {}\n loss_metrics = {}\n imgs = data_batch[0]\n labels = data_batch[1:]\n for idx, item in enumerate(self.model_name_list):\n model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n out[model_type] = self.forward_model(imgs, model_name, model)"
+ },
+ {
+ "comment": "In this code snippet, the get_loss and get_acc functions are used to calculate loss and accuracy metrics for a \"Student\" model. The test_step function tests the Student model using forward_model function, and the infer_step function is not implemented here. This code seems related to evaluating the performance of a student model in image recognition tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":194-223",
+ "content": " # Loss of student with gt: out_student, label\n loss = self.get_loss(out, labels, 'Val')\n loss_metrics['loss'] = loss\n top1, top5 = self.get_acc(out['Student'], labels, 'Val')\n loss_metrics['top1'] = top1\n loss_metrics['top5'] = top5\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n # Use Student to test\n for idx, item in enumerate(self.model_name_list):\n model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n if model_type == \"Student\":\n out = self.forward_model(imgs, model_name, model)\n return out\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n # Use Student to infer\n for idx, item in enumerate(self.model_name_list):"
+ },
+ {
+ "comment": "The code selects a model from the model_list based on the idx, and assigns its name to model_name. If the model type is \"Student\", it calls forward_model function passing imgs, model_name, and model as parameters, and returns the output.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerDistillation.py\":224-230",
+ "content": " model = self.model_list[idx]\n model_name = list(item.keys())[0]\n model_type = item[model_name] # Teacher or Student\n if model_type == \"Student\":\n out = self.forward_model(imgs, model_name, model)\n return out"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6f66441f-8644-4395-8219-f58f193d243f.json b/docs/doc/6f66441f-8644-4395-8219-f58f193d243f.json
new file mode 100644
index 000000000..87a254f82
--- /dev/null
+++ b/docs/doc/6f66441f-8644-4395-8219-f58f193d243f.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The \"AddsHead\" class in PaddleVideo handles object detection, loss calculation during training and metrics like abs_rel, rmse in inference, while supporting multi-GPU scenarios with all-reduce operations.",
+ "details": [
+ {
+ "comment": "This code is part of the PaddleVideo library, defining a class called AddsHead for object detection. It uses input features with specific number of channels and classes to be classified. The class is registered in the registry under HEADS. MIN_DEPTH and MAX_DEPTH constants define the minimum and maximum depth values respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/adds_head.py\":0-32",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport cv2\nimport numpy as np\nimport paddle.nn as nn\nfrom paddlevideo.utils import get_dist_info\nimport paddle\nfrom ..builder import build_loss\nfrom ..registry import HEADS\nMIN_DEPTH = 1e-3\nMAX_DEPTH = 80\n@HEADS.register()\nclass AddsHead(nn.Layer):\n \"\"\"TimeSformerHead Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature."
+ },
+ {
+ "comment": "The code represents the initialization and forward pass of a class named \"AddsHead\". The class takes in parameters like avg_reprojection, disparity_smoothness, no_ssim, etc. It builds a loss function using build_loss method with the provided configuration (loss_cfg). During training, it returns the result of the loss function on inputs and outputs. In inference mode, it uses get_metrics method to calculate metrics such as abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/adds_head.py\":33-61",
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n avg_reprojection,\n disparity_smoothness,\n no_ssim,\n loss_cfg=dict(name='ADDSLoss'),\n max_gt_depth=60,\n pred_depth_scale_factor=1):\n super(AddsHead, self).__init__()\n loss_cfg['avg_reprojection'] = avg_reprojection\n loss_cfg['disparity_smoothness'] = disparity_smoothness\n loss_cfg['no_ssim'] = no_ssim\n self.max_gt_depth = max_gt_depth\n self.pred_depth_scale_factor = pred_depth_scale_factor\n self.loss_func = build_loss(loss_cfg)\n def forward(self):\n raise NotImplemented\n def loss(self, inputs, outputs):\n if self.training:\n return self.loss_func(inputs, outputs)\n else:\n abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3 = self.get_metrics("
+ },
+ {
+ "comment": "This code snippet defines an \"AddsHead\" class that returns a dictionary of metrics including absolute relative error, squared relative error, root mean square error, and additional error measures. The get_metrics function resizes the predicted displacement to match the ground truth depth, scales and adjusts the predicted depth based on certain factors, and then computes the specified errors using another function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/adds_head.py\":62-94",
+ "content": " outputs['pred_disp'], outputs['gt'])\n outputs['abs_rel'] = abs_rel\n outputs['sq_rel'] = sq_rel\n outputs['rmse'] = rmse\n outputs['rmse_log'] = rmse_log\n outputs['a1'] = a1\n outputs['a2'] = a2\n outputs['a3'] = a3\n return outputs\n def get_metrics(self, pred_disp, gt_depth):\n gt_height, gt_width = gt_depth.shape[:2]\n pred_disp = cv2.resize(pred_disp, (gt_width, gt_height))\n pred_depth = 1 / pred_disp\n mask = gt_depth > 0\n pred_depth = pred_depth[mask]\n gt_depth = gt_depth[mask]\n pred_depth *= self.pred_depth_scale_factor\n ratio = np.median(gt_depth) / np.median(pred_depth)\n pred_depth *= ratio\n pred_depth[pred_depth < MIN_DEPTH] = MIN_DEPTH\n pred_depth[pred_depth > MAX_DEPTH] = MAX_DEPTH\n mask2 = gt_depth <= self.max_gt_depth\n pred_depth = pred_depth[mask2]\n gt_depth = gt_depth[mask2]\n abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3 = self.compute_errors("
+ },
+ {
+ "comment": "This code is performing all-reduce operations on tensors for multi-GPU scenarios, ensuring that the sum of tensor values across GPUs is reduced and then divided by the total number of participating GPUs. This allows for accurate averaging of results when working with multiple GPUs in a distributed environment.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/adds_head.py\":95-116",
+ "content": " gt_depth, pred_depth)\n _, world_size = get_dist_info()\n if world_size > 1:\n # educe sum when valid\n # TODO: there are some problems with multi gpu gather code.\n abs_rel = paddle.to_tensor(abs_rel)\n sq_rel = paddle.to_tensor(sq_rel)\n rmse = paddle.to_tensor(rmse)\n rmse_log = paddle.to_tensor(rmse_log)\n a1 = paddle.to_tensor(a1)\n a2 = paddle.to_tensor(a2)\n a3 = paddle.to_tensor(a3)\n abs_rel = paddle.distributed.all_reduce(\n abs_rel, op=paddle.distributed.ReduceOp.SUM) / world_size\n sq_rel = paddle.distributed.all_reduce(\n sq_rel, op=paddle.distributed.ReduceOp.SUM) / world_size\n rmse = paddle.distributed.all_reduce(\n rmse, op=paddle.distributed.ReduceOp.SUM) / world_size\n rmse_log = paddle.distributed.all_reduce(\n rmse_log, op=paddle.distributed.ReduceOp.SUM) / world_size\n a1 = paddle.distributed.all_reduce("
+ },
+ {
+ "comment": "The code defines a function that computes error metrics between predicted and ground truth depths. It uses all-reduce operations to calculate average values and returns multiple metrics including absolute relative error, squared relative error, RMSE, log RMSE, and three averages (a1, a2, a3) indicating the percentage of thresholds below certain levels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/adds_head.py\":117-143",
+ "content": " a1, op=paddle.distributed.ReduceOp.SUM) / world_size\n a2 = paddle.distributed.all_reduce(\n a2, op=paddle.distributed.ReduceOp.SUM) / world_size\n a3 = paddle.distributed.all_reduce(\n a3, op=paddle.distributed.ReduceOp.SUM) / world_size\n return abs_rel.item(), sq_rel.item(), rmse.item(), rmse_log.item(\n ), a1.item(), a2.item(), a3.item()\n return abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3\n def compute_errors(self, gt, pred):\n \"\"\"Computation of error metrics between predicted and ground truth depths\n \"\"\"\n thresh = np.maximum((gt / pred), (pred / gt))\n a1 = (thresh < 1.25).mean()\n a2 = (thresh < 1.25**2).mean()\n a3 = (thresh < 1.25**3).mean()\n rmse = (gt - pred)**2\n rmse = np.sqrt(rmse.mean())\n rmse_log = (np.log(gt) - np.log(pred))**2\n rmse_log = np.sqrt(rmse_log.mean())\n abs_rel = np.mean(np.abs(gt - pred) / gt)\n sq_rel = np.mean(((gt - pred)**2) / gt)"
+ },
+ {
+ "comment": "This code returns six metrics: abs_rel, sq_rel, rmse, rmse_log, a1, and a2. These metrics are likely related to evaluating the performance of some model or algorithm in a regression task. The 'abs_rel' might stand for absolute relative error, 'sq_rel' could be squared relative error, 'rmse' represents root mean squared error, 'rmse_log' could be the logarithm of rmse, and 'a1', 'a2', and 'a3' are possibly other evaluation metrics.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/adds_head.py\":145-145",
+ "content": " return abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6fa4e1bb-e7ce-485a-b81a-1905868b1ed0.json b/docs/doc/6fa4e1bb-e7ce-485a-b81a-1905868b1ed0.json
new file mode 100644
index 000000000..85a0b6d58
--- /dev/null
+++ b/docs/doc/6fa4e1bb-e7ce-485a-b81a-1905868b1ed0.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code creates a 2D image classifier model using PaddleVideo's RecognizerMRI, with train_step and val_step calculating loss metrics, and test_step for testing without calling head.loss during inference.",
+ "details": [
+ {
+ "comment": "Code is from PaddleVideo's RecognizerMRI class, a 2D recognizer model framework. It has a forward_net method that takes imgs as input and returns the output of the network. The number of segments is obtained from the image shape and used to call the self.head method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerMRI.py\":0-26",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nimport paddle\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerMRI(BaseRecognizer):\n \"\"\"2D recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n # NOTE: As the num_segs is an attribute of dataset phase, and didn't pass to build_head phase, should obtain it from imgs(paddle.Tensor) now, then call self.head method.\n num_segs = imgs.shape[\n 1] # imgs.shape=[N,T,C,H,W], for most commonly case"
+ },
+ {
+ "comment": "This code defines a model for image classification. It first reshapes and casts the input images to float32 type, then passes them through a backbone network if one is defined. After that, it sends the resulting feature map through a head network (if defined) to produce class scores. The train_step function uses these class scores to calculate loss metrics during training, while the val_step function performs similar operations but does not compute losses.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerMRI.py\":27-58",
+ "content": " imgs = paddle.reshape_(imgs, [-1] + list(imgs.shape[2:]))\n imgs = paddle.cast(imgs, \"float32\") #############\n imgs = imgs.unsqueeze(1)\n if self.backbone != None:\n feature = self.backbone(imgs)\n else:\n feature = imgs\n if self.head != None:\n cls_score = self.head(feature, num_segs)\n else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score, labels, if_top5=False)\n return loss_metrics\n def val_step(self, data_batch):\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score,"
+ },
+ {
+ "comment": "The code defines a test_step and infer_step function for a model, which takes in data_batch as input and returns the classification scores from the forward_net function. The test_step specifically mentions that during testing, the net won't call head.loss.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizerMRI.py\":59-75",
+ "content": " labels,\n valid_mode=True,\n if_top5=False)\n return loss_metrics\n def test_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n # NOTE: (shipping) when testing, the net won't call head.loss, we deal with the test processing in /paddlevideo/metrics\n imgs = data_batch[0]\n cls_score = self.forward_net(imgs)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to test, from input to output.\"\"\"\n imgs = data_batch[0]\n cls_score = self.forward_net(imgs)\n return cls_score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6fd87d67-4e55-45c8-978e-44ad5f14fddb.json b/docs/doc/6fd87d67-4e55-45c8-978e-44ad5f14fddb.json
new file mode 100644
index 000000000..b45fffc9c
--- /dev/null
+++ b/docs/doc/6fd87d67-4e55-45c8-978e-44ad5f14fddb.json
@@ -0,0 +1,100 @@
+{
+ "summary": "The code trains and applies Ma-Net model for video object detection, preprocesses images, visualizes results, and uses neural network segmentation and classification. The \"train_stage1.py\" file in PaddleVideo/applications/Ma-Net project creates a manager object and trains it with numbers as arguments or configuration.",
+ "details": [
+ {
+ "comment": "The code imports necessary libraries, defines classes for data loaders and networks, sets up device and environment configurations, and initializes the Manager class with optional parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":0-33",
+ "content": "import cv2\nimport paddle\nimport paddle.nn as nn\nimport os\nimport numpy as np\nfrom paddle.io import DataLoader\nimport paddle.optimizer as optim\nfrom paddle.vision import transforms\nfrom dataloaders.davis_2017_f import DAVIS2017_VOS_Train, DAVIS2017_VOS_Test\nimport dataloaders.custom_transforms_f as tr\nfrom dataloaders.samplers import RandomIdentitySampler\nfrom networks.deeplab import DeepLab\nfrom networks.IntVOS import IntVOS\nfrom networks.loss import Added_BCEWithLogitsLoss, Added_CrossEntropyLoss\nfrom config import cfg\nfrom utils.api import float_, clip_grad_norm_, int_, long_\nfrom utils.meters import AverageMeter\nfrom utils.mask_damaging import damage_masks\nfrom utils.utils import label2colormap\nfrom PIL import Image\nimport scipy.misc as sm\nimport time\n# import logging\npaddle.disable_static()\npaddle.device.set_device('gpu:0')\nclass Manager(object):\n def __init__(self,\n use_gpu=True,\n time_budget=None,\n save_result_dir=cfg.SAVE_RESULT_DIR,\n pretrained=True,"
+ },
+ {
+ "comment": "The code initializes a model for stage 1 training in Ma-Net application. It loads pretrained model if specified and sets the model to train mode. The `train` method starts the actual training by setting the model to train mode, initializing loss meters, and looping over batches to compute loss and time metrics.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":34-58",
+ "content": " interactive_test=False,\n freeze_bn=False):\n self.save_res_dir = save_result_dir\n self.time_budget = time_budget\n self.feature_extracter = DeepLab(backbone='resnet', freeze_bn=freeze_bn)\n if pretrained:\n pretrained_dict = paddle.load(cfg.PRETRAINED_MODEL)\n # pretrained_dict = np.load(cfg.PRETRAINED_MODEL, allow_pickle=True).item()\n pretrained_dict = pretrained_dict['state_dict']\n self.load_network(self.feature_extracter, pretrained_dict)\n print('load pretrained model successfully.')\n self.model = IntVOS(cfg, self.feature_extracter)\n self.use_gpu = use_gpu\n if use_gpu:\n self.model = self.model\n def train(self,\n damage_initial_previous_frame_mask=True,\n lossfunc='cross_entropy',\n model_resume=False):\n ###################\n self.model.train()\n running_loss = AverageMeter()\n running_time = AverageMeter()"
+ },
+ {
+ "comment": "This code defines a training stage for the Ma-Net model. It initializes parameters for three parts of the model: feature extractor, semantic embedding, and dynamic segment head. An optimizer using momentum is set up with specified learning rate, momentum, weight decay, and gradient clipping. A series of data transformations are applied to input images.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":60-86",
+ "content": " param_list = [{\n 'params': self.model.feature_extracter.parameters()\n }, {\n 'params': self.model.semantic_embedding.parameters()\n }, {\n 'params': self.model.dynamic_seghead.parameters()\n }]\n ########\n clip = paddle.nn.ClipGradByGlobalNorm(\n clip_norm=cfg.TRAIN_CLIP_GRAD_NORM)\n # clip = None\n optimizer = optim.Momentum(parameters=param_list,\n learning_rate=cfg.TRAIN_LR,\n momentum=cfg.TRAIN_MOMENTUM,\n weight_decay=cfg.TRAIN_WEIGHT_DECAY,\n use_nesterov=True,\n grad_clip=clip)\n self.param_list = param_list\n ###################\n composed_transforms = transforms.Compose([\n tr.RandomHorizontalFlip(cfg.DATA_RANDOMFLIP),\n tr.RandomScale(),\n tr.RandomCrop((cfg.DATA_RANDOMCROP, cfg.DATA_RANDOMCROP), 5),"
+ },
+ {
+ "comment": "The code initializes the training dataset and creates a data loader for it. It also defines the loss function based on user input, either binary cross-entropy or cross-entropy, and sets the maximum number of iterations to run. The dataset processing includes resizing and converting images to tensors using specified transforms.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":87-113",
+ "content": " tr.Resize(cfg.DATA_RESCALE),\n tr.ToTensor()\n ])\n print('dataset processing...')\n train_dataset = DAVIS2017_VOS_Train(root=cfg.DATA_ROOT,\n transform=composed_transforms)\n trainloader = DataLoader(\n train_dataset,\n collate_fn=None,\n batch_size=cfg.TRAIN_BATCH_SIZE,\n shuffle=True,\n num_workers=8,\n )\n print('dataset processing finished.')\n if lossfunc == 'bce':\n criterion = Added_BCEWithLogitsLoss(cfg.TRAIN_TOP_K_PERCENT_PIXELS,\n cfg.TRAIN_HARD_MINING_STEP)\n elif lossfunc == 'cross_entropy':\n criterion = Added_CrossEntropyLoss(cfg.TRAIN_TOP_K_PERCENT_PIXELS,\n cfg.TRAIN_HARD_MINING_STEP)\n else:\n print(\n 'unsupported loss funciton. Please choose from [cross_entropy,bce]'\n )\n max_itr = cfg.TRAIN_TOTAL_STEPS"
+ },
+ {
+ "comment": "This code initializes the model, loads previously saved data if specified and resumes training from a certain step. It keeps track of training time per epoch and adjusts learning rate as needed. The loop iterates through the dataset, performing operations on each sample until maximum number of iterations is reached.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":115-143",
+ "content": " step = 0\n if model_resume:\n saved_model_ = os.path.join(self.save_res_dir, cfg.TRAIN_RESUME_DIR)\n saved_model_ = paddle.load(saved_model_)\n self.model = self.load_network(self.model, saved_model_)\n step = int(cfg.RESUME_DIR.split('.')[0].split('_')[-1])\n print('resume from step {}'.format(step))\n while step < cfg.TRAIN_TOTAL_STEPS:\n if step > 100001:\n break\n t1 = time.time()\n if step > 0:\n running_time.update(time.time() - t1)\n print(\n f'{time.asctime()}: new epoch starts. last epoch time: {running_time.avg:.3f} s.',\n )\n for ii, sample in enumerate(trainloader):\n now_lr = self._adjust_lr(optimizer, step, max_itr)\n if step >= max_itr:\n step += 1\n break\n ref_imgs = sample['ref_img'] # batch_size * 3 * h * w\n img1s = sample['img1']"
+ },
+ {
+ "comment": "This code segment prepares the input data for a model by concatenating images and assigning labels. It also handles any potential errors with damage masks and adjusts the label1s accordingly. The GPU usage is conditionally set based on whether `self.use_gpu` is true or false.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":144-171",
+ "content": " img2s = sample['img2']\n ref_scribble_labels = sample[\n 'ref_scribble_label'] # batch_size * 1 * h * w\n label1s = sample['label1']\n label2s = sample['label2']\n seq_names = sample['meta']['seq_name']\n obj_nums = sample['meta']['obj_num']\n bs, _, h, w = img2s.shape\n inputs = paddle.concat((ref_imgs, img1s, img2s), 0)\n if damage_initial_previous_frame_mask:\n try:\n label1s = damage_masks(label1s)\n except:\n label1s = label1s\n print('damage_error')\n ##########\n if self.use_gpu:\n inputs = inputs\n ref_scribble_labels = ref_scribble_labels\n label1s = label1s\n label2s = label2s\n ##########\n tmp_dic = self.model(inputs,\n ref_scribble_labels,"
+ },
+ {
+ "comment": "This code initializes label and object dictionaries using the provided label1s. It then iterates over sequence names, creating key-value pairs for the label_and_obj_dic dictionary. For each sequence, it interpolates the temporary prediction logits with bilinear mode, aligning corners. Lastly, it retrieves the label and object number from the label_and_obj_dic and generates a tensor of object ids.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":172-193",
+ "content": " label1s,\n use_local_map=True,\n seq_names=seq_names,\n gt_ids=obj_nums,\n k_nearest_neighbors=cfg.KNNS)\n label_and_obj_dic = {}\n label_dic = {}\n for i, seq_ in enumerate(seq_names):\n label_and_obj_dic[seq_] = (label2s[i], obj_nums[i])\n for seq_ in tmp_dic.keys():\n tmp_pred_logits = tmp_dic[seq_]\n tmp_pred_logits = nn.functional.interpolate(\n tmp_pred_logits,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n tmp_dic[seq_] = tmp_pred_logits\n label_tmp, obj_num = label_and_obj_dic[seq_]\n obj_ids = np.arange(1, obj_num + 1)\n obj_ids = paddle.to_tensor(obj_ids)"
+ },
+ {
+ "comment": "This code snippet is training a video object detection model in stages. It applies binary cross-entropy or cross-entropy loss depending on the specified loss function, updates the model's parameters using an optimizer, and tracks the average loss during training. The progress is logged every 50 steps along with the current learning rate.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":194-216",
+ "content": " obj_ids = int_(obj_ids)\n if lossfunc == 'bce':\n label_tmp = label_tmp.transpose([1, 2, 0])\n label = (float_(label_tmp) == float_(obj_ids))\n label = label.unsqueeze(-1).transpose([3, 2, 0, 1])\n label_dic[seq_] = float_(label)\n elif lossfunc == 'cross_entropy':\n label_dic[seq_] = long_(label_tmp)\n loss = criterion(tmp_dic, label_dic, step)\n loss = loss / bs\n optimizer.clear_grad()\n loss.backward()\n optimizer.step()\n running_loss.update(loss.item(), bs)\n ##############Visulization during training\n if step % 50 == 0:\n print(time.asctime(), end='\\t')\n log = 'step:{},now_lr:{} ,loss:{:.4f}({:.4f})'.format(\n step, now_lr, running_loss.val, running_loss.avg)\n print(log)"
+ },
+ {
+ "comment": "This code extracts images, applies normalization, and visualizes the reference image, two input images, and a ground truth label. The images are normalized by subtracting mean values and dividing by standard deviation. The ground truth label is converted to a color map for visualization. The predicted output is interpolated to match the size of the other images before visualizing them.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":217-239",
+ "content": " # logging.info(log)\n show_ref_img = ref_imgs.numpy()[0]\n show_img1 = img1s.numpy()[0]\n show_img2 = img2s.numpy()[0]\n mean = np.array([[[0.485]], [[0.456]], [[0.406]]])\n sigma = np.array([[[0.229]], [[0.224]], [[0.225]]])\n show_ref_img = show_ref_img * sigma + mean\n show_img1 = show_img1 * sigma + mean\n show_img2 = show_img2 * sigma + mean\n show_gt = label2s[0]\n show_gt = show_gt.squeeze(0).numpy()\n show_gtf = label2colormap(show_gt).transpose((2, 0, 1))\n show_preds = tmp_dic[seq_names[0]]\n show_preds = nn.functional.interpolate(show_preds,\n size=(h, w),\n mode='bilinear',\n align_corners=True)"
+ },
+ {
+ "comment": "Applies sigmoid function to binary cross-entropy predictions, converts them to segmentation masks using argmax for cross entropy, calculates pixel accuracy, and saves network at specified intervals.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":240-265",
+ "content": " show_preds = show_preds.squeeze(0)\n if lossfunc == 'bce':\n show_preds = (paddle.nn.functional.sigmoid(show_preds) >\n 0.5)\n show_preds_s = paddle.zeros((h, w))\n for i in range(show_preds.size(0)):\n show_preds_s[show_preds[i]] = i + 1\n elif lossfunc == 'cross_entropy':\n show_preds_s = paddle.argmax(show_preds, axis=0)\n show_preds_s = show_preds_s.numpy()\n show_preds_sf = label2colormap(show_preds_s).transpose(\n (2, 0, 1))\n pix_acc = np.sum(show_preds_s == show_gt) / (h * w)\n ###########TODO\n if step % 20000 == 0 and step != 0:\n self.save_network(self.model, step)\n step += 1\n def test_VOS(self, use_gpu=True):\n seqs = []\n with open(\n os.path.join(cfg.DATA_ROOT, 'ImageSets', '2017',"
+ },
+ {
+ "comment": "Loading and preparing the test datasets for sequence processing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":266-285",
+ "content": " 'val' + '.txt')) as f:\n seqs_tmp = f.readlines()\n seqs_tmp = list(map(lambda elem: elem.strip(), seqs_tmp))\n seqs.extend(seqs_tmp)\n print('model loading...')\n saved_model_dict = os.path.join(self.save_res_dir, cfg.TEST_CHECKPOINT)\n pretrained_dict = paddle.load(saved_model_dict)\n self.model = self.load_network(self.model, pretrained_dict)\n print('model load finished')\n self.model.eval()\n with paddle.no_grad():\n for seq_name in seqs:\n print('prcessing seq:{}'.format(seq_name))\n test_dataset = DAVIS2017_VOS_Test(root=cfg.DATA_ROOT,\n transform=tr.ToTensor(),\n result_root=cfg.RESULT_ROOT,\n seq_name=seq_name)\n test_dataloader = DataLoader(test_dataset,\n batch_size=1,"
+ },
+ {
+ "comment": "This code creates a Paddle data loader for the test dataset, ensuring it doesn't overwrite existing results. It then iterates through each sample in the test loader, extracting necessary images and labels, and concatenating them into an input array. If GPU is used, the inputs and labels are transferred to GPU memory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":286-305",
+ "content": " shuffle=False,\n num_workers=0)\n if not os.path.exists(os.path.join(cfg.RESULT_ROOT, seq_name)):\n os.makedirs(os.path.join(cfg.RESULT_ROOT, seq_name))\n time_start = time.time()\n for ii, sample in enumerate(test_dataloader):\n ref_img = sample['ref_img']\n prev_img = sample['prev_img']\n current_img = sample['current_img']\n ref_label = sample['ref_label']\n prev_label = sample['prev_label']\n obj_num = sample['meta']['obj_num']\n seqnames = sample['meta']['seq_name']\n imgname = sample['meta']['current_name']\n bs, _, h, w = current_img.shape\n inputs = paddle.concat((ref_img, prev_img, current_img), 0)\n if use_gpu:\n inputs = inputs\n ref_label = ref_label"
+ },
+ {
+ "comment": "Feature extraction and model prediction for a video frame. Time measurement for feature extractor and model execution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":306-325",
+ "content": " prev_label = prev_label\n ################\n t1 = time.time()\n tmp = self.model.extract_feature(inputs)\n ref_frame_embedding, previous_frame_embedding, current_frame_embedding = paddle.split(\n tmp, num_or_sections=3, axis=0)\n t2 = time.time()\n print('feature_extracter time:{}'.format(t2 - t1))\n tmp_dic = self.model.prop_seghead(\n ref_frame_embedding, previous_frame_embedding,\n current_frame_embedding, ref_label, prev_label, True,\n seqnames, obj_num, cfg.KNNS, self.model.dynamic_seghead)\n t3 = time.time()\n print('after time:{}'.format(t3 - t2))\n #######################\n pred_label = tmp_dic[seq_name]\n pred_label = nn.functional.interpolate(pred_label,\n size=(h, w),"
+ },
+ {
+ "comment": "This code segment is part of a function that saves the predicted labels for each frame as an image with a palette. It extracts the predictions from a pre-trained network, converts them to an image, and saves it in a specified directory. The function also prints the time taken for processing each frame.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":326-347",
+ "content": " mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_label = pred_label.squeeze(0)\n pred_label = pred_label.numpy()\n im = Image.fromarray(pred_label.astype('uint8')).convert(\n 'P', )\n im.putpalette(_palette)\n im.save(\n os.path.join(cfg.RESULT_ROOT, seq_name,\n imgname[0].split('.')[0] + '.png'))\n one_frametime = time.time()\n print('seq name:{} frame:{} time:{}'.format(\n seq_name, imgname[0], one_frametime - time_start))\n time_start = time.time()\n def load_network(self, net, pretrained_dict):\n # pretrained_dict = pretrained_dict\n model_dict = net.state_dict()\n # 1. filter out unnecessary keys"
+ },
+ {
+ "comment": "This code snippet is part of a training process for a neural network called Ma-Net. It loads pretrained weights into the model and then saves the network at different steps. The learning rate is adjusted during the training process to improve performance. The _palette variable appears unrelated, as it stores RGB values for colors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":348-377",
+ "content": " pretrained_dict = {\n k: v\n for k, v in pretrained_dict.items() if k in model_dict\n }\n # 2. overwrite entries in the existing state dict\n # for k in model_dict:\n # if k not in pretrained_dict:\n # print(k, 'not in loaded weights.')\n model_dict.update(pretrained_dict)\n net.set_state_dict(model_dict)\n return net\n def save_network(self, net, step):\n save_path = self.save_res_dir\n if not os.path.exists(save_path):\n os.makedirs(save_path)\n save_file = 'save_step_%s.pth' % (step)\n paddle.save(net.state_dict(), os.path.join(save_path, save_file))\n def _adjust_lr(self, optimizer, itr, max_itr):\n now_lr = cfg.TRAIN_LR * (1 - itr / (max_itr + 1))**cfg.TRAIN_POWER\n optimizer._param_groups[0]['lr'] = now_lr\n return now_lr\n_palette = [\n 0, 0, 0, 128, 0, 0, 0, 128, 0, 128, 128, 0, 0, 0, 128, 128, 0, 128, 0, 128,\n 128, 128, 128, 128, 64, 0, 0, 191, 0, 0, 64, 128, 0, 191, 128, 0, 64, 0,"
+ },
+ {
+ "comment": "This code is a list of RGB values for different objects in an image, possibly for object detection or classification.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":378-390",
+ "content": " 128, 191, 0, 128, 64, 128, 128, 191, 128, 128, 0, 64, 0, 128, 64, 0, 0, 191,\n 0, 128, 191, 0, 0, 64, 128, 128, 64, 128, 22, 22, 22, 23, 23, 23, 24, 24,\n 24, 25, 25, 25, 26, 26, 26, 27, 27, 27, 28, 28, 28, 29, 29, 29, 30, 30, 30,\n 31, 31, 31, 32, 32, 32, 33, 33, 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 37,\n 37, 37, 38, 38, 38, 39, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 43, 43,\n 43, 44, 44, 44, 45, 45, 45, 46, 46, 46, 47, 47, 47, 48, 48, 48, 49, 49, 49,\n 50, 50, 50, 51, 51, 51, 52, 52, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56,\n 56, 56, 57, 57, 57, 58, 58, 58, 59, 59, 59, 60, 60, 60, 61, 61, 61, 62, 62,\n 62, 63, 63, 63, 64, 64, 64, 65, 65, 65, 66, 66, 66, 67, 67, 67, 68, 68, 68,\n 69, 69, 69, 70, 70, 70, 71, 71, 71, 72, 72, 72, 73, 73, 73, 74, 74, 74, 75,\n 75, 75, 76, 76, 76, 77, 77, 77, 78, 78, 78, 79, 79, 79, 80, 80, 80, 81, 81,\n 81, 82, 82, 82, 83, 83, 83, 84, 84, 84, 85, 85, 85, 86, 86, 86, 87, 87, 87,\n 88, 88, 88, 89, 89, 89, 90, 90, 90, 91, 91, 91, 92, 92, 92, 93, 93, 93, 94,"
+ },
+ {
+ "comment": "This code appears to contain a sequence of numbers, possibly representing some kind of iteration or indexing in the larger context of the script. The exact meaning would depend on the specifics of how and where this code is used within the \"train_stage1.py\" file of the PaddleVideo/applications/Ma-Net project.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":391-403",
+ "content": " 94, 94, 95, 95, 95, 96, 96, 96, 97, 97, 97, 98, 98, 98, 99, 99, 99, 100,\n 100, 100, 101, 101, 101, 102, 102, 102, 103, 103, 103, 104, 104, 104, 105,\n 105, 105, 106, 106, 106, 107, 107, 107, 108, 108, 108, 109, 109, 109, 110,\n 110, 110, 111, 111, 111, 112, 112, 112, 113, 113, 113, 114, 114, 114, 115,\n 115, 115, 116, 116, 116, 117, 117, 117, 118, 118, 118, 119, 119, 119, 120,\n 120, 120, 121, 121, 121, 122, 122, 122, 123, 123, 123, 124, 124, 124, 125,\n 125, 125, 126, 126, 126, 127, 127, 127, 128, 128, 128, 129, 129, 129, 130,\n 130, 130, 131, 131, 131, 132, 132, 132, 133, 133, 133, 134, 134, 134, 135,\n 135, 135, 136, 136, 136, 137, 137, 137, 138, 138, 138, 139, 139, 139, 140,\n 140, 140, 141, 141, 141, 142, 142, 142, 143, 143, 143, 144, 144, 144, 145,\n 145, 145, 146, 146, 146, 147, 147, 147, 148, 148, 148, 149, 149, 149, 150,\n 150, 150, 151, 151, 151, 152, 152, 152, 153, 153, 153, 154, 154, 154, 155,\n 155, 155, 156, 156, 156, 157, 157, 157, 158, 158, 158, 159, 159, 159, 160,"
+ },
+ {
+ "comment": "The code consists of a sequence of numbers, possibly representing frame indices or image IDs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":404-416",
+ "content": " 160, 160, 161, 161, 161, 162, 162, 162, 163, 163, 163, 164, 164, 164, 165,\n 165, 165, 166, 166, 166, 167, 167, 167, 168, 168, 168, 169, 169, 169, 170,\n 170, 170, 171, 171, 171, 172, 172, 172, 173, 173, 173, 174, 174, 174, 175,\n 175, 175, 176, 176, 176, 177, 177, 177, 178, 178, 178, 179, 179, 179, 180,\n 180, 180, 181, 181, 181, 182, 182, 182, 183, 183, 183, 184, 184, 184, 185,\n 185, 185, 186, 186, 186, 187, 187, 187, 188, 188, 188, 189, 189, 189, 190,\n 190, 190, 191, 191, 191, 192, 192, 192, 193, 193, 193, 194, 194, 194, 195,\n 195, 195, 196, 196, 196, 197, 197, 197, 198, 198, 198, 199, 199, 199, 200,\n 200, 200, 201, 201, 201, 202, 202, 202, 203, 203, 203, 204, 204, 204, 205,\n 205, 205, 206, 206, 206, 207, 207, 207, 208, 208, 208, 209, 209, 209, 210,\n 210, 210, 211, 211, 211, 212, 212, 212, 213, 213, 213, 214, 214, 214, 215,\n 215, 215, 216, 216, 216, 217, 217, 217, 218, 218, 218, 219, 219, 219, 220,\n 220, 220, 221, 221, 221, 222, 222, 222, 223, 223, 223, 224, 224, 224, 225,"
+ },
+ {
+ "comment": "The code creates a manager object and calls its train function. The list of numbers appears to be arguments or configuration for the training process, but without context it's difficult to determine their exact purpose.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage1.py\":417-428",
+ "content": " 225, 225, 226, 226, 226, 227, 227, 227, 228, 228, 228, 229, 229, 229, 230,\n 230, 230, 231, 231, 231, 232, 232, 232, 233, 233, 233, 234, 234, 234, 235,\n 235, 235, 236, 236, 236, 237, 237, 237, 238, 238, 238, 239, 239, 239, 240,\n 240, 240, 241, 241, 241, 242, 242, 242, 243, 243, 243, 244, 244, 244, 245,\n 245, 245, 246, 246, 246, 247, 247, 247, 248, 248, 248, 249, 249, 249, 250,\n 250, 250, 251, 251, 251, 252, 252, 252, 253, 253, 253, 254, 254, 254, 255,\n 255, 255\n]\nmanager = Manager()\nmanager.train()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/6ff25d46-6c9b-49dc-88ce-c9f0fdfdd53e.json b/docs/doc/6ff25d46-6c9b-49dc-88ce-c9f0fdfdd53e.json
new file mode 100644
index 000000000..c6163e057
--- /dev/null
+++ b/docs/doc/6ff25d46-6c9b-49dc-88ce-c9f0fdfdd53e.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code uses PaddlePaddle and AttentionLstmErnie for multimodal video tagging, including model building, adjusting batch size, calculating metrics, testing, and saving parameters. The main function handles argument parsing and evaluation.",
+ "details": [
+ {
+ "comment": "This code is an evaluation function for a multimodal video tagging application. It imports necessary libraries, disables dynamic memory allocation, sets up the PaddlePaddle environment, and includes functions for reading data, defining the model architecture, and calculating metrics. The code also defines a \"parse_args\" function to handle command line arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/eval_and_save_model.py\":0-36",
+ "content": "\"\"\"\neval main\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os\nimport sys\nimport time\nimport argparse\nimport logging\nimport pickle\nimport numpy as np\nimport paddle\npaddle.enable_static()\nimport paddle.static as static\nfrom accuracy_metrics import MetricsCalculator\nfrom datareader import get_reader\nfrom config import parse_config, merge_configs, print_configs\nfrom models.attention_lstm_ernie import AttentionLstmErnie\nfrom utils import test_with_pyreader\ndef parse_args():"
+ },
+ {
+ "comment": "This code defines an argument parser for the Paddle Video evaluate script. It allows users to input a model name, config file path, pretrain weights path, output path, use_gpu flag, and save_model_param_dir. The default values are provided for each argument in case they aren't specified by the user.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/eval_and_save_model.py\":37-63",
+ "content": " \"\"\"parse_args\n \"\"\"\n parser = argparse.ArgumentParser(\"Paddle Video evaluate script\")\n parser.add_argument('--model_name',\n type=str,\n default='BaiduNet',\n help='name of model to train.')\n parser.add_argument('--config',\n type=str,\n default='configs/conf.txt',\n help='path to config file of model')\n parser.add_argument(\n '--pretrain',\n type=str,\n default=None,\n help=\n 'path to pretrain weights. None to use default weights path in ~/.paddle/weights.'\n )\n parser.add_argument('--output', type=str, default=None, help='output path')\n parser.add_argument('--use_gpu',\n type=bool,\n default=True,\n help='default use gpu.')\n parser.add_argument('--save_model_param_dir',\n type=str,\n default=None,\n help='checkpoint path')"
+ },
+ {
+ "comment": "This code defines command-line arguments for saving and evaluating an inference model, parses the configuration file, builds a model using AttentionLstmErnie, and sets up static programs for evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/eval_and_save_model.py\":64-93",
+ "content": " parser.add_argument('--save_inference_model',\n type=str,\n default=None,\n help='save inference path')\n parser.add_argument('--save_only',\n action='store_true',\n default=False,\n help='only save model, do not evaluate model')\n args = parser.parse_args()\n return args\ndef evaluate(args):\n \"\"\"evaluate\n \"\"\"\n # parse config\n config = parse_config(args.config)\n valid_config = merge_configs(config, 'valid', vars(args))\n print_configs(valid_config, 'Valid')\n # build model\n valid_model = AttentionLstmErnie(args.model_name,\n valid_config,\n mode='valid')\n startup = static.Program()\n valid_prog = static.default_main_program().clone(for_test=True)\n with static.program_guard(valid_prog, startup):\n paddle.disable_static()\n valid_model.build_input(True)\n valid_model.build_model()"
+ },
+ {
+ "comment": "This code is loading the model from a specified directory, compiling the program and running it. It checks if the save weight directory exists and loads the test weights into the model. If necessary, it saves the inference model and if only saving the model is required, it exits. The batch size is adjusted by dividing it by a denominator.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/eval_and_save_model.py\":94-122",
+ "content": " valid_feeds = valid_model.feeds()\n valid_outputs = valid_model.outputs()\n valid_loss = valid_model.loss()\n valid_pyreader = valid_model.pyreader()\n paddle.enable_static()\n place = paddle.CUDAPlace(0) if args.use_gpu else paddle.CPUPlace()\n exe = static.Executor(place)\n exe.run(startup)\n compiled_valid_prog = static.CompiledProgram(valid_prog)\n # load weights\n assert os.path.exists(args.save_model_param_dir), \\\n \"Given save weight dir {} not exist.\".format(args.save_model_param_dir)\n valid_model.load_test_weights_file(exe, args.save_model_param_dir,\n valid_prog, place)\n if args.save_inference_model:\n save_model_params(exe, valid_prog, valid_model,\n args.save_inference_model)\n if args.save_only is True:\n print('save model only, exit')\n return\n # get reader\n bs_denominator = 1\n valid_config.VALID.batch_size = int(valid_config.VALID.batch_size /\n bs_denominator)"
+ },
+ {
+ "comment": "This code retrieves a valid reader, calculates metrics, and decorates the sample list generator with specified execution places. It then tests the model using the reader, program, and fetch list to obtain test loss and accuracy, which are printed. The function `save_model_params` saves the model parameters in the provided directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/eval_and_save_model.py\":123-144",
+ "content": " valid_reader = get_reader(args.model_name.upper(), 'valid', valid_config)\n # get metrics\n valid_metrics = MetricsCalculator(args.model_name.upper(), 'valid',\n valid_config)\n valid_fetch_list = [valid_loss.name] + [x.name for x in valid_outputs\n ] + [valid_feeds[-1].name]\n # get reader\n exe_places = static.cuda_places() if args.use_gpu else static.cpu_places()\n valid_pyreader.decorate_sample_list_generator(valid_reader,\n places=exe_places)\n test_loss, metrics_dict_test = test_with_pyreader(exe, compiled_valid_prog,\n valid_pyreader,\n valid_fetch_list,\n valid_metrics)\n test_acc1 = metrics_dict_test['avg_acc1']\n print(test_loss)\n print(test_acc1)\ndef save_model_params(exe, program, model_object, save_dir):"
+ },
+ {
+ "comment": "This code defines a function \"save_model_params\" that takes the directory path, saves the inference model, and specifies the feeded variable names, main program, target variables, executor, and filenames for the model and parameters. It also includes a main function that parses arguments and evaluates them.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/eval_and_save_model.py\":145-158",
+ "content": " \"\"\"save_model_params\n \"\"\"\n feeded_var_names = [var.name for var in model_object.feeds()][:-1]\n static.save_inference_model(dirname=save_dir,\n feeded_var_names=feeded_var_names,\n main_program=program,\n target_vars=model_object.outputs(),\n executor=exe,\n model_filename='model',\n params_filename='params')\nif __name__ == \"__main__\":\n args = parse_args()\n evaluate(args)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/70cc0e1d-d3a5-4774-8f93-8fd07d6bfeea.json b/docs/doc/70cc0e1d-d3a5-4774-8f93-8fd07d6bfeea.json
new file mode 100644
index 000000000..ee5b820e0
--- /dev/null
+++ b/docs/doc/70cc0e1d-d3a5-4774-8f93-8fd07d6bfeea.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines ActBertLoss and actBertLoss classes as loss functions for ActBert model, using CrossEntropyLoss and nn.KLDivLoss. The total loss is calculated by summing masked text, masked image, masked action, and next sentence losses, based on predictions and labels from various sources.",
+ "details": [
+ {
+ "comment": "This code defines the ActBertLoss class, which is a loss function for the ActBert model. It uses the CrossEntropyLoss from PaddlePaddle's nn library and takes two arguments: vocab_size and a_target_size. The class inherits from BaseWeightedLoss.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/actbert_loss.py\":0-31",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport numpy as np\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom ..registry import LOSSES\nfrom .base import BaseWeightedLoss\n@LOSSES.register()\nclass ActBertLoss(BaseWeightedLoss):\n \"\"\"Loss for ActBert model\n \"\"\"\n def __init__(self, vocab_size=30522, a_target_size=700):\n super().__init__()\n self.vocab_size = vocab_size\n self.a_target_size = a_target_size\n self.loss_fct = nn.CrossEntropyLoss(ignore_index=-1)"
+ },
+ {
+ "comment": "This code defines a class for an actBert loss function. It uses the nn.KLDivLoss function as a criterion and takes in prediction scores, sequence labels, image labels, image targets, action labels, and next sentence labels as input to compute the visual loss (img_loss). The prediction_scores_v variable is modified by removing the first element from each sequence, likely for consistency purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/actbert_loss.py\":32-49",
+ "content": " self.vis_criterion = nn.KLDivLoss(reduction=\"none\")\n def forward(self, prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score, \\\n text_labels, image_label, image_target, action_label, next_sentence_label):\n \"\"\"\n Args:\n text_label: text label(with mask). Shape: [batch_size, seqence_length]\n image_label: image label(with mask). Shape: [batch_size, region_length]\n image_target: label of image feature distribution,\n Shape: [batch_size, region_length-1, num_image_class](minus 1 for xxx).\n action label: action label(with mask), Shape: [batch_size, action_length]\n next_sentence_label: is next sentence or not. Shape: [batch_size]\n \"\"\"\n prediction_scores_v = prediction_scores_v[:,\n 1:] #8,37,1601 --> 8,36,1601\n img_loss = self.vis_criterion(\n F.log_softmax(prediction_scores_v, axis=2),"
+ },
+ {
+ "comment": "This code calculates a total loss by summing the masked text loss, masked image loss, masked action loss, and next sentence loss. The losses are calculated based on predictions and labels from various sources. The `loss_fct` function is used to compute these losses, and they are reshaped before being added together for the final total loss.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/actbert_loss.py\":50-74",
+ "content": " image_target #8,36,1601\n )\n masked_img_loss = paddle.sum(\n img_loss * (image_label == 1).unsqueeze(2).astype('float32')) / max(\n paddle.sum((image_label == 1).astype('float32')), 1e-6)\n masked_text_loss = self.loss_fct(\n prediction_scores_t.reshape([-1, self.vocab_size]), #8,36,30522\n text_labels.reshape([-1]), #8,36 # label -1 will be ignored\n )\n masked_action_loss = self.loss_fct(\n prediction_scores_a.reshape([-1, self.a_target_size]), #8,5,700\n action_label.reshape([-1]), #8,5\n )\n next_sentence_loss = self.loss_fct(\n seq_relationship_score.reshape([-1, 2]),\n next_sentence_label.reshape([-1]) #8,2\n )\n total_loss = masked_text_loss.unsqueeze(0) + masked_img_loss.unsqueeze(\n 0) + masked_action_loss.unsqueeze(0) + next_sentence_loss.unsqueeze(\n 0)\n return total_loss"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/71c0a7a0-1364-471c-ad22-19596c791961.json b/docs/doc/71c0a7a0-1364-471c-ad22-19596c791961.json
new file mode 100644
index 000000000..d7e02ec30
--- /dev/null
+++ b/docs/doc/71c0a7a0-1364-471c-ad22-19596c791961.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The FeatureReader class in Python reads data for YouTube-8M dataset, supports LSTM, Attention Cluster, and NextVLAD models, initializes feature reader with parameters, shuffles proposals, generates batches, and yields when batch size is reached.",
+ "details": [
+ {
+ "comment": "This code is a Python class called FeatureReader, which inherits from DataReader. It serves as a data reader for the YouTube-8M dataset, using features extracted by prior networks. It supports three models: LSTM, Attention Cluster, and NextVLAD. The class imports necessary libraries and modules to read, parse, and manipulate the dataset efficiently.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/feature_reader.py\":0-32",
+ "content": "\"\"\"\nattention-lstm feature reader\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport sys\ntry:\n import cPickle as pickle\n from cStringIO import StringIO\nexcept ImportError:\n import pickle\nimport numpy as np\nimport random\nimport code\nfrom .reader_utils import DataReader\nclass FeatureReader(DataReader):\n \"\"\"\n Data reader for youtube-8M dataset, which was stored as features extracted by prior networks\n This is for the three models: lstm, attention cluster, nextvlad"
+ },
+ {
+ "comment": "The code initializes a feature reader, takes in parameters such as name, mode, configuration, and material (featuring image, audio, and pcm features). It shuffles the proposals if in training mode. The reader function generates batches of data by iterating through the proposal list, extracting relevant features from specific ID ranges, and storing them in a batch_out list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/feature_reader.py\":34-70",
+ "content": " dataset cfg: num_classes\n batch_size\n list\n NextVlad only: eigen_file\n \"\"\"\n def __init__(self, name, mode, cfg, material=None):\n self.name = name\n self.mode = mode\n self.batch_size = cfg[self.name.upper()]['batch_size']\n self.feature = material['feature']\n self.proposal = material['proposal']\n self.fps = 5\n def create_reader(self):\n \"\"\"\n create_reader\n \"\"\"\n image_feature_list = self.feature['image_feature']\n audio_feature_list = self.feature['audio_feature']\n pcm_feature_list = self.feature['pcm_feature']\n pcm_feature_list = pcm_feature_list.reshape((pcm_feature_list.shape[0] * 5, 640))\n fl = self.proposal\n if self.mode == 'train':\n random.shuffle(fl)\n def reader():\n \"\"\"\n reader\n \"\"\"\n batch_out = []\n for prop_info in fl:\n start_id = int(prop_info['start'])\n end_id = int(prop_info['end'])"
+ },
+ {
+ "comment": "This code snippet is part of a feature reader for an action detection system. It reads image, audio, and pcm features from feature lists, concatenates them if needed, creates a batch, and yields the batch when it reaches the specified batch size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/feature_reader.py\":71-85",
+ "content": " bmn_score = float(prop_info['score'])\n try:\n image_feature = image_feature_list[start_id: end_id]\n audio_feature = audio_feature_list[int(start_id / self.fps): int(end_id / self.fps)]\n pcm_feature = pcm_feature_list[start_id: end_id]\n image_feature = np.concatenate((image_feature, pcm_feature), axis=1)\n batch_out.append((image_feature, audio_feature, 0, prop_info))\n if len(batch_out) == self.batch_size:\n yield batch_out\n batch_out = []\n except Exception as e:\n continue\n return reader"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/7341eb50-8847-4b5f-954c-8958e26d6f93.json b/docs/doc/7341eb50-8847-4b5f-954c-8958e26d6f93.json
new file mode 100644
index 000000000..29ca72e47
--- /dev/null
+++ b/docs/doc/7341eb50-8847-4b5f-954c-8958e26d6f93.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code lists various application cases in PaddleVideo, including football action detection, basketball action detection, table tennis action recognition, figure skating action identification, video tagging, multimodal video classification, video quality assessment, 3DMRI medical image recognition, video interactive segmentation tool, UAV detection, abnormal behavior detection, and human analysis scenario action recognition.",
+ "details": [
+ {
+ "comment": "This code lists various application cases in PaddleVideo, including football action detection, basketball action detection, table tennis action recognition, figure skating action identification, video tagging, multimodal video classification, video quality assessment, 3DMRI medical image recognition, video interactive segmentation tool, UAV detection, abnormal behavior detection, and human analysis scenario action recognition.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/README.md\":0-17",
+ "content": "# \u5e94\u7528\u6848\u4f8b\n## 1. \u6982\u89c8\n| Applications | Descriptions |\n| :--------------- | :-------- |\n| [FootballAction](./FootballAction) | \u8db3\u7403\u52a8\u4f5c\u68c0\u6d4b\u65b9\u6848|\n| [BasketballAction](./BasketballAction) | \u7bee\u7403\u52a8\u4f5c\u68c0\u6d4b\u65b9\u6848 |\n| [TableTennis](./TableTennis) | \u4e52\u4e53\u7403\u52a8\u4f5c\u8bc6\u522b\u65b9\u6848|\n| [FigureSkating](./FigureSkating) | \u82b1\u6837\u6ed1\u51b0\u52a8\u4f5c\u8bc6\u522b\u65b9\u6848|\n| [VideoTag](./VideoTag) | 3000\u7c7b\u5927\u89c4\u6a21\u89c6\u9891\u5206\u7c7b\u65b9\u6848 |\n| [MultimodalVideoTag](./MultimodalVideoTag) | \u591a\u6a21\u6001\u89c6\u9891\u5206\u7c7b\u65b9\u6848|\n| [VideoQualityAssessment](.s/VideoQualityAssessment) | \u89c6\u9891\u8d28\u91cf\u8bc4\u4f30\u65b9\u6848|\n| [PP-Care](./PP-Care) | 3DMRI\u533b\u7597\u56fe\u50cf\u8bc6\u522b\u65b9\u6848 |\n| [EIVideo](./EIVideo) | \u89c6\u9891\u4ea4\u4e92\u5f0f\u5206\u5272\u5de5\u5177|\n| [Anti-UAV](./Anti-UAV) |\u65e0\u4eba\u673a\u68c0\u6d4b\u65b9\u6848|\n| [AbnormalActionDetection](./AbnormalActionDetection) |\u5f02\u5e38\u884c\u4e3a\u68c0\u6d4b\u65b9\u6848|\n| [PP-Human](./PPHuman) | \u884c\u4eba\u5206\u6790\u573a\u666f\u52a8\u4f5c\u8bc6\u522b\u65b9\u6848 |"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/73925f43-4347-466a-8d2e-03273cc295ba.json b/docs/doc/73925f43-4347-466a-8d2e-03273cc295ba.json
new file mode 100644
index 000000000..a6f436c59
--- /dev/null
+++ b/docs/doc/73925f43-4347-466a-8d2e-03273cc295ba.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a `FrozenBatchNorm2d` class for batch normalization without updating statistics and a `DeepLab` class with backbone, ASPP module, decoder, and methods to freeze batch norm layers. It also provides a function that iterates through certain modules, yielding parameters requiring gradient updates for potentially applying different learning rates.",
+ "details": [
+ {
+ "comment": "The code defines a `FrozenBatchNorm2d` class that extends the `nn.Layer` and overrides the `forward()` function to perform batch normalization without updating statistics. The `DeepLab` class inherits from `nn.Layer` and serves as a backbone for the deeplab network architecture, incorporating a backbone network, ASPP module, and decoder.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/deeplab.py\":0-30",
+ "content": "import paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom networks.aspp import build_aspp\nfrom networks.decoder import build_decoder\nfrom networks.backbone import build_backbone\nclass FrozenBatchNorm2d(nn.Layer):\n def __init__(self, n):\n super(FrozenBatchNorm2d, self).__init__()\n self.register_buffer(\"weight\", paddle.ones(n))\n self.register_buffer(\"bias\", paddle.zeros(n))\n self.register_buffer(\"running_mean\", paddle.zeros(n))\n self.register_buffer(\"running_var\", paddle.ones(n))\n def forward(self, x):\n if x.dtype == paddle.float16:\n self.weight = self.weight.half()\n self.bias = self.bias.half()\n self.running_mean = self.running_mean.half()\n self.running_var = self.running_var.half()\n scale = self.weight * self.running_var.rsqrt()\n bias = self.bias - self.running_mean * scale\n scale = scale.reshape(1, -1, 1, 1)\n bias = bias.reshape(1, -1, 1, 1)\n return x * scale + bias\nclass DeepLab(nn.Layer):"
+ },
+ {
+ "comment": "This code defines the DeepLab class with an initializer that takes arguments for backbone, output stride, number of classes, and whether to freeze batch normalization layers. It also includes methods to freeze batch norm layers, retrieve parameters for 1x learning rate, and a forward pass function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/deeplab.py\":31-63",
+ "content": " def __init__(self,\n backbone='resnet',\n output_stride=16,\n num_classes=21,\n sync_bn=True,\n freeze_bn=False):\n super(DeepLab, self).__init__()\n if backbone == 'drn':\n output_stride = 8\n if freeze_bn == True:\n print(\"Use frozen BN in DeepLab\")\n BatchNorm = FrozenBatchNorm2d\n else:\n BatchNorm = nn.BatchNorm2D\n self.backbone = build_backbone(backbone, output_stride, BatchNorm)\n self.aspp = build_aspp(backbone, output_stride, BatchNorm)\n self.decoder = build_decoder(num_classes, backbone, BatchNorm)\n def forward(self, input):\n x, low_level_feat = self.backbone(input)\n x = self.aspp(x)\n x = self.decoder(x, low_level_feat)\n return x\n def freeze_bn(self):\n for m in self.sublayers():\n if isinstance(m, nn.BatchNorm2D):\n m.eval()\n def get_1x_lr_params(self):\n modules = [self.backbone]"
+ },
+ {
+ "comment": "This code defines a function that iterates through certain modules of the network, specifically looking for convolution and batch normalization layers. It then yields the parameters of these layers that require gradient updates. This process is used in both the main body and the get_10x_lr_params method to potentially apply different learning rates to specific parts of the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/networks/deeplab.py\":64-80",
+ "content": " for i in range(len(modules)):\n for m in modules[i].named_modules():\n if isinstance(m[1], nn.Conv2D) or isinstance(\n m[1], nn.BatchNorm2D):\n for p in m[1].parameters():\n if p.requires_grad:\n yield p\n def get_10x_lr_params(self):\n modules = [self.aspp, self.decoder]\n for i in range(len(modules)):\n for m in modules[i].named_modules():\n if isinstance(m[1], nn.Conv2D) or isinstance(\n m[1], nn.BatchNorm2D):\n for p in m[1].parameters():\n if p.requires_grad:\n yield p"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/73bbcc1a-2f27-45c9-8e1a-c6aca413d5a7.json b/docs/doc/73bbcc1a-2f27-45c9-8e1a-c6aca413d5a7.json
new file mode 100644
index 000000000..c9f7f6998
--- /dev/null
+++ b/docs/doc/73bbcc1a-2f27-45c9-8e1a-c6aca413d5a7.json
@@ -0,0 +1,25 @@
+{
+ "summary": "The code imports libraries, defines the RecognizerTransformer_MRI model class with forward method and training/validation steps, using loss metrics. It includes two inference methods: 'test_step' and 'infer_step', which split input into multiple views for classification score generation. The average_view function combines these scores across views, using either 'score' or 'prob' averaging types.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries, defines a class for the RecognizerTransformer_MRI model, and sets the input image shape. The forward_net method preprocesses input images by casting them to float32 type and adding an extra dimension for compatibility with the transformer architecture.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py\":0-31",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nimport paddle\nimport paddle.nn.functional as F\nfrom paddlevideo.utils import get_logger\nfrom ...registry import RECOGNIZERS\nfrom .base import BaseRecognizer\nlogger = get_logger(\"paddlevideo\")\n@RECOGNIZERS.register()\nclass RecognizerTransformer_MRI(BaseRecognizer):\n \"\"\"Transformer's recognizer model framework.\"\"\"\n def forward_net(self, imgs):\n # imgs.shape=[N,C,T,H,W], for transformer case\n imgs = paddle.cast(imgs, \"float32\") #############\n imgs = imgs.unsqueeze(1)\n if self.backbone != None:"
+ },
+ {
+ "comment": "This code defines a recognizer transformer model for image classification. The `forward_net` method processes images and returns class scores, while the `train_step` and `val_step` methods perform training and validation steps by passing data batches to the model and computing loss metrics using sigmoid activation and the head's loss function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py\":32-62",
+ "content": " feature = self.backbone(imgs)\n else:\n feature = imgs\n if self.head != None:\n cls_score = self.head(feature)\n else:\n cls_score = None\n return cls_score\n def train_step(self, data_batch):\n \"\"\"Define how the model is going to train, from input to output.\n \"\"\"\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score, labels, if_top5=False)\n return loss_metrics\n def val_step(self, data_batch):\n imgs = data_batch[0]\n labels = data_batch[1:]\n cls_score = self.forward_net(imgs)\n cls_score = paddle.nn.functional.sigmoid(cls_score)\n loss_metrics = self.head.loss(cls_score,\n labels,\n valid_mode=True,\n if_top5=False)\n return loss_metrics"
+ },
+ {
+ "comment": "The code defines two methods, 'test_step' and 'infer_step', for the model to infer from input to output. It splits the input into multiple views based on the number of segments in each view. For each view, it applies the forward network to generate a set of classification scores. Finally, it averages the scores across all views using the average_view method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py\":64-88",
+ "content": " def test_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n imgs = data_batch[0]\n num_views = imgs.shape[2] // self.backbone.seg_num\n cls_score = []\n for i in range(num_views):\n view = imgs[:, :, i * self.backbone.seg_num:(i + 1) *\n self.backbone.seg_num]\n cls_score.append(self.forward_net(view))\n cls_score = self.average_view(cls_score)\n return cls_score\n def infer_step(self, data_batch):\n \"\"\"Define how the model is going to infer, from input to output.\"\"\"\n imgs = data_batch[0]\n num_views = imgs.shape[2] // self.backbone.seg_num\n cls_score = []\n for i in range(num_views):\n view = imgs[:, :, i * self.backbone.seg_num:(i + 1) *\n self.backbone.seg_num]\n cls_score.append(self.forward_net(view))\n cls_score = self.average_view(cls_score)\n return cls_score\n def average_view(self, cls_score, average_type='score'):"
+ },
+ {
+ "comment": "This function combines the scores of multiple views, taking two arguments: a list of cls_scores and an optional average_type. It asserts that average_type is either 'score' or 'prob'. If 'score', it adds all scores in the list and divides by the count. If 'avg', it first applies softmax to each score, then adds them and divides by the count. Otherwise, it raises a NotImplementedError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/recognizers/recognizer_transformer_MRI.py\":89-103",
+ "content": " \"\"\"Combine the scores of different views\n Args:\n cls_score (list): Scores of multiple views\n average_type (str, optional): Average calculation method. Defaults to 'score'.\n \"\"\"\n assert average_type in ['score', 'prob'], \\\n f\"Currently only the average of 'score' or 'prob' is supported, but got {average_type}\"\n if average_type == 'score':\n return paddle.add_n(cls_score) / len(cls_score)\n elif average_type == 'avg':\n return paddle.add_n([F.softmax(score)\n for score in cls_score]) / len(cls_score)\n else:\n raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/73d53f98-0749-46af-9c68-554b9e10f342.json b/docs/doc/73d53f98-0749-46af-9c68-554b9e10f342.json
new file mode 100644
index 000000000..63a0f0f4c
--- /dev/null
+++ b/docs/doc/73d53f98-0749-46af-9c68-554b9e10f342.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code imports various head classes from different modules in the PaddleVideo library for video object detection, segmentation, or action recognition tasks, and adds them to the `__all__` list for easy access.",
+ "details": [
+ {
+ "comment": "This code is importing various classes from different modules in the PaddleVideo library. These classes represent different types of heads used in video modeling, such as AttentionLstmHead and BBoxHeadAVA. The code also includes licenses and copyright information for the PaddlePaddle Authors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/__init__.py\":0-24",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .adds_head import AddsHead\nfrom .asrf_head import ASRFHead\nfrom .attention_lstm_head import AttentionLstmHead, ActionAttentionLstmHead\nfrom .base import BaseHead\nfrom .bbox_head import BBoxHeadAVA\nfrom .cfbi_head import CollaborativeEnsemblerMS\nfrom .i3d_head import I3DHead\nfrom .movinet_head import MoViNetHead\nfrom .ms_tcn_head import MSTCNHead\nfrom .pptimesformer_head import ppTimeSformerHead\nfrom .pptsm_head import ppTSMHead"
+ },
+ {
+ "comment": "This code imports various head classes from different modules and adds them to the `__all__` list, making them accessible for import when using this module. These head classes are used in video object detection, segmentation or action recognition tasks. They include ppTSNHead, TSNHead, TSMHead, ppTSMHead, SlowFastHead, TimeSformerHead and more. Each head class has its own specific functionality for different tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/__init__.py\":25-48",
+ "content": "from .pptsn_head import ppTSNHead\nfrom .roi_head import AVARoIHead\nfrom .single_straight3d import SingleRoIExtractor3D\nfrom .slowfast_head import SlowFastHead\nfrom .stgcn_head import STGCNHead\nfrom .timesformer_head import TimeSformerHead\nfrom .transnetv2_head import TransNetV2Head\nfrom .tsm_head import TSMHead\nfrom .tsn_head import TSNHead\nfrom .ms_tcn_head import MSTCNHead\nfrom .asrf_head import ASRFHead\nfrom .ctrgcn_head import CTRGCNHead\nfrom .movinet_head import MoViNetHead\nfrom .agcn2s_head import AGCN2sHead\nfrom .token_shift_head import TokenShiftHead\n__all__ = [\n 'BaseHead', 'TSNHead', 'TSMHead', 'ppTSMHead', 'ppTSNHead', 'SlowFastHead',\n 'AttentionLstmHead', 'TimeSformerHead', 'STGCNHead', 'TransNetV2Head',\n 'I3DHead', 'SingleRoIExtractor3D', 'AVARoIHead', 'BBoxHeadAVA', 'AddsHead',\n 'ppTimeSformerHead', 'CollaborativeEnsemblerMS', 'MSTCNHead', 'ASRFHead',\n 'MoViNetHead', 'CTRGCNHead', 'TokenShiftHead', 'ActionAttentionLstmHead',\n 'AGCN2sHead'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/743e660a-5a99-4037-aa2b-61f14ad50502.json b/docs/doc/743e660a-5a99-4037-aa2b-61f14ad50502.json
new file mode 100644
index 000000000..fb153aab0
--- /dev/null
+++ b/docs/doc/743e660a-5a99-4037-aa2b-61f14ad50502.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The script initializes a larger application, sets up logging and imports modules, predicts video tags using PaddleVideo's models, configures parameters, builds the model, prepares inputs/outputs, runs inference, checks file existence, retrieves infer reader, sets up data feeder, fetches model outputs, collects results with video IDs, logs/saves average processing time, and checks GPU availability and version compatibility before running.",
+ "details": [
+ {
+ "comment": "This code appears to be an import and initialization script for a larger application. It sets up logging, imports various modules and libraries, checks the CUDA availability, and performs version checking. The code also includes licensing information and copyright notices.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/predict.py\":0-36",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport os\nimport sys\nimport time\nimport logging\nimport argparse\nimport ast\nimport numpy as np\nimport paddle\nimport paddle.static as static\ntry:\n import cPickle as pickle\nexcept:\n import pickle\nfrom utils.config_utils import *\nimport models\nfrom reader import get_reader\nfrom metrics import get_metrics\nfrom utils.utility import check_cuda\nfrom utils.utility import check_version\nlogging.root.handlers = []\nFORMAT = '[%(levelname)s: %(filename)s: %(lineno)4d]: %(message)s'"
+ },
+ {
+ "comment": "The code imports logging, sets up a logger with debug level and configures the format. It then defines a function 'parse_args' that uses argparse to set default values for model name, config file path, whether to use GPU or not, weight path, and batch size for inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/predict.py\":37-63",
+ "content": "logging.basicConfig(level=logging.DEBUG, format=FORMAT, stream=sys.stdout)\nlogger = logging.getLogger(__name__)\ndef parse_args():\n parser = argparse.ArgumentParser()\n parser.add_argument('--model_name',\n type=str,\n default='AttentionCluster',\n help='name of model to train.')\n parser.add_argument('--config',\n type=str,\n default='configs/attention_cluster.txt',\n help='path to config file of model')\n parser.add_argument('--use_gpu',\n type=ast.literal_eval,\n default=True,\n help='default use gpu.')\n parser.add_argument(\n '--weights',\n type=str,\n default='./data/checkpoints/AttentionLSTM_epoch9.pdparams',\n help='weight path.')\n parser.add_argument('--batch_size',\n type=int,\n default=1,\n help='sample number in a batch for inference.')"
+ },
+ {
+ "comment": "This code snippet is part of a Python script for video tag prediction. It uses an argument parser to specify input files, log intervals, top k predictions and output directory. The default directories and paths are provided if no arguments are specified by the user.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/predict.py\":64-86",
+ "content": " parser.add_argument('--filelist',\n type=str,\n default=None,\n help='path to inferenece data file lists file.')\n parser.add_argument('--log_interval',\n type=int,\n default=1,\n help='mini-batch interval to log.')\n parser.add_argument('--infer_topk',\n type=int,\n default=20,\n help='topk predictions to restore.')\n parser.add_argument('--save_dir',\n type=str,\n default=os.path.join('data', 'predict_results',\n 'attention_lstm'),\n help='directory to store results')\n parser.add_argument('--video_path',\n type=str,\n default=None,\n help='directory to store results')\n parser.add_argument('--label_file',\n type=str,"
+ },
+ {
+ "comment": "The code defines a function that takes arguments, parses the config file, and builds an inference model using PaddleVideo's models. It then builds the inputs and outputs of the model, sets up the Executor based on GPU availability, and runs the startup program. Finally, it checks if the video or filelist path exists before proceeding with the inference process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/predict.py\":87-114",
+ "content": " default='label_3396.txt',\n help='chinese label file path')\n args = parser.parse_args()\n return args\ndef infer(args):\n # parse config\n config = parse_config(args.config)\n infer_config = merge_configs(config, 'infer', vars(args))\n print_configs(infer_config, \"Infer\")\n infer_model = models.get_model(args.model_name, infer_config, mode='infer')\n infer_model.build_input(use_dataloader=False)\n infer_model.build_model()\n infer_feeds = infer_model.feeds()\n infer_outputs = infer_model.outputs()\n place = paddle.CUDAPlace(0) if args.use_gpu else paddle.CPUPlace()\n exe = static.Executor(place)\n exe.run(static.default_startup_program())\n filelist = args.filelist or infer_config.INFER.filelist\n filepath = args.video_path or infer_config.INFER.get('filepath', '')\n if filepath != '':\n assert os.path.exists(filepath), \"{} not exist.\".format(filepath)\n else:\n assert os.path.exists(filelist), \"{} not exist.\".format(filelist)"
+ },
+ {
+ "comment": "This code retrieves an infer reader, checks and loads weights for the model, sets up a data feeder, fetches outputs from the model, and collects results with video IDs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/predict.py\":116-140",
+ "content": " # get infer reader\n infer_reader = get_reader(args.model_name.upper(), 'infer', infer_config)\n if args.weights:\n assert os.path.exists(\n args.weights), \"Given weight dir {} not exist.\".format(args.weights)\n # if no weight files specified, download weights from paddle\n weights = args.weights or infer_model.get_weights()\n infer_model.load_test_weights(exe, weights, static.default_main_program())\n infer_feeder = paddle.fluid.DataFeeder(place=place, feed_list=infer_feeds)\n fetch_list = infer_model.fetches()\n infer_metrics = get_metrics(args.model_name.upper(), 'infer', infer_config)\n infer_metrics.reset()\n periods = []\n cur_time = time.time()\n for infer_iter, data in enumerate(infer_reader()):\n data_feed_in = [items[:-1] for items in data]\n video_id = [items[-1] for items in data]\n infer_outs = exe.run(fetch_list=fetch_list,\n feed=infer_feeder.feed(data_feed_in))\n infer_result_list = [item for item in infer_outs] + [video_id]"
+ },
+ {
+ "comment": "The code calculates the average processing time for each sample, logs the information, and saves the final output. It uses a log interval to report progress, and the `infer_metrics` object accumulates data for logging and saving. The code also checks for GPU availability and version compatibility before running.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/predict.py\":142-170",
+ "content": " prev_time = cur_time\n cur_time = time.time()\n period = cur_time - prev_time\n periods.append(period)\n infer_metrics.accumulate(infer_result_list)\n if args.log_interval > 0 and infer_iter % args.log_interval == 0:\n logger.info('Processed {} samples'.format(\n (infer_iter + 1) * len(video_id)))\n logger.info('[INFER] infer finished. average time: {}'.format(\n np.mean(periods)))\n if not os.path.isdir(args.save_dir):\n os.makedirs(args.save_dir)\n infer_metrics.finalize_and_log_out(savedir=args.save_dir,\n label_file=args.label_file)\nif __name__ == \"__main__\":\n args = parse_args()\n # check whether the installed paddle is compiled with GPU\n check_cuda(args.use_gpu)\n check_version()\n logger.info(args)\n infer(args)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/74717c3a-94e8-48a6-82cc-b5cc16dc457b.json b/docs/doc/74717c3a-94e8-48a6-82cc-b5cc16dc457b.json
new file mode 100644
index 000000000..be55baa12
--- /dev/null
+++ b/docs/doc/74717c3a-94e8-48a6-82cc-b5cc16dc457b.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code defines a TokenShiftHead class for classification tasks in Paddle. It inherits from BaseHead, uses Linear module, and returns classification scores after passing input tensor x through fully connected layer self.fc.",
+ "details": [
+ {
+ "comment": "The code is defining a class called TokenShiftHead, which is a Transformer head for classification tasks. It has attributes such as num_classes, in_channels, and num_seg (defaulted to 8). The class inherits from BaseHead and is registered under the HEADS registry. The code imports necessary modules and functions, and uses Paddle's Linear module for the layer implementation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/token_shift_head.py\":0-29",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom paddle.nn import Linear\nimport paddle\nfrom ..registry import HEADS\nfrom ..weight_init import trunc_normal_, weight_init_\nfrom .base import BaseHead\n@HEADS.register()\nclass TokenShiftHead(BaseHead):\n \"\"\"TokenShift Transformer Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature.\n num_seg(int): The number of segments. Default: 8. "
+ },
+ {
+ "comment": "__init__ function initializes the class with specified parameters, and init_weights is used to initialize the FC layer's parameters using truncated normal distribution.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/token_shift_head.py\":30-59",
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n ls_eps (float): Label smoothing epsilon. Default: 0.01.\n std (float): Std(Scale) Value in normal initilizar. Default: 0.02.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n num_seg=8,\n loss_cfg=dict(name='CrossEntropyLoss'),\n ls_eps=0.01,\n std=0.02,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, ls_eps)\n self.num_seg = num_seg\n self.std = std\n self.fc = Linear(self.in_channels, self.num_classes)\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'TruncatedNormal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.0,\n std=self.std)\n # NOTE: Temporarily use trunc_normal_ instead of TruncatedNormal"
+ },
+ {
+ "comment": "This code defines a TokenShiftHead, which performs classification tasks. The forward function takes input tensor x and passes it through fully connected layer self.fc, resulting in classification scores for each sample. It then reshapes the score to average predictions for every frame, finally squeezing the axis to return the final score.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/token_shift_head.py\":60-78",
+ "content": " trunc_normal_(self.fc.weight, std=self.std)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # XXX: check dropout location!\n # x.shape = [N, embed_dim]\n score = self.fc(x)\n # [N*T, num_class]\n _, _m = score.shape\n _t = self.num_seg\n score = score.reshape([-1, _t, _m])\n score = paddle.mean(score, 1) # averaging predictions for every frame\n score = paddle.squeeze(score, axis=1)\n return score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/75917cee-6bd6-49b8-9390-93ec47bdaa44.json b/docs/doc/75917cee-6bd6-49b8-9390-93ec47bdaa44.json
new file mode 100644
index 000000000..485d92a14
--- /dev/null
+++ b/docs/doc/75917cee-6bd6-49b8-9390-93ec47bdaa44.json
@@ -0,0 +1,10 @@
+{
+ "summary": "The code introduces a video action segmentation dataset that utilizes breakfast, 50salads, and gtea datasets. The pre-training model's extracted features are used for the dataset. The dataset tree and data tree structure are provided, along with details of their folder contents.",
+ "details": [
+ {
+ "comment": "The code introduces a video action segmentation dataset that utilizes breakfast, 50salads, and gtea datasets. The pre-training model's extracted features are used for the dataset. The dataset tree and data tree structure are provided, along with details of their folder contents.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/SegmentationDataset.md\":0-34",
+ "content": "English | [\u7b80\u4f53\u4e2d\u6587](../../zh-CN/dataset/SegmentationDataset.md)\n# Video Action Segmentation Dataset\nThe video motion segmentation model uses breakfast, 50salads and gtea data sets. The use method is to use the features extracted by the pre training model, which can be obtained from the ms-tcn official code base.[feat](https://zenodo.org/record/3625992#.Xiv9jGhKhPY)\n- Dataset tree\n```txt\n\u2500\u2500\u2500 gtea\n \u251c\u2500\u2500 features\n \u2502 \u251c\u2500\u2500 S1_Cheese_C1.npy\n \u2502 \u251c\u2500\u2500 S1_Coffee_C1.npy\n \u2502 \u251c\u2500\u2500 S1_CofHoney_C1.npy\n \u2502 \u2514\u2500\u2500 ...\n \u251c\u2500\u2500 groundTruth\n \u2502 \u251c\u2500\u2500 S1_Cheese_C1.txt\n \u2502 \u251c\u2500\u2500 S1_Coffee_C1.txt\n \u2502 \u251c\u2500\u2500 S1_CofHoney_C1.txt\n \u2502 \u2514\u2500\u2500 ...\n \u251c\u2500\u2500 splits\n \u2502 \u251c\u2500\u2500 test.split1.bundle\n \u2502 \u251c\u2500\u2500 test.split2.bundle\n \u2502 \u251c\u2500\u2500 test.split3.bundle\n \u2502 \u2514\u2500\u2500 ...\n \u2514\u2500\u2500 mapping.txt\n```\n- data tree\n```txt\n\u2500\u2500\u2500 data\n \u251c\u2500\u2500 50salads\n \u251c\u2500\u2500 breakfast\n \u251c\u2500\u2500 gtea\n \u2514\u2500\u2500 ...\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/75ac2cd6-4dbc-4b74-9b97-d5f8a78af490.json b/docs/doc/75ac2cd6-4dbc-4b74-9b97-d5f8a78af490.json
new file mode 100644
index 000000000..b74e0a8d2
--- /dev/null
+++ b/docs/doc/75ac2cd6-4dbc-4b74-9b97-d5f8a78af490.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code provides usage instructions for various tools in PaddleVideo. It shows how to retrieve model parameters, calculate FLOPs, and test an exported model (coming soon). The code examples use Python 3.7 and require specific configuration files.",
+ "details": [
+ {
+ "comment": "This code provides usage instructions for various tools in PaddleVideo. It shows how to retrieve model parameters, calculate FLOPs, and test an exported model (coming soon). The code examples use Python 3.7 and require specific configuration files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tools.md\":0-21",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../zh-CN/tools.md) | English\n# Tools\nThis page includes the usage of some useful tools in PaddleVideo.\n## Params\nTo get the params of a model.\n```shell\npython3.7 tools/summary.py -c configs/recognization/tsm/tsm.yaml\n```\n## FLOPS\nto print FLOPs.\n```shell\npython3.7 tools/summary.py -c configs/recognization/tsm/tsm.yaml --FLOPs\n```\n## Test the export model coming soon"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/75dbef95-cd79-46a1-83f1-5311534490bc.json b/docs/doc/75dbef95-cd79-46a1-83f1-5311534490bc.json
new file mode 100644
index 000000000..aee2c63d2
--- /dev/null
+++ b/docs/doc/75dbef95-cd79-46a1-83f1-5311534490bc.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The code implements the CFBI Video Object Segmentation model proposed by Baidu in ECCV 2020, considering background and foreground for segmentation, predicting on current frames given reference frame and previous frame. It follows DAVIS guidelines, uses \"cfbip_davis.yaml\" configuration file, pretrained weights \"CFBIp_davis.pdparams\", saves predictions to \"result_root\", provides evaluation metrics including J&F-Mean, and references checkpoint file \"CFBIp_r101_davis.pdparams\".",
+ "details": [
+ {
+ "comment": "This code describes the CFBI Video Object Segmentation model, proposed by Baidu in ECCV 2020. It considers background as important as foreground and uses collaborative integration for segmentation. The model predicts segmentation of current frames given reference frame and previous frame. Data preparation follows DAVIS guidelines.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/cfbi.md\":0-28",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/recognition/cfbi.md) | English\n# CFBI\n---\n## Contents\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Test](#Test)\n- [Reference](#Reference)\n## Introduction\nCFBI is a Video Object Segmentation model proposed by Baidu in ECCV 2020. This method consider background should be equally treated and thus propose Collaborative video object segmentation by Foreground-Background Integration (CFBI) approach. Our CFBI implicitly imposes the feature embedding from the target foreground object and its corresponding background to be contrastive, promoting the segmentation results accordingly. Given the image and target segmentation of the reference frame (the first frame) and the previous frame, the model will predict the segmentation of the current frame.\n
\n
![](\"../../../images/cfbi.png\")
\n
\n
![](\"../../../images/st-gcn.png\")
\n
\n
\n
\n
\n
![](\"./docs/images/features_en.png\")
\n
\n
![\"image-20210628210446041\"/](\"../../../images/timesformer_attention_arch.png\")
![\"image-20210628210446041\"](\"../../../images/timesformer_attention_visualize.png\")
\n
\n![](\"../../../images/asrf.png\")
\nMS-TCN Overview\n
\n## Data\nASRF can choose 50salads, breakfast, gtea as trianing set. Please refer to Video Action Segmentation dataset download and preparation doc [Video Action Segmentation dataset](../../dataset/SegmentationDataset.md)\nUnlike MS-TCN, ASRF model requires additional data construction. The script process is as follows\n```bash\npython data/50salads/prepare_asrf_data.py --dataset_dir data/\n```\n## Train\nAfter prepare dataset, we can run sprits."
+ },
+ {
+ "comment": "This code is running a training command for an action segmentation model (ASRF) on the GTEA dataset, specifically without using pre-training. It uses CUDA device 3 and a provided configuration file. The test command tests MS-TCN on a dataset using a previously trained model's weights. The index calculation in the test refers to an evaluation script provided by the original author of ms-tcn. The codebase is from the official ASRF repository in PyTorch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/asrf.md\":36-54",
+ "content": "```bash\n# gtea dataset\nexport CUDA_VISIBLE_DEVICES=3\npython3.7 main.py --validate -c configs/segmentation/asrf/asrf_gtea.yaml\n```\n- Start the training by using the above command line or script program. There is no need to use the pre training model. The video action segmentation model is usually a full convolution network. Due to the different lengths of videos, the `DATASET.batch_size` of the video action segmentation model is usually set to `1`, that is, batch training is not required. At present, only **single sample** training is supported.\n## Test\nTest MS-TCN on dataset scripts:\n```bash\npython main.py --test -c configs/segmentation/asrf/asrf_gtea.yaml --weights=./output/ASRF/ASRF_split_1.pdparams\n```\n- The specific implementation of the index is to calculate ACC, edit and F1 scores by referring to the test script[evel.py](https://github.com/yabufarha/ms-tcn/blob/master/eval.py) provided by the author of ms-tcn.\nThe reproduction of pytorch comes from the official [code base](https://github.com/yiskw713/asrf)"
+ },
+ {
+ "comment": "The code provides accuracy results for different models on three datasets, Breakfast, 50salads, and GTEA, using a 4 or 5-fold validation method as per the MS-TCN paper. The performance metrics include Accuracy (Acc), Edit Distance (Edit), and F1 scores at different thresholds (F1@0.1, F1@0.25, F1@0.5).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/asrf.md\":56-79",
+ "content": "- The evaluation method of data set adopts the folding verification method in ms-tcn paper, and the division method of folding is the same as that in ms-tcn paper.\nAccuracy on Breakfast dataset(4 folding verification):\n| Model | Acc | Edit | F1@0.1 | F1@0.25 | F1@0.5 |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| paper | 67.6% | 72.4% | 74.3% | 68.9% | 56.1% |\n| pytorch | 65.8% | 71.0% | 72.3% | 66.5% | 54.9% |\n| paddle | 66.1% | 71.9% | 73.3% | 67.9% | 55.7% |\nAccuracy on 50salads dataset(5 folding verification):\n| Model | Acc | Edit | F1@0.1 | F1@0.25 | F1@0.5 |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| paper | 84.5% | 79.3% | 82.9% | 83.5% | 77.3% |\n| pytorch | 81.4% | 75.6% | 82.7% | 81.2% | 77.2% |\n| paddle | 81.6% | 75.8% | 83.0% | 81.5% | 74.8% |\nAccuracy on gtea dataset(4 folding verification):\n| Model | Acc | Edit | F1@0.1 | F1@0.25 | F1@0.5 |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| paper | 77.3% | 83.7% | 89.4% | 87.8% | 79.8% |\n| pytorch | 76.3% | 79.6% | 87.3% | 85.8% | 74.9% |"
+ },
+ {
+ "comment": "Table showing model weight for gtea with corresponding F1@0.5 and checkpoint links, followed by command to export inference model for ASRF_gtea using given parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/asrf.md\":80-99",
+ "content": "| paddle | 77.1% | 83.3% | 88.9% | 87.5% | 79.1% |\nModel weight for gtea\nTest_Data| F1@0.5 | checkpoints |\n| :----: | :----: | :---- |\n| gtea_split1 | 72.4409 | [ASRF_gtea_split_1.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ASRF_gtea_split_1.pdparams) |\n| gtea_split2 | 76.6666 | [ASRF_gtea_split_2.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ASRF_gtea_split_2.pdparams) |\n| gtea_split3 | 84.5528 | [ASRF_gtea_split_3.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ASRF_gtea_split_3.pdparams) |\n| gtea_split4 | 82.6771 | [ASRF_gtea_split_4.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ASRF_gtea_split_4.pdparams) |\n## Infer\n### export inference model\n```bash\npython3.7 tools/export_model.py -c configs/segmentation/asrf/asrf_gtea.yaml \\\n -p data/ASRF_gtea_split_1.pdparams \\\n -o inference/ASRF\n```\nTo get model architecture file `ASRF.pdmodel` and parameters file `ASRF.pdiparams`, use:"
+ },
+ {
+ "comment": "This code provides an example of how to run model inference using the ASRF segmentation model from PaddleVideo. The input file should contain a list of .npy files, and the code demonstrates how to execute it with specific configuration, model, and parameter files. It also shows the location where the results will be written after inference is complete.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/asrf.md\":101-130",
+ "content": "- Args usage please refer to [Model Inference](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0/docs/zh-CN/start.md#2-%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86).\n### infer\nInput file are the file list for infering, for example:\n```\nS1_Cheese_C1.npy\nS1_CofHoney_C1.npy\nS1_Coffee_C1.npy\nS1_Hotdog_C1.npy\n...\n```\n```bash\npython3.7 tools/predict.py --input_file data/gtea/splits/test.split1.bundle \\\n --config configs/segmentation/asrf/asrf_gtea.yaml \\\n --model_file inference/ASRF/ASRF.pdmodel \\\n --params_file inference/ASRF/ASRF.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nexample of logs:\n```bash\nresult write in : ./inference/infer_results/S1_Cheese_C1.txt\nresult write in : ./inference/infer_results/S1_CofHoney_C1.txt\nresult write in : ./inference/infer_results/S1_Coffee_C1.txt\nresult write in : ./inference/infer_results/S1_Hotdog_C1.txt\nresult write in : ./inference/infer_results/S1_Pealate_C1.txt"
+ },
+ {
+ "comment": "Writes the inference results to separate text files for \"Peanut\" and \"Tea\" scenes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/segmentation/asrf.md\":131-138",
+ "content": "result write in : ./inference/infer_results/S1_Peanut_C1.txt\nresult write in : ./inference/infer_results/S1_Tea_C1.txt\n```\n## Reference\n- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://arxiv.org/pdf/2007.06866v1.pdf), Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, Hirokatsu Kataoka"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8bc86c23-1eb5-460c-92d4-d58ace8bfbc3.json b/docs/doc/8bc86c23-1eb5-460c-92d4-d58ace8bfbc3.json
new file mode 100644
index 000000000..5ad09918c
--- /dev/null
+++ b/docs/doc/8bc86c23-1eb5-460c-92d4-d58ace8bfbc3.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code imports libraries, registers a backbone model in PaddleVideo, initializes an ASRF class for computer vision tasks, and sets layer biases using init_bias function. The ASRF forward method performs convolution on input x and iterates through shared layers before returning the output.",
+ "details": [
+ {
+ "comment": "This code block is importing necessary libraries and modules, as well as registering a backbone model within the PaddleVideo framework. It also includes references to external repositories for inspiration or implementation guidance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/asrf.py\":0-29",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# https://github.com/yabufarha/ms-tcn/blob/master/model.py\n# https://github.com/yiskw713/asrf/libs/models/tcn.py\nimport paddle\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nimport numpy as np\nimport copy\nimport random\nimport math\nfrom paddle import ParamAttr\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom .ms_tcn import DilatedResidualLayer\nfrom ..framework.segmenters.utils import init_bias, KaimingUniform_like_torch"
+ },
+ {
+ "comment": "The ASRF class is a type of backbone model for computer vision tasks. It initializes convolutional layers and shared dilated residual layers, and sets their weights using KaimingUniform initialization. The number of features, stages, and layers are configurable parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/asrf.py\":32-64",
+ "content": "@BACKBONES.register()\nclass ASRF(nn.Layer):\n def __init__(self, in_channel, num_features, num_classes, num_stages,\n num_layers):\n super().__init__()\n self.in_channel = in_channel\n self.num_features = num_features\n self.num_classes = num_classes\n self.num_stages = num_stages\n self.num_layers = num_layers\n # define layers\n self.conv_in = nn.Conv1D(self.in_channel, self.num_features, 1)\n shared_layers = [\n DilatedResidualLayer(2**i, self.num_features, self.num_features)\n for i in range(self.num_layers)\n ]\n self.shared_layers = nn.LayerList(shared_layers)\n self.init_weights()\n def init_weights(self):\n \"\"\"\n initialize model layers' weight\n \"\"\"\n # init weight\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv1D):\n layer.weight.set_value(\n KaimingUniform_like_torch(layer.weight).astype('float32'))\n if layer.bias is not None:"
+ },
+ {
+ "comment": "This code sets the initial values of layer biases using init_bias function. The ASRF forward method performs convolution on input x, then iterates through shared layers to modify the output before returning it.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/asrf.py\":65-74",
+ "content": " layer.bias.set_value(\n init_bias(layer.weight, layer.bias).astype('float32'))\n def forward(self, x):\n \"\"\" ASRF forward\n \"\"\"\n out = self.conv_in(x)\n for layer in self.shared_layers:\n out = layer(out)\n return out"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8bdc6b2c-d69d-441b-9c9b-0e5a04ddfd3f.json b/docs/doc/8bdc6b2c-d69d-441b-9c9b-0e5a04ddfd3f.json
new file mode 100644
index 000000000..8ab35edac
--- /dev/null
+++ b/docs/doc/8bdc6b2c-d69d-441b-9c9b-0e5a04ddfd3f.json
@@ -0,0 +1,65 @@
+{
+ "summary": "TSMINFReader is a multiprocessing-enabled video reader in jpg format, applying transformations for machine learning models. It computes crop positions, performs random cropping, resizing, flipping, and centering on groups of images with fault-tolerant image reading.",
+ "details": [
+ {
+ "comment": "TSMINFReader is a data reader for video datasets in jpg format. It inherits from DataReader and takes parameters name, mode, and cfg. It supports multiprocessing for improved performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":0-36",
+ "content": "\"\"\"\ntsn frame reader\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nimport os\nimport sys\nimport random\nimport functools\nimport concurrent.futures\nimport multiprocessing\nimport numpy as np\nimport paddle\nfrom PIL import Image, ImageEnhance\nfrom .reader_utils import DataReader\nclass TSMINFReader(DataReader):\n \"\"\"\n Data reader for video dataset of jpg folder.\n \"\"\"\n def __init__(self, name, mode, cfg, material=None):\n super(TSMINFReader, self).__init__(name, mode, cfg)"
+ },
+ {
+ "comment": "The code sets various configuration parameters such as number of segments, segment length, short and target image sizes, batch size, reader threads, buffer size, video path, and image mean and standard deviation for a TSN inference reader. It then creates the TSN inference reader using these parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":37-65",
+ "content": " name = name.upper()\n self.num_seg = cfg[name]['num_seg']\n self.seglen = cfg[name]['seglen']\n self.short_size = cfg[name]['short_size']\n self.target_size = cfg[name]['target_size']\n self.batch_size = cfg[name]['batch_size']\n self.reader_threads = cfg[name]['reader_threads']\n self.buf_size = cfg[name]['buf_size']\n self.video_path = cfg[name]['frame_list']\n self.img_mean = np.array(cfg[name]['image_mean']).reshape(\n [3, 1, 1]).astype(np.float32)\n self.img_std = np.array(cfg[name]['image_std']).reshape(\n [3, 1, 1]).astype(np.float32)\n self.material = material\n def create_reader(self):\n \"\"\"\n batch loader for TSN\n \"\"\"\n _reader = self._inference_reader_creator_longvideo(\n self.video_path,\n self.mode,\n num_seg=self.num_seg,\n seglen=self.seglen,\n short_size=self.short_size,\n target_size=self.target_size,\n img_mean=self.img_mean,"
+ },
+ {
+ "comment": "This code defines a video inference reader for the PaddleVideo application's BasketballAction module. It creates a batch reader to process images and labels from the given video path, considering various parameters such as image mean, standard deviation, number of threads, and buffer size. The batch reader yields batches of images and labels until reaching the specified batch size or finishing processing all data.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":66-96",
+ "content": " img_std=self.img_std,\n num_threads=self.reader_threads,\n buf_size=self.buf_size)\n def _batch_reader():\n batch_out = []\n for imgs, label in _reader():\n if imgs is None:\n continue\n batch_out.append((imgs, label))\n if len(batch_out) == self.batch_size:\n yield batch_out\n batch_out = []\n if len(batch_out) > 1:\n yield batch_out[:-1]\n return _batch_reader\n def _inference_reader_creator_longvideo(self, video_path, mode, num_seg,\n seglen, short_size, target_size,\n img_mean, img_std, num_threads,\n buf_size):\n \"\"\"\n inference reader for video\n \"\"\"\n def reader():\n \"\"\"\n reader\n \"\"\"\n def image_buf(image_id_path_buf):\n \"\"\""
+ },
+ {
+ "comment": "This code uses multithreading to process video frames into images. It opens image paths, converts them to RGB format, and stores them in a list for later use. The code then maps the image processing task onto multiple threads to increase efficiency. Finally, it collects the processed images from each thread and stores them in the imgs_seg_list variable.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":97-119",
+ "content": " image_buf reader\n \"\"\"\n try:\n img_path = image_id_path_buf[1]\n img = Image.open(img_path).convert(\"RGB\")\n image_id_path_buf[2] = img\n except:\n image_id_path_buf[2] = None\n frame_len = len(video_path)\n read_thread_num = num_seg\n for i in range(0, frame_len, read_thread_num):\n image_list_part = video_path[i:i + read_thread_num]\n image_id_path_buf_list = []\n for k in range(len(image_list_part)):\n image_id_path_buf_list.append([k, image_list_part[k], None])\n with concurrent.futures.ThreadPoolExecutor(\n max_workers=read_thread_num) as executor:\n executor.map(\n lambda image_id_path_buf: image_buf(image_id_path_buf),\n image_id_path_buf_list)\n imgs_seg_list = [x[2] for x in image_id_path_buf_list]"
+ },
+ {
+ "comment": "This code handles fault-tolerant reading of images for a specified range. It checks if the image buffer is None and if so, attempts to retrieve it from other segments. If an image cannot be retrieved, it prints an error message and exits. Additionally, it appends extra segments with the last image in case there are fewer than num_segments required. Finally, it yields the updated imgs_seg_list for further processing in the inference_imgs_transform function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":121-143",
+ "content": " # add the fault-tolerant for bad image\n for k in range(len(image_id_path_buf_list)):\n img_buf = image_id_path_buf_list[k][2]\n pad_id = 1\n while pad_id < num_seg and img_buf is None:\n img_buf = imgs_seg_list[(k + pad_id) % num_seg][2]\n if img_buf is None:\n print(\"read img erro from {} to {}\".format(\n i, i + read_thread_num))\n exit(0)\n else:\n imgs_seg_list[k] = img_buf\n for pad_id in range(len(imgs_seg_list), num_seg):\n imgs_seg_list.append(imgs_seg_list[-1])\n yield imgs_seg_list\n def inference_imgs_transform(imgs_list, mode, num_seg, seglen, short_size,\\\n target_size, img_mean, img_std):\n \"\"\"\n inference_imgs_transform\n \"\"\"\n imgs_ret = imgs_transform(imgs_list, mode, num_seg, seglen,"
+ },
+ {
+ "comment": "This code defines a function `imgs_transform` that takes in images, mode, number of segments, segment length, short size, target size as input. It applies transformations to the images based on the given parameters and returns the transformed images. The `mapper` is defined as a partial function of `inference_imgs_transform`, with parameters such as mode, num_seg, seglen, short_size, target_size, img_mean, and img_std. Finally, the code returns the result of `paddle.reader.xmap_readers` which applies the mapper function to the reader, with given parameters like num_threads and buf_size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":144-171",
+ "content": " short_size, target_size, img_mean,\n img_std)\n label_ret = 0\n return imgs_ret, label_ret\n mapper = functools.partial(inference_imgs_transform,\n mode=mode,\n num_seg=num_seg,\n seglen=seglen,\n short_size=short_size,\n target_size=target_size,\n img_mean=img_mean,\n img_std=img_std)\n return paddle.reader.xmap_readers(mapper,\n reader,\n num_threads,\n buf_size,\n order=True)\ndef imgs_transform(imgs,\n mode,\n num_seg,\n seglen,\n short_size,\n target_size,"
+ },
+ {
+ "comment": "The code defines a function \"imgs_transform\" that takes in images and applies various transformations depending on the mode ('train' or 'test'). For training, it performs multi-scale cropping, random cropping, and random flipping. For testing, it centers crops the images. It then normalizes the images by subtracting the mean and dividing by standard deviation. Finally, it reshapes the images into a specific format and returns them.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":172-202",
+ "content": " img_mean,\n img_std,\n name=''):\n \"\"\"\n imgs_transform\n \"\"\"\n imgs = group_scale(imgs, short_size)\n if mode == 'train':\n if name == \"TSM\":\n imgs = group_multi_scale_crop(imgs, short_size)\n imgs = group_random_crop(imgs, target_size)\n imgs = group_random_flip(imgs)\n else:\n imgs = group_center_crop(imgs, target_size)\n np_imgs = (np.array(imgs[0]).astype('float32').transpose(\n (2, 0, 1))).reshape(1, 3, target_size, target_size) / 255\n for i in range(len(imgs) - 1):\n img = (np.array(imgs[i + 1]).astype('float32').transpose(\n (2, 0, 1))).reshape(1, 3, target_size, target_size) / 255\n np_imgs = np.concatenate((np_imgs, img))\n imgs = np_imgs\n imgs -= img_mean\n imgs /= img_std\n imgs = np.reshape(imgs, (num_seg, seglen * 3, target_size, target_size))\n return imgs\ndef group_multi_scale_crop(img_group, target_size, scales=None, \\\n max_distort=1, fix_crop=True, more_fix_crop=True):"
+ },
+ {
+ "comment": "This code generates a random crop size based on predefined scales and applies it to an input image. It ensures the generated crop size is close to the target size and adjusts the offset to maintain aspect ratio if necessary. The function also handles cases where the maximum distance between width and height is specified by the max_distort parameter. If fix_crop is False, it further adds random offsets to the selected crop size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":203-238",
+ "content": " \"\"\"\n group_multi_scale_crop\n \"\"\"\n scales = scales if scales is not None else [1, .875, .75, .66]\n input_size = [target_size, target_size]\n im_size = img_group[0].size\n # get random crop offset\n def _sample_crop_size(im_size):\n \"\"\"\n _sample_crop_size\n \"\"\"\n image_w, image_h = im_size[0], im_size[1]\n base_size = min(image_w, image_h)\n crop_sizes = [int(base_size * x) for x in scales]\n crop_h = [\n input_size[1] if abs(x - input_size[1]) < 3 else x\n for x in crop_sizes\n ]\n crop_w = [\n input_size[0] if abs(x - input_size[0]) < 3 else x\n for x in crop_sizes\n ]\n pairs = []\n for i, h in enumerate(crop_h):\n for j, w in enumerate(crop_w):\n if abs(i - j) <= max_distort:\n pairs.append((w, h))\n crop_pair = random.choice(pairs)\n if not fix_crop:\n w_offset = random.randint(0, image_w - crop_pair[0])\n h_offset = random.randint(0, image_h - crop_pair[1])"
+ },
+ {
+ "comment": "This code calculates crop positions for an image based on its width and height. It generates a list of potential cropping locations, including upper left/right, lower left/right, center, center left/right, upper left quarter, upper right quarter, and center. The calculations are done in case more_fix_crop is set to True, otherwise only the basic crop positions will be included.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":239-261",
+ "content": " else:\n w_step = (image_w - crop_pair[0]) / 4\n h_step = (image_h - crop_pair[1]) / 4\n ret = list()\n ret.append((0, 0)) # upper left\n if w_step != 0:\n ret.append((4 * w_step, 0)) # upper right\n if h_step != 0:\n ret.append((0, 4 * h_step)) # lower left\n if h_step != 0 and w_step != 0:\n ret.append((4 * w_step, 4 * h_step)) # lower right\n if h_step != 0 or w_step != 0:\n ret.append((2 * w_step, 2 * h_step)) # center\n if more_fix_crop:\n ret.append((0, 2 * h_step)) # center left\n ret.append((4 * w_step, 2 * h_step)) # center right\n ret.append((2 * w_step, 4 * h_step)) # lower center\n ret.append((2 * w_step, 0 * h_step)) # upper center\n ret.append((1 * w_step, 1 * h_step)) # upper left quarter\n ret.append((3 * w_step, 1 * h_step)) # upper right quarter"
+ },
+ {
+ "comment": "The code defines a function `group_random_crop` that takes an image group and a target size as input, performs random cropping on each image in the group with different crop sizes, offsets, and resizes them to the specified target size. The cropped images are then returned in a group.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":262-297",
+ "content": " ret.append((1 * w_step, 3 * h_step)) # lower left quarter\n ret.append((3 * w_step, 3 * h_step)) # lower righ quarter\n w_offset, h_offset = random.choice(ret)\n crop_info = {\n 'crop_w': crop_pair[0],\n 'crop_h': crop_pair[1],\n 'offset_w': w_offset,\n 'offset_h': h_offset\n }\n return crop_info\n crop_info = _sample_crop_size(im_size)\n crop_w = crop_info['crop_w']\n crop_h = crop_info['crop_h']\n offset_w = crop_info['offset_w']\n offset_h = crop_info['offset_h']\n crop_img_group = [\n img.crop((offset_w, offset_h, offset_w + crop_w, offset_h + crop_h))\n for img in img_group\n ]\n ret_img_group = [\n img.resize((input_size[0], input_size[1]), Image.BILINEAR)\n for img in crop_img_group\n ]\n return ret_img_group\ndef group_random_crop(img_group, target_size):\n \"\"\"\n group_random_crop\n \"\"\"\n w, h = img_group[0].size\n th, tw = target_size, target_size"
+ },
+ {
+ "comment": "This code is used to preprocess images by cropping, flipping, and centering them for machine learning models. The \"group_center_crop\" function crops images to a specific target size while ensuring the image dimensions are larger than the crop size. The \"group_random_flip\" function randomly flips the images horizontally with a 50% chance. The preprocessed images are returned in a list format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":299-337",
+ "content": " assert (w >= target_size) and (h >= target_size), \\\n \"image width({}) and height({}) should be larger than crop size\".format(w, h)\n out_images = []\n x1 = random.randint(0, w - tw)\n y1 = random.randint(0, h - th)\n for img in img_group:\n if w == tw and h == th:\n out_images.append(img)\n else:\n out_images.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n return out_images\ndef group_random_flip(img_group):\n \"\"\"\n group_random_flip\n \"\"\"\n v = random.random()\n if v < 0.5:\n ret = [img.transpose(Image.FLIP_LEFT_RIGHT) for img in img_group]\n return ret\n else:\n return img_group\ndef group_center_crop(img_group, target_size):\n \"\"\"\n group_center_crop\n \"\"\"\n img_crop = []\n for img in img_group:\n w, h = img.size\n th, tw = target_size, target_size\n assert (w >= target_size) and (h >= target_size), \\\n \"image width({}) and height({}) should be larger than crop size\".format(w, h)\n x1 = int(round((w - tw) / 2.))"
+ },
+ {
+ "comment": "The code defines two functions: \"crop_imgs\" and \"group_scale\". The \"crop_imgs\" function takes an image and crops it based on the provided top-left x, y coordinates, width, and height. It then appends the cropped images to a list and returns that list. The \"group_scale\" function resizes a group of images to a target size by checking if each image's dimensions already match the target size, and if not, it adjusts the dimensions using a 4:3 aspect ratio. It then appends the resized images to a list and returns that list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/reader/tsminf_reader.py\":338-365",
+ "content": " y1 = int(round((h - th) / 2.))\n img_crop.append(img.crop((x1, y1, x1 + tw, y1 + th)))\n return img_crop\ndef group_scale(imgs, target_size):\n \"\"\"\n group_scale\n \"\"\"\n resized_imgs = []\n for i in range(len(imgs)):\n img = imgs[i]\n w, h = img.size\n if (w <= h and w == target_size) or (h <= w and h == target_size):\n resized_imgs.append(img)\n continue\n if w < h:\n ow = target_size\n oh = int(target_size * 4.0 / 3.0)\n resized_imgs.append(img.resize((ow, oh), Image.BILINEAR))\n else:\n oh = target_size\n ow = int(target_size * 4.0 / 3.0)\n resized_imgs.append(img.resize((ow, oh), Image.BILINEAR))\n return resized_imgs"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8d127c8b-b6bb-428d-adf0-597f36b6649f.json b/docs/doc/8d127c8b-b6bb-428d-adf0-597f36b6649f.json
new file mode 100644
index 000000000..760bdb453
--- /dev/null
+++ b/docs/doc/8d127c8b-b6bb-428d-adf0-597f36b6649f.json
@@ -0,0 +1,15 @@
+{
+ "summary": "RoIAlign is a class for region of interest alignment. It takes features, regions of interest (roi), and number of roi as inputs, and uses PaddlePaddle's roi_align operation to extract aligned features.",
+ "details": [
+ {
+ "comment": "RoIAlign is a class used for region of interest alignment. It takes features, regions of interest (roi), and the number of roi as inputs. The aligned parameter specifies whether to return aligned features or not.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/roi_extractor.py\":0-30",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\n#@register\nclass RoIAlign(object):\n def __init__(self,\n resolution=14,\n spatial_scale=0.0625,\n sampling_ratio=0,\n aligned=False):\n super(RoIAlign, self).__init__()\n self.resolution = resolution\n self.spatial_scale = spatial_scale\n self.sampling_ratio = sampling_ratio\n self.aligned = aligned\n def __call__(self, feats, roi, rois_num):"
+ },
+ {
+ "comment": "This code concatenates ROIs and ensures correct data type, then uses the PaddlePaddle library's roi_align operation to extract features from input features (feats) based on ROIs. If there is only one feature, it performs alignment for all ROIs. Otherwise, it creates a list of aligned feature tensors.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/roi_extractor.py\":31-52",
+ "content": " roi = paddle.concat(roi) if len(roi) > 1 else roi[0]\n rois_num = paddle.to_tensor(rois_num, dtype='int32')\n rois_num = paddle.cast(rois_num, dtype='int32')\n if len(feats) == 1:\n roi_feat = paddle.vision.ops.roi_align(feats,\n roi,\n rois_num,\n self.resolution,\n self.spatial_scale,\n self.sampling_ratio,\n self.aligned)\n else:\n rois_feat_list = []\n roi_feat = paddle.vision.ops.roi_align(feats,\n roi,\n rois_num,\n self.resolution,\n self.spatial_scale,\n self.sampling_ratio,\n self.aligned)\n return roi_feat"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8d2bff4b-1fa7-4560-bc92-160a274fd5a9.json b/docs/doc/8d2bff4b-1fa7-4560-bc92-160a274fd5a9.json
new file mode 100644
index 000000000..305b37326
--- /dev/null
+++ b/docs/doc/8d2bff4b-1fa7-4560-bc92-160a274fd5a9.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is the initialization file of PaddleVideo's partitioner module. It imports base and TransNetV2Partitioner classes, and declares them as part of the public interface (__all__). The license and copyright information are also included.",
+ "details": [
+ {
+ "comment": "This code is the initialization file of PaddleVideo's partitioner module. It imports base and TransNetV2Partitioner classes, and declares them as part of the public interface (__all__). The license and copyright information are also included.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/partitioners/__init__.py\":0-17",
+ "content": "# copyright (c) 2020 paddlepaddle authors. all rights reserved.\n#\n# licensed under the apache license, version 2.0 (the \"license\"\n# you may not use this file except in compliance with the license.\n# you may obtain a copy of the license at\n#\n# http://www.apache.org/licenses/license-2.0\n#\n# unless required by applicable law or agreed to in writing, software\n# distributed under the license is distributed on an \"as is\" basis,\n# without warranties or conditions of any kind, either express or implied.\n# see the license for the specific language governing permissions and\n# limitations under the license.\nfrom .base import BasePartitioner\nfrom .transnetv2_partitioner import TransNetV2Partitioner\n__all__ = ['BasePartitioner', 'TransNetV2Partitioner']"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8d7f9b6a-41f7-4abc-a197-22246acbf883.json b/docs/doc/8d7f9b6a-41f7-4abc-a197-22246acbf883.json
new file mode 100644
index 000000000..dd8845b52
--- /dev/null
+++ b/docs/doc/8d7f9b6a-41f7-4abc-a197-22246acbf883.json
@@ -0,0 +1,50 @@
+{
+ "summary": "The PaddleVideo library evaluates video classification models using GAP, hit@one, precision error, and loss metrics. The `EvaluationMetrics` class accumulates these metrics per mini-batch or epoch using AveragePrecisionCalculator for GAP calculation.",
+ "details": [
+ {
+ "comment": "Code snippet is a part of the PaddleVideo library, providing functions to help evaluate video classification models. It includes flattening list functionality and calculates hit at one for predictions and actuals using numpy operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":0-27",
+ "content": "# Copyright 2016 Google Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS-IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Provides functions to help with evaluating models.\"\"\"\nimport datetime\nimport numpy\nfrom . import mean_average_precision_calculator as map_calculator\nfrom . import average_precision_calculator as ap_calculator\ndef flatten(l):\n \"\"\" Merges a list of lists into a single list. \"\"\"\n return [item for sublist in l for item in sublist]\ndef calculate_hit_at_one(predictions, actuals):\n \"\"\"Performs a local (numpy) calculation of the hit at one."
+ },
+ {
+ "comment": "This code calculates the average hit at one and precision at equal recall rate for a batch of predictions and corresponding actuals. These are metrics commonly used in evaluation of machine learning models, particularly in video classification tasks. The functions take as input two matrices: 'predictions' containing model outputs and 'actuals' with ground truth labels. They return the average hit at one and precision at equal recall rate across the entire batch respectively.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":29-58",
+ "content": " Args:\n predictions: Matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n actuals: Matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n Returns:\n float: The average hit at one across the entire batch.\n \"\"\"\n top_prediction = numpy.argmax(predictions, 1)\n hits = actuals[numpy.arange(actuals.shape[0]), top_prediction]\n return numpy.average(hits)\ndef calculate_precision_at_equal_recall_rate(predictions, actuals):\n \"\"\"Performs a local (numpy) calculation of the PERR.\n Args:\n predictions: Matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n actuals: Matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n Returns:\n float: The average precision at equal recall rate across the entire batch.\n \"\"\"\n aggregated_precision = 0.0\n num_videos = actuals.shape[0]\n for row in numpy.arange(num_videos):\n num_labels = int(numpy.sum(actuals[row]))"
+ },
+ {
+ "comment": "The code calculates the global average precision (GAP) using the top_k predictions and actuals for each video. It uses a function called AveragePrecisionCalculator to calculate the metric. The function first partitions the predictions based on their values, then iterates through the top indices, adding up the correct ones to calculate item precision. Finally, it averages the item precisions across all videos to get the GAP.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":59-86",
+ "content": " top_indices = numpy.argpartition(predictions[row],\n -num_labels)[-num_labels:]\n item_precision = 0.0\n for label_index in top_indices:\n if predictions[row][label_index] > 0:\n item_precision += actuals[row][label_index]\n item_precision /= top_indices.size\n aggregated_precision += item_precision\n aggregated_precision /= num_videos\n return aggregated_precision\ndef calculate_gap(predictions, actuals, top_k=20):\n \"\"\"Performs a local (numpy) calculation of the global average precision.\n Only the top_k predictions are taken for each of the videos.\n Args:\n predictions: Matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n actuals: Matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n top_k: How many predictions to use per video.\n Returns:\n float: The global average precision.\n \"\"\"\n gap_calculator = ap_calculator.AveragePrecisionCalculator()"
+ },
+ {
+ "comment": "This code extracts the top k predictions for each video, sorted by class. It returns a tuple containing the sparse_predictions, sparse_labels, and num_positives. The gap_calculator accumulates the flattened sparse_predictions, flattened sparse_labels, and sum of num_positives. Finally, it returns the average precision at n using peek_ap_at_n() from the gap_calculator.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":87-108",
+ "content": " sparse_predictions, sparse_labels, num_positives = top_k_by_class(\n predictions, actuals, top_k)\n gap_calculator.accumulate(flatten(sparse_predictions),\n flatten(sparse_labels), sum(num_positives))\n return gap_calculator.peek_ap_at_n()\ndef top_k_by_class(predictions, labels, k=20):\n \"\"\"Extracts the top k predictions for each video, sorted by class.\n Args:\n predictions: A numpy matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n k: the top k non-zero entries to preserve in each prediction.\n Returns:\n A tuple (predictions,labels, true_positives). 'predictions' and 'labels'\n are lists of lists of floats. 'true_positives' is a list of scalars. The\n length of the lists are equal to the number of classes. The entries in the\n predictions variable are probability predictions, and\n the corresponding entries in the labels variable are the ground truth for\n those predictions. The entries in 'true_positives' are the number of true"
+ },
+ {
+ "comment": "Function evaluates predictions and labels for each video and calculates top-k triplets (prediction, class) for each class. If k is not a positive integer, it raises ValueError. It returns out_predictions, out_labels, and out_true_positives for further analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":109-134",
+ "content": " positives for each class in the ground truth.\n Raises:\n ValueError: An error occurred when the k is not a positive integer.\n \"\"\"\n if k <= 0:\n raise ValueError(\"k must be a positive integer.\")\n k = min(k, predictions.shape[1])\n num_classes = predictions.shape[1]\n prediction_triplets = []\n for video_index in range(predictions.shape[0]):\n prediction_triplets.extend(\n top_k_triplets(predictions[video_index], labels[video_index], k))\n out_predictions = [[] for v in range(num_classes)]\n out_labels = [[] for v in range(num_classes)]\n for triplet in prediction_triplets:\n out_predictions[triplet[0]].append(triplet[1])\n out_labels[triplet[0]].append(triplet[2])\n out_true_positives = [numpy.sum(labels[:, i]) for i in range(num_classes)]\n return out_predictions, out_labels, out_true_positives\ndef top_k_triplets(predictions, labels, k=20):\n \"\"\"Get the top_k for a 1-d numpy array. Returns a sparse list of tuples in\n (prediction, class) format\"\"\""
+ },
+ {
+ "comment": "The code defines a class `EvaluationMetrics` to store various evaluation metrics for video classification. The `__init__` method initializes the metrics such as hit@one, precision error (perr), and loss. It also initializes two calculators: MeanAveragePrecisionCalculator and AveragePrecisionCalculator. The `accumulate` method updates these metrics based on predictions, labels, and loss values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":135-163",
+ "content": " m = len(predictions)\n k = min(k, m)\n indices = numpy.argpartition(predictions, -k)[-k:]\n return [(index, predictions[index], labels[index]) for index in indices]\nclass EvaluationMetrics(object):\n \"\"\"A class to store the evaluation metrics.\"\"\"\n def __init__(self, num_class, top_k):\n \"\"\"Construct an EvaluationMetrics object to store the evaluation metrics.\n Args:\n num_class: A positive integer specifying the number of classes.\n top_k: A positive integer specifying how many predictions are considered per video.\n Raises:\n ValueError: An error occurred when MeanAveragePrecisionCalculator cannot\n not be constructed.\n \"\"\"\n self.sum_hit_at_one = 0.0\n self.sum_perr = 0.0\n self.sum_loss = 0.0\n self.map_calculator = map_calculator.MeanAveragePrecisionCalculator(\n num_class)\n self.global_ap_calculator = ap_calculator.AveragePrecisionCalculator()\n self.top_k = top_k\n self.num_examples = 0\n #def accumulate(self, predictions, labels, loss):"
+ },
+ {
+ "comment": "The code defines a function \"accumulate\" that takes in predictions, labels and loss from a mini-batch. It calculates three metrics: mean_hit_at_one, mean_perr, and mean_loss. The function then returns a dictionary containing these metrics for the batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":164-189",
+ "content": " def accumulate(self, loss, predictions, labels):\n \"\"\"Accumulate the metrics calculated locally for this mini-batch.\n Args:\n predictions: A numpy matrix containing the outputs of the model.\n Dimensions are 'batch' x 'num_classes'.\n labels: A numpy matrix containing the ground truth labels.\n Dimensions are 'batch' x 'num_classes'.\n loss: A numpy array containing the loss for each sample.\n Returns:\n dictionary: A dictionary storing the metrics for the mini-batch.\n Raises:\n ValueError: An error occurred when the shape of predictions and actuals\n does not match.\n \"\"\"\n batch_size = labels.shape[0]\n mean_hit_at_one = calculate_hit_at_one(predictions, labels)\n mean_perr = calculate_precision_at_equal_recall_rate(\n predictions, labels)\n mean_loss = numpy.mean(loss)\n # Take the top 20 predictions.\n sparse_predictions, sparse_labels, num_positives = top_k_by_class(\n predictions, labels, self.top_k)"
+ },
+ {
+ "comment": "This code calculates and accumulates various evaluation metrics during the epoch, including hit_at_one, perr, and loss. It then returns a dictionary with these metrics after an entire epoch of training. If no examples were accumulated during the epoch, it raises a ValueError.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":190-218",
+ "content": " self.map_calculator.accumulate(sparse_predictions, sparse_labels,\n num_positives)\n self.global_ap_calculator.accumulate(flatten(sparse_predictions),\n flatten(sparse_labels),\n sum(num_positives))\n self.num_examples += batch_size\n self.sum_hit_at_one += mean_hit_at_one * batch_size\n self.sum_perr += mean_perr * batch_size\n self.sum_loss += mean_loss * batch_size\n return {\n \"hit_at_one\": mean_hit_at_one,\n \"perr\": mean_perr,\n \"loss\": mean_loss\n }\n def get(self):\n \"\"\"Calculate the evaluation metrics for the whole epoch.\n Raises:\n ValueError: If no examples were accumulated.\n Returns:\n dictionary: a dictionary storing the evaluation metrics for the epoch. The\n dictionary has the fields: avg_hit_at_one, avg_perr, avg_loss, and\n aps (default nan).\n \"\"\"\n if self.num_examples <= 0:"
+ },
+ {
+ "comment": "This code defines a class for evaluating metrics in video tagging. It calculates average hit at one, perr, and loss, as well as maps and global APs. The clear method resets the metrics to zero.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/metrics/youtube8m/eval_util.py\":219-243",
+ "content": " raise ValueError(\"total_sample must be positive.\")\n avg_hit_at_one = self.sum_hit_at_one / self.num_examples\n avg_perr = self.sum_perr / self.num_examples\n avg_loss = self.sum_loss / self.num_examples\n aps = self.map_calculator.peek_map_at_n()\n gap = self.global_ap_calculator.peek_ap_at_n()\n epoch_info_dict = {}\n return {\n \"avg_hit_at_one\": avg_hit_at_one,\n \"avg_perr\": avg_perr,\n \"avg_loss\": avg_loss,\n \"aps\": aps,\n \"gap\": gap\n }\n def clear(self):\n \"\"\"Clear the evaluation metrics and reset the EvaluationMetrics object.\"\"\"\n self.sum_hit_at_one = 0.0\n self.sum_perr = 0.0\n self.sum_loss = 0.0\n self.map_calculator.clear()\n self.global_ap_calculator.clear()\n self.num_examples = 0"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8d8ad343-569e-45a7-a33f-92251d72be90.json b/docs/doc/8d8ad343-569e-45a7-a33f-92251d72be90.json
new file mode 100644
index 000000000..036e17203
--- /dev/null
+++ b/docs/doc/8d8ad343-569e-45a7-a33f-92251d72be90.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code introduces the ActBert model for multimodal tasks, including training and validation steps. It utilizes a backbone function for predictions with text, video, and action scores along with sequence relationship scores. The infer_step is yet to be implemented.",
+ "details": [
+ {
+ "comment": "This code snippet defines the ActBert class, which is a multimodal model framework. It registers the model under MULTIMODAL in the registry and includes a forward_net method for processing text, action, and image data. The self.backbone function is used to make predictions based on this data. The code also includes import statements, variable definitions, and a get_logger function call for logging purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/multimodal/actbert.py\":0-26",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom ...registry import MULTIMODAL\nfrom .base import BaseMultimodal\nimport paddle\nfrom paddlevideo.utils import get_logger\nlogger = get_logger(\"paddlevideo\")\n@MULTIMODAL.register()\nclass ActBert(BaseMultimodal):\n \"\"\"ActBert model framework.\"\"\"\n def forward_net(self, text_ids, action_feat, image_feat, image_loc,\n token_type_ids, text_mask, image_mask, action_mask):\n pred = self.backbone(text_ids, action_feat, image_feat, image_loc,\n token_type_ids, text_mask, image_mask, action_mask)"
+ },
+ {
+ "comment": "This code defines a train_step and val_step for ActBert Dataset. In the train_step, it takes input data (text_ids, action_feat, image_feat, etc.), passes them through the backbone model to get prediction scores, calculates loss, and returns a loss metric dictionary. The val_step does not appear to have any additional functionality.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/multimodal/actbert.py\":27-45",
+ "content": " return pred\n def train_step(self, data_batch):\n \"\"\"For ActBert Dataset. Define how the model is going to train, from input to output.\n \"\"\"\n text_ids, action_feat, image_feat, image_loc, \\\n token_type_ids, text_mask, image_mask, action_mask, \\\n text_labels, action_label, next_sentence_label, image_label, image_target = data_batch\n loss_metrics = dict()\n pred = self.backbone(text_ids, action_feat, image_feat, image_loc,\n token_type_ids, text_mask, image_mask, action_mask)\n prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score = pred\n total_loss = self.loss(prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score, \\\n text_labels, image_label, image_target, action_label, next_sentence_label)\n loss_metrics['loss'] = paddle.mean(total_loss)\n return loss_metrics\n def val_step(self, data_batch):\n \"\"\"For ActBert Dataset. Define how the model is going to val, from input to output."
+ },
+ {
+ "comment": "The code defines a model that takes in multiple inputs like text_ids, action_feat, image_feat, and more. It performs testing with the test_step() function and returns prediction scores for text, video, and action, along with the sequence relationship score. The infer_step() function is not implemented yet.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/framework/multimodal/actbert.py\":46-63",
+ "content": " \"\"\"\n return self.train_step(data_batch)\n def test_step(self, data_batch):\n \"\"\"For MSR-VTT Dataset. Define how the model is going to test, from input to output.\"\"\"\n text_ids, action_feat, image_feat, image_loc, token_type_ids, text_mask, image_mask, action_mask = data_batch[:\n -1]\n action_feat = action_feat.squeeze(0)\n image_feat = image_feat.squeeze(0)\n image_loc = image_loc.squeeze(0)\n image_mask = image_mask.squeeze(0)\n action_mask = action_mask.squeeze(0)\n prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score = self.forward_net(text_ids, \\\n action_feat, image_feat, image_loc, token_type_ids, text_mask, image_mask, action_mask)\n return prediction_scores_t, prediction_scores_v, prediction_scores_a, seq_relationship_score\n def infer_step(self, data_batch):\n pass"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8da98a26-70c5-4704-ac2b-a021a899805a.json b/docs/doc/8da98a26-70c5-4704-ac2b-a021a899805a.json
new file mode 100644
index 000000000..3d35c9d07
--- /dev/null
+++ b/docs/doc/8da98a26-70c5-4704-ac2b-a021a899805a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is importing modules and creating registries for pipelines and datasets in the PaddleVideo application. The registries allow easy management of different pipeline and dataset types, making it convenient to extend or customize them later on.",
+ "details": [
+ {
+ "comment": "This code is importing modules and creating registries for pipelines and datasets in the PaddleVideo application. The registries allow easy management of different pipeline and dataset types, making it convenient to extend or customize them later on.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/registry.py\":0-19",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom ..utils import Registry\nPIPELINES = Registry(\"pipeline\")\nDATASETS = Registry(\"datasets\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8dd65d2e-d515-4060-a2e7-336f403298db.json b/docs/doc/8dd65d2e-d515-4060-a2e7-336f403298db.json
new file mode 100644
index 000000000..fb10ead62
--- /dev/null
+++ b/docs/doc/8dd65d2e-d515-4060-a2e7-336f403298db.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This guide details fine-tuning the VideoTag model using custom data, covering AttentionLSTM and TSN models, feature extraction, and multi/single GPU support. The code trains, evaluates, predicts with TSN, requires specific weight files, allows save directories, and preprocesses videos into images.",
+ "details": [
+ {
+ "comment": "This is a guide for fine-tuning the VideoTag pre-trained model with custom training data, covering AttentionLSTM and TSN models, principle explanations, and reference papers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/FineTune.md\":0-31",
+ "content": "# \u6a21\u578b\u5fae\u8c03\u6307\u5357\n---\n## \u5185\u5bb9\n\u53c2\u8003\u672c\u6587\u6863\uff0c\u60a8\u53ef\u4ee5\u4f7f\u7528\u81ea\u5df1\u7684\u8bad\u7ec3\u6570\u636e\u5728VideoTag\u9884\u8bad\u7ec3\u6a21\u578b\u4e0a\u8fdb\u884cfine-tune\uff0c\u8bad\u7ec3\u51fa\u81ea\u5df1\u7684\u6a21\u578b\u3002\n\u6587\u6863\u5185\u5bb9\u5305\u62ec:\n- [\u539f\u7406\u89e3\u6790](#\u539f\u7406\u89e3\u6790)\n- [\u5bf9AttentionLSTM\u6a21\u578b\u8fdb\u884c\u5fae\u8c03](#\u5bf9AttentionLSTM\u6a21\u578b\u8fdb\u884c\u5fae\u8c03)\n- [\u5bf9TSN\u6a21\u578b\u8fdb\u884c\u5fae\u8c03](#\u5bf9TSN\u6a21\u578b\u8fdb\u884c\u5fae\u8c03)\n- [\u6269\u5c55\u5185\u5bb9](#\u6269\u5c55\u5185\u5bb9)\n- [\u53c2\u8003\u8bba\u6587](#\u53c2\u8003\u8bba\u6587)\n## \u539f\u7406\u89e3\u6790\nVideoTag\u91c7\u7528\u4e24\u9636\u6bb5\u5efa\u6a21\u65b9\u5f0f\uff0c\u7531\u4e24\u4e2a\u6a21\u578b\u7ec4\u6210: TSN + AttentionLSTM\u3002\nTemporal Segment Network (TSN) \u662f\u7ecf\u5178\u7684\u57fa\u4e8e2D-CNN\u7684\u89c6\u9891\u5206\u7c7b\u6a21\u578b\u3002\u8be5\u6a21\u578b\u901a\u8fc7\u7a00\u758f\u91c7\u6837\u89c6\u9891\u5e27\u7684\u65b9\u5f0f\uff0c\u5728\u6355\u83b7\u89c6\u9891\u65f6\u5e8f\u4fe1\u606f\u7684\u540c\u65f6\u964d\u4f4e\u4e86\u8ba1\u7b97\u91cf\u3002\u8be6\u7ec6\u5185\u5bb9\u8bf7\u53c2\u8003\u8bba\u6587[Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://arxiv.org/abs/1608.00859)\nAttentionLSTM\u4ee5\u89c6\u9891\u7684\u7279\u5f81\u5411\u91cf\u4f5c\u4e3a\u8f93\u5165\uff0c\u91c7\u7528\u53cc\u5411\u957f\u77ed\u65f6\u8bb0\u5fc6\u7f51\u7edc\uff08LSTM\uff09\u5bf9\u6240\u6709\u5e27\u7279\u5f81\u8fdb\u884c\u7f16\u7801\uff0c\u5e76\u589e\u52a0Attention\u5c42\uff0c\u5c06\u6bcf\u4e2a\u65f6\u523b\u7684\u9690\u72b6\u6001\u8f93\u51fa\u4e0e\u81ea\u9002\u5e94\u6743\u91cd\u7ebf\u6027\u52a0\u6743\u5f97\u5230\u6700\u7ec8\u5206\u7c7b\u5411\u91cf\u3002\u8be6\u7ec6\u5185\u5bb9\u8bf7\u53c2\u8003\u8bba\u6587[AttentionCluster](https://arxiv.org/abs/1711.09550)\nVideoTag\u8bad\u7ec3\u65f6\u5206\u4e24\u4e2a\u9636\u6bb5: \u7b2c\u4e00\u9636\u6bb5\u4f7f\u7528\u5c11\u91cf\u89c6\u9891\u6837\u672c\uff08\u5341\u4e07\u7ea7\u522b\uff09\u8bad\u7ec3\u5927\u89c4\u6a21\u89c6\u9891\u7279\u5f81\u63d0\u53d6\u6a21\u578b(TSN)\uff1b\u7b2c\u4e8c\u9636\u6bb5\u4f7f\u7528\u5343\u4e07\u7ea7\u6570\u636e\u8bad\u7ec3\u9884\u6d4b\u5668(AttentionLSTM)\u3002\nVideoTag\u9884\u6d4b\u65f6\u4e5f\u5206\u4e24\u4e2a\u9636\u6bb5: \u7b2c\u4e00\u9636\u6bb5\u4ee5\u89c6\u9891\u6587\u4ef6\u4f5c\u4e3a\u8f93\u5165\uff0c\u7ecf\u8fc7\u53bb\u9664\u4e86\u5168\u8fde\u63a5\u5c42\u4ee5\u53ca\u635f\u5931\u51fd\u6570\u5c42\u7684TSN\u7f51\u7edc\u540e\u5f97\u5230\u8f93\u51fa\u7279\u5f81\u5411\u91cf\uff1b\u7b2c\u4e8c\u9636\u6bb5\u4ee5TSN\u7f51\u7edc\u8f93\u51fa\u7684\u7279\u5f81\u5411\u91cf\u4f5c\u4e3a\u8f93\u5165\uff0c\u7ecf\u8fc7AttentionLSTM\u540e\u5f97\u5230\u6700\u7ec8\u7684\u5206\u7c7b\u7ed3\u679c\u3002\n\u57fa\u4e8e\u6211\u4eec\u7684\u9884\u6a21\u578b\uff0c\u60a8\u53ef\u4ee5\u4f7f\u7528\u81ea\u5df1\u7684\u8bad\u7ec3\u6570\u636e\u8fdb\u884cfine-tune:\n- [\u5bf9AttentionLSTM\u6a21\u578b\u8fdb\u884c\u5fae\u8c03](#\u5bf9AttentionLSTM\u6a21\u578b\u8fdb\u884c\u5fae\u8c03)\n- [\u5bf9TSN\u6a21\u578b\u8fdb\u884c\u5fae\u8c03](#\u5bf9TSN\u6a21\u578b\u8fdb\u884c\u5fae\u8c03)\n## \u5bf9AttentionLSTM\u6a21\u578b\u8fdb\u884c\u5fae\u8c03\nAttentionLSTM\u4ee5\u89c6\u9891\u7279\u5f81\u4f5c\u4e3a\u8f93\u5165\uff0c\u663e\u5b58\u5360\u7528\u5c11\uff0c\u8bad\u7ec3\u901f\u5ea6\u8f83TSN\u66f4\u5feb\uff0c\u56e0\u6b64\u63a8\u8350\u4f18\u5148\u5bf9AttentionLSTM\u6a21\u578b\u8fdb\u884c\u5fae\u8c03\u3002\u8f93\u5165\u89c6\u9891\u9996\u5148\u7ecf\u8fc7TSN\u9884\u8bad\u7ec3\u6a21\u578b\u63d0\u53d6\u7279\u5f81\u5411\u91cf\uff0c\u7136\u540e\u5c06\u7279\u5f81\u5411\u91cf\u4f5c\u4e3a\u8bad\u7ec3\u8f93\u5165\u6570\u636e\uff0c\u5fae\u8c03AttentionLSTM\u6a21\u578b\u3002"
+ },
+ {
+ "comment": "Extract features from TSN pre-trained model, save the extracted features in specified directory. AttentionLSTM model fine-tuning requires TSN extracted features with corresponding labels in the train.list file. Label indices are defined in a separate text file, e.g., label_3396.txt.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/FineTune.md\":33-80",
+ "content": "### TSN\u9884\u6a21\u578b\u63d0\u53d6\u7279\u5f81\u5411\u91cf\n#### \u6570\u636e\u51c6\u5907\n- \u9884\u8bad\u7ec3\u6743\u91cd\u4e0b\u8f7d: \u53c2\u8003[\u6837\u4f8b\u4ee3\u7801\u8fd0\u884c\u6307\u5357-\u6570\u636e\u51c6\u5907-\u9884\u8bad\u7ec3\u6743\u91cd\u4e0b\u8f7d](./Run.md)\n- \u51c6\u5907\u8bad\u7ec3\u6570\u636e: \u51c6\u5907\u597d\u5f85\u8bad\u7ec3\u7684\u89c6\u9891\u6570\u636e\uff0c\u5e76\u5728video\\_tag/data/TsnExtractor.list\u6587\u4ef6\u4e2d\u6307\u5b9a\u5f85\u8bad\u7ec3\u7684\u6587\u4ef6\u8def\u5f84\uff0c\u5185\u5bb9\u683c\u5f0f\u5982\u4e0b:\n```\nmy_video_path/my_video_file1.mp4\nmy_video_path/my_video_file2.mp4\n...\n```\n#### \u7279\u5f81\u63d0\u53d6\n\u7279\u5f81\u63d0\u53d6\u811a\u672c\u5982\u4e0b:\n```\npython tsn_extractor.py --model_name=TSN --config=./configs/tsn.yaml --weights=./weights/tsn.pdparams\n```\n- \u901a\u8fc7--weights\u53ef\u6307\u5b9aTSN\u6743\u91cd\u53c2\u6570\u7684\u5b58\u50a8\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3avideo\\_tag/weights/tsn.pdparams\n- \u901a\u8fc7--save\\_dir\u53ef\u6307\u5b9a\u7279\u5f81\u5411\u91cf\u4fdd\u5b58\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3avideo\\_tag/data/tsn\\_features\uff0c\u4e0d\u540c\u8f93\u5165\u89c6\u9891\u7684\u7279\u5f81\u5411\u91cf\u63d0\u53d6\u7ed3\u679c\u5206\u6587\u4ef6\u4fdd\u5b58\u5728\u4e0d\u540c\u7684npy\u6587\u4ef6\u4e2d\uff0c\u76ee\u5f55\u5f62\u5f0f\u4e3a:\n```\nvideo_tag\n \u251c\u2500\u2500data\n \u251c\u2500\u2500tsn_features\n \u251c\u2500\u2500 my_feature_file1.npy\n \u251c\u2500\u2500 my_feature_file2.npy\n ...\n```\n- tsn\u63d0\u53d6\u7684\u7279\u5f81\u5411\u91cf\u7ef4\u5ea6\u4e3a```\u5e27\u6570*\u7279\u5f81\u7ef4\u5ea6```\uff0c\u9ed8\u8ba4\u4e3a300 * 2048\u3002\n### AttentionLSTM\u6a21\u578bFine-tune\n#### \u6570\u636e\u51c6\u5907\nVideoTag\u4e2d\u7684AttentionLSTM\u4ee5TSN\u6a21\u578b\u63d0\u53d6\u7684\u7279\u5f81\u5411\u91cf\u4f5c\u4e3a\u8f93\u5165\u3002\u5728video\\_tag/data/dataset/attention\\_lstm/train.list\u6587\u4ef6\u4e2d\u6307\u5b9a\u5f85\u8bad\u7ec3\u7684\u6587\u4ef6\u8def\u5f84\u548c\u5bf9\u5e94\u7684\u6807\u7b7e\uff0c\u5185\u5bb9\u683c\u5f0f\u5982\u4e0b:\n```\nmy_feature_path/my_feature_file1.npy label1 label2\nmy_feature_path/my_feature_file2.npy label1\n...\n```\n- \u4e00\u4e2a\u8f93\u5165\u89c6\u9891\u53ef\u4ee5\u6709\u591a\u4e2a\u6807\u7b7e\uff0c\u6807\u7b7e\u7d22\u5f15\u4e3a\u6574\u578b\u6570\u636e\uff0c\u6587\u4ef6\u540d\u4e0e\u6807\u7b7e\u4e4b\u95f4\u3001\u591a\u4e2a\u6807\u7b7e\u4e4b\u95f4\u4ee5\u4e00\u4e2a\u7a7a\u683c\u5206\u9694\uff1b\n- \u6807\u7b7e\u7d22\u5f15\u4e0e\u6807\u7b7e\u540d\u79f0\u7684\u4e4b\u95f4\u7684\u5bf9\u5e94\u5173\u7cfb\u4ee5list\u6587\u4ef6\u6307\u5b9a\uff0c\u53ef\u53c2\u8003VideoTag\u7528\u5230\u7684label_3396.txt\u6587\u4ef6\u6784\u9020\uff0c\u884c\u7d22\u5f15\u5bf9\u5e94\u6807\u7b7e\u7d22\u5f15;"
+ },
+ {
+ "comment": "This code chunk is for fine-tuning the AttentionLSTM model in PaddleVideo's VideoTag application. It provides instructions for training the model with multiple GPUs or a single GPU, and specifies the configuration file and pretrained weights required. The code also demonstrates how to evaluate the trained model using eval.py script. The precision metrics printed include GAP and Hit@1.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/FineTune.md\":82-112",
+ "content": "- \u9a8c\u8bc1\u96c6\u3001\u6d4b\u8bd5\u96c6\u4ee5\u53ca\u9884\u6d4b\u6570\u636e\u96c6\u7684\u6784\u9020\u65b9\u5f0f\u540c\u8bad\u7ec3\u96c6\u7c7b\u4f3c\uff0c\u4ec5\u9700\u8981\u5728video\\_tag/data/attention\\_lstm/\u76ee\u5f55\u4e0b\u5bf9\u5e94\u7684list\u6587\u4ef6\u4e2d\u6307\u5b9a\u76f8\u5173\u6587\u4ef6\u8def\u5f84/\u6807\u7b7e\u5373\u53ef\u3002\n#### \u6a21\u578b\u8bad\u7ec3\n\u4f7f\u7528VideoTag\u4e2d\u7684AttentionLSTM\u9884\u6a21\u578b\u8fdb\u884cfine-tune\u8bad\u7ec3\u811a\u672c\u5982\u4e0b:\n```\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\npython train.py --model_name=AttentionLSTM --config=./configs/attention_lstm.yaml --pretrain=./weights/attention_lstm\n```\n- AttentionLSTM\u6a21\u578b\u9ed8\u8ba4\u4f7f\u75288\u5361\u8bad\u7ec3\uff0c\u603b\u7684batch size\u6570\u662f1024\u3002\u82e5\u4f7f\u7528\u5355\u5361\u8bad\u7ec3\uff0c\u8bf7\u4fee\u6539\u73af\u5883\u53d8\u91cf\uff0c\u811a\u672c\u5982\u4e0b:\n```\nexport CUDA_VISIBLE_DEVICES=0\npython train.py --model_name=AttentionLSTM --config=./configs/attention_lstm-single.yaml --pretrain=./weights/attention_lstm\n```\n- \u8bf7\u786e\u4fdd\u8bad\u7ec3\u6837\u672c\u6570\u5927\u4e8ebatch_size\u6570\n- \u901a\u8fc7--pretrain\u53c2\u6570\u53ef\u6307\u5b9aAttentionLSTM\u9884\u8bad\u7ec3\u6a21\u578b\u7684\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3a./weights/attention\\_lstm\uff1b\n- \u6a21\u578b\u76f8\u5173\u914d\u7f6e\u5199\u5728video_tag/configs/attention\\_lstm.yaml\u6587\u4ef6\u4e2d\uff0c\u53ef\u4ee5\u65b9\u4fbf\u7684\u8c03\u8282\u5404\u9879\u8d85\u53c2\u6570\uff1b\n- \u901a\u8fc7--save_dir\u53c2\u6570\u53ef\u6307\u5b9a\u8bad\u7ec3\u6a21\u578b\u53c2\u6570\u7684\u4fdd\u5b58\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3a./data/checkpoints\uff1b\n#### \u6a21\u578b\u8bc4\u4f30\n\u53ef\u7528\u5982\u4e0b\u65b9\u5f0f\u8fdb\u884c\u6a21\u578b\u8bc4\u4f30:\n```\npython eval.py --model_name=AttentionLSTM --config=./configs/attention_lstm.yaml --weights=./data/checkpoints/AttentionLSTM_epoch9.pdparams\n```\n- \u901a\u8fc7--weights\u53c2\u6570\u53ef\u6307\u5b9a\u8bc4\u4f30\u9700\u8981\u7684\u6743\u91cd\uff0c\u9ed8\u8ba4\u4e3a./data/checkpoints/AttentionLSTM_epoch9.pdparams\uff1b\n- \u8bc4\u4f30\u7ed3\u679c\u4ee5log\u7684\u5f62\u5f0f\u76f4\u63a5\u6253\u5370\u8f93\u51faGAP\u3001Hit@1\u7b49\u7cbe\u5ea6\u6307\u6807\u3002"
+ },
+ {
+ "comment": "This code provides instructions for model inference and fine-tuning using the PaddleVideo framework's VideoTag application. It explains how to specify the model, configuration file, weights, label files, and save directory for prediction results. Additionally, it outlines the steps for preparing data, training, and executing a pre-trained TSN model in the VideoTag application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/FineTune.md\":114-151",
+ "content": "#### \u6a21\u578b\u63a8\u65ad\n\u53ef\u7528\u5982\u4e0b\u65b9\u5f0f\u8fdb\u884c\u6a21\u578b\u63a8\u65ad:\n```\npython predict.py --model_name=AttentionLSTM --config=./configs/attention_lstm.yaml --weights=./data/checkpoints/AttentionLSTM_epoch9.pdparams\n```\n- \u901a\u8fc7--weights\u53c2\u6570\u53ef\u6307\u5b9a\u63a8\u65ad\u9700\u8981\u7684\u6743\u91cd\uff0c\u9ed8\u8ba4\u4e3a./data/checkpoints/AttentionLSTM_epoch9.pdparams\uff1b\n- \u901a\u8fc7--label_file\u53c2\u6570\u6307\u5b9a\u6807\u7b7e\u6587\u4ef6\uff0c\u8bf7\u6839\u636e\u81ea\u5df1\u7684\u6570\u636e\u4fee\u6539\uff0c\u9ed8\u8ba4\u4e3a./label_3396.txt;\n- \u9884\u6d4b\u7ed3\u679c\u4f1a\u4ee5\u65e5\u5fd7\u5f62\u5f0f\u6253\u5370\u51fa\u6765\uff0c\u540c\u65f6\u4e5f\u4fdd\u5b58\u5728json\u6587\u4ef6\u4e2d\uff0c\u901a\u8fc7--save_dir\u53c2\u6570\u53ef\u6307\u5b9a\u9884\u6d4b\u7ed3\u679c\u4fdd\u5b58\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3a./data/predict_results/attention_lstm\u3002\n## \u5bf9TSN\u6a21\u578b\u8fdb\u884c\u5fae\u8c03\nVideoTag\u4e2d\u4f7f\u7528\u7684TSN\u6a21\u578b\u4ee5mp4\u6587\u4ef6\u4e3a\u8f93\u5165\uff0cbackbone\u4e3aResNet101\u3002\n### \u6570\u636e\u51c6\u5907\n\u51c6\u5907\u597d\u8bad\u7ec3\u89c6\u9891\u6587\u4ef6\u540e\uff0c\u5728video\\_tag/data/dataset/tsn/train.list\u6587\u4ef6\u4e2d\u6307\u5b9a\u5f85\u8bad\u7ec3\u7684\u6587\u4ef6\u8def\u5f84\u548c\u5bf9\u5e94\u7684\u6807\u7b7e\u5373\u53ef\uff0c\u5185\u5bb9\u683c\u5f0f\u5982\u4e0b:\n```\nmy_video_path/my_video_file1.mp4 label1\nmy_video_path/my_video_file2.mp4 label2\n...\n```\n- \u4e00\u4e2a\u8f93\u5165\u89c6\u9891\u53ea\u80fd\u6709\u4e00\u4e2a\u6807\u7b7e\uff0c\u6807\u7b7e\u7d22\u5f15\u4e3a\u6574\u578b\u6570\u636e\uff0c\u6807\u7b7e\u7d22\u5f15\u4e0e\u6587\u4ef6\u540d\u4e4b\u95f4\u4ee5\u4e00\u4e2a\u7a7a\u683c\u5206\u9694\uff1b\n- \u9a8c\u8bc1\u96c6\u3001\u6d4b\u8bd5\u96c6\u4ee5\u53ca\u9884\u6d4b\u6570\u636e\u96c6\u7684\u6784\u9020\u65b9\u5f0f\u540c\u8bad\u7ec3\u96c6\u7c7b\u4f3c\uff0c\u4ec5\u9700\u8981\u5728video\\_tag/data/dataset/tsn\u76ee\u5f55\u4e0b\u5bf9\u5e94\u7684list\u6587\u4ef6\u4e2d\u6307\u5b9a\u76f8\u5173\u6587\u4ef6\u8def\u5f84/\u6807\u7b7e\u5373\u53ef\u3002\n#### \u6a21\u578b\u8bad\u7ec3\n\u4f7f\u7528VideoTag\u4e2d\u7684TSN\u9884\u6a21\u578b\u8fdb\u884cfine-tune\u8bad\u7ec3\u811a\u672c\u5982\u4e0b:\n```\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\npython train.py --model_name=TSN --config=./configs/tsn.yaml --pretrain=./weights/tsn\n```\n- TSN\u6a21\u578b\u9ed8\u8ba4\u4f7f\u75288\u5361\u8bad\u7ec3\uff0c\u603b\u7684batch size\u6570\u662f256\u3002\u82e5\u4f7f\u7528\u5355\u5361\u8bad\u7ec3\uff0c\u8bf7\u4fee\u6539\u73af\u5883\u53d8\u91cf\uff0c\u811a\u672c\u5982\u4e0b:\n```"
+ },
+ {
+ "comment": "This code is for training, evaluating and predicting with the TSN model. It uses different Python scripts (train.py, eval.py, and predict.py) along with a configuration file (tsn.yaml). The TSN model requires specific weight files saved at certain locations. It also has options to specify save directories for checkpoints, evaluation results, and prediction outputs. To speed up the training process, videos can be preprocessed into images before training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/FineTune.md\":152-187",
+ "content": "export CUDA_VISIBLE_DEVICES=0\npython train.py --model_name=TSN --config=./configs/tsn-single.yaml --pretrain=./weights/tsn\n```\n- \u901a\u8fc7--pretrain\u53c2\u6570\u53ef\u6307\u5b9aTSN\u9884\u8bad\u7ec3\u6a21\u578b\u7684\u8def\u5f84\uff0c\u793a\u4f8b\u4e3a./weights/tsn\uff1b\n- \u6a21\u578b\u76f8\u5173\u914d\u7f6e\u5199\u5728video_tag/configs/tsn.yaml\u6587\u4ef6\u4e2d\uff0c\u53ef\u4ee5\u65b9\u4fbf\u7684\u8c03\u8282\u5404\u9879\u8d85\u53c2\u6570\uff1b\n- \u901a\u8fc7--save_dir\u53c2\u6570\u53ef\u6307\u5b9a\u8bad\u7ec3\u6a21\u578b\u53c2\u6570\u7684\u4fdd\u5b58\u8def\u5f84\uff0c\u9ed8\u8ba4\u4e3a./data/checkpoints\uff1b\n#### \u6a21\u578b\u8bc4\u4f30\n\u53ef\u7528\u5982\u4e0b\u65b9\u5f0f\u8fdb\u884c\u6a21\u578b\u8bc4\u4f30:\n```\npython eval.py --model_name=TSN --config=./configs/tsn.yaml --weights=./data/checkpoints/TSN_epoch44.pdparams\n```\n- \u901a\u8fc7--weights\u53c2\u6570\u53ef\u6307\u5b9a\u8bc4\u4f30\u9700\u8981\u7684\u6743\u91cd\uff0c\u793a\u4f8b\u4e3a./data/checkpoints/TSN_epoch44.pdparams\uff1b\n- \u8bc4\u4f30\u7ed3\u679c\u4ee5log\u7684\u5f62\u5f0f\u76f4\u63a5\u6253\u5370\u8f93\u51faTOP1_ACC\u3001TOP5_ACC\u7b49\u7cbe\u5ea6\u6307\u6807\u3002\n#### \u6a21\u578b\u63a8\u65ad\n\u53ef\u7528\u5982\u4e0b\u65b9\u5f0f\u8fdb\u884c\u6a21\u578b\u63a8\u65ad:\n```\npython predict.py --model_name=TSN --config=./configs/tsn.yaml --weights=./data/checkpoints/TSN_epoch44.pdparams --save_dir=./data/predict_results/tsn/\n```\n- \u901a\u8fc7--weights\u53c2\u6570\u53ef\u6307\u5b9a\u63a8\u65ad\u9700\u8981\u7684\u6743\u91cd\uff0c\u793a\u4f8b\u4e3a./data/checkpoints/TSN_epoch44.pdparams\uff1b\n- \u901a\u8fc7--label_file\u53c2\u6570\u6307\u5b9a\u6807\u7b7e\u6587\u4ef6\uff0c\u8bf7\u6839\u636e\u81ea\u5df1\u7684\u6570\u636e\u4fee\u6539\uff0c\u9ed8\u8ba4\u4e3a./label_3396.txt;\n- \u9884\u6d4b\u7ed3\u679c\u4f1a\u4ee5\u65e5\u5fd7\u5f62\u5f0f\u6253\u5370\u51fa\u6765\uff0c\u540c\u65f6\u4e5f\u4fdd\u5b58\u5728json\u6587\u4ef6\u4e2d\uff0c\u901a\u8fc7--save_dir\u53c2\u6570\u53ef\u6307\u5b9a\u9884\u6d4b\u7ed3\u679c\u4fdd\u5b58\u8def\u5f84\uff0c\u793a\u4f8b\u4e3a./data/predict_results/tsn\u3002\n### \u8bad\u7ec3\u52a0\u901f\nTSN\u6a21\u578b\u9ed8\u8ba4\u4ee5mp4\u7684\u89c6\u9891\u6587\u4ef6\u4f5c\u4e3a\u8f93\u5165\uff0c\u8bad\u7ec3\u65f6\u9700\u8981\u5148\u5bf9\u89c6\u9891\u6587\u4ef6\u89e3\u7801\uff0c\u518d\u5c06\u89e3\u7801\u540e\u7684\u6570\u636e\u9001\u5165\u7f51\u7edc\u8fdb\u884c\u8bad\u7ec3\uff0c\u5982\u679c\u89c6\u9891\u6587\u4ef6\u5f88\u5927\uff0c\u8fd9\u4e2a\u8fc7\u7a0b\u5c06\u4f1a\u5f88\u8017\u65f6\u3002\n\u4e3a\u52a0\u901f\u8bad\u7ec3\uff0c\u53ef\u4ee5\u5148\u5c06\u89c6\u9891\u89e3\u7801\u6210\u56fe\u7247\uff0c\u7136\u540e\u4fdd\u5b58\u4e0b\u6765\uff0c\u8bad\u7ec3\u65f6\u76f4\u63a5\u6839\u636e\u7d22\u5f15\u8bfb\u53d6\u5e27\u56fe\u7247\u4f5c\u4e3a\u8f93\u5165\uff0c\u52a0\u5feb\u8bad\u7ec3\u8fc7\u7a0b\u3002"
+ },
+ {
+ "comment": "The code is preparing the data by decoding videos into frames and generating a file path list for these frames. It modifies the configuration file, changing the model format to \"frames\" and updating the filelist accordingly. Additional information about TSN and AttentionLSTM models can be found in their respective PaddleCV repositories, with references provided to the original papers as well.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/FineTune.md\":189-205",
+ "content": "- \u6570\u636e\u51c6\u5907: \u9996\u5148\u5c06\u89c6\u9891\u89e3\u7801\uff0c\u5b58\u6210\u5e27\u56fe\u7247\uff1b\u7136\u540e\u751f\u6210\u5e27\u56fe\u7247\u7684\u6587\u4ef6\u8def\u5f84\u5217\u8868\u3002\u5b9e\u73b0\u8fc7\u7a0b\u53ef\u53c2\u8003[ucf-101\u6570\u636e\u51c6\u5907](../../../../dygraph/tsn/data/dataset/ucf101/README.md)\n- \u4fee\u6539\u914d\u7f6e\u6587\u4ef6: \u4fee\u6539\u914d\u7f6e\u6587\u4ef6./config/tsn.yaml\uff0c\u5176\u4e2dMODEL.format\u503c\u6539\u4e3a\"frames\"\uff0c\u4e0d\u540c\u6a21\u5f0f\u4e0b\u7684filelist\u503c\u6539\u4e3a\u5bf9\u5e94\u7684\u5e27\u56fe\u7247\u6587\u4ef6list\u3002\n## \u6269\u5c55\u5185\u5bb9\n- \u66f4\u591a\u5173\u4e8eTSN\u6a21\u578b\u7684\u5185\u5bb9\u53ef\u53c2\u8003PaddleCV\u89c6\u9891\u5e93[TSN\u89c6\u9891\u5206\u7c7b\u6a21\u578b](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/video/models/tsn/README.md)\u3002\n- \u66f4\u591a\u5173\u4e8eAttentionLSTM\u6a21\u578b\u7684\u5185\u5bb9\u53ef\u53c2\u8003PaddleCV\u89c6\u9891\u5e93[AttentionLSTM\u89c6\u9891\u5206\u7c7b\u6a21\u578b](https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/video/models/attention_lstm)\u3002\n## \u53c2\u8003\u8bba\u6587\n- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://arxiv.org/abs/1608.00859), Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool\n- [Beyond Short Snippets: Deep Networks for Video Classification](https://arxiv.org/abs/1503.08909) Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8dddd838-dabc-4192-8aba-da185088ea18.json b/docs/doc/8dddd838-dabc-4192-8aba-da185088ea18.json
new file mode 100644
index 000000000..ce83f6090
--- /dev/null
+++ b/docs/doc/8dddd838-dabc-4192-8aba-da185088ea18.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The text outlines the process of preparing UCF24 dataset with PaddleVideo's build_split.py command, installing unrar tool and providing a download script for ease of access. The file structure contains video sequences and split files for training and testing.",
+ "details": [
+ {
+ "comment": "Introduction to UCF24 dataset preparation process, including download of RGB frame and annotation files. PaddleVideo provides a download script for easier access. Requires unrar tool installation. RGB frames stored in rgb-images directory, annotations in labels directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ucf24.md\":0-19",
+ "content": "English | [\u7b80\u4f53\u4e2d\u6587](../../zh-CN/dataset/ucf24.md)\n# UCF24 Data Preparation\nThis document mainly introduces the preparation process of UCF24 dataset. It mainly includes the download of the RGB frame files, the annotation files and the pathlist of the generated file.\n---\n## 1. Data Download\nDetailed information on UCF24 data can be found on the website [UCF24](http://www.thumos.info/download.html). For ease of use, PaddleVideo provides a download script for the RGB frame, annotation file of the UCF24 data.\nFirst, please ensure access to the [data/ucf24/ directory](../../../data/ucf24) and enter the following command for downloading the RGB frame, annotation file of the UCF24 dataset.\n```shell\nbash download_frames_annotations.sh\n```\n- To run this command you need to install the unrar decompression tool, which can be installed using the pip method.\n- The RGB frame files will be stored in the [data/ucf24/rgb-images/ directory](../../../data/ucf24/rgb-images)\n- The annotation files will be stored in the [data/ucf24/lables/ directory](../../../data/ucf24/labels)"
+ },
+ {
+ "comment": "This code describes the process of generating file path lists and the resulting folder structure for UCF24 dataset preparation using PaddleVideo's build_split.py command with the raw_path parameter, dividing data into groundtruths_ucf, labels, and rgb-images subfolders containing video clips and corresponding files.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ucf24.md\":21-59",
+ "content": "---\n## 2. File Pathlist Generation\nTo specify the format for dividing the file, enter the following command\n```python\npython build_split.py --raw_path ./splitfiles\n```\n**Description of parameters**\n`--raw_path`\uff1a indicates the storage path of the original division file\n# Folder Structure\nAfter the whole data pipeline for UCF24 preparation, the folder structure will look like:\n```\n\u251c\u2500\u2500 data\n\u2502 \u251c\u2500\u2500 ucf24\n\u2502 | \u251c\u2500\u2500 groundtruths_ucf\n\u2502 | \u251c\u2500\u2500 labels\n\u2502 | | \u251c\u2500\u2500 Basketball\n\u2502 | | | \u251c\u2500\u2500 v_Basketball_g01_c01\n\u2502 | | | | \u251c\u2500\u2500 00009.txt\n\u2502 | | | | \u251c\u2500\u2500 00010.txt\n\u2502 | | | | \u251c\u2500\u2500 ...\n\u2502 | | | | \u251c\u2500\u2500 00050.txt\n\u2502 | | | | \u251c\u2500\u2500 00051.txt\n\u2502 | | \u251c\u2500\u2500 ...\n\u2502 | | \u251c\u2500\u2500 WalkingWithDog\n\u2502 | | | \u251c\u2500\u2500 v_WalkingWithDog_g01_c01\n\u2502 | | | \u251c\u2500\u2500 ...\n\u2502 | | | \u251c\u2500\u2500 v_WalkingWithDog_g25_c04\n\u2502 | \u251c\u2500\u2500 rgb-images\n\u2502 | | \u251c\u2500\u2500 Basketball\n\u2502 | | | \u251c\u2500\u2500 v_Basketball_g01_c01\n\u2502 | | | | \u251c\u2500\u2500 00001.jpg\n\u2502 | | | | \u251c\u2500\u2500 00002.jpg\n\u2502 | | | | \u251c\u2500\u2500 ..."
+ },
+ {
+ "comment": "The code represents a file structure of the UCF101 dataset, containing various video sequences and split files for training and testing purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/dataset/ucf24.md\":60-72",
+ "content": "\u2502 | | | | \u251c\u2500\u2500 00140.jpg\n\u2502 | | | | \u251c\u2500\u2500 00141.jpg\n\u2502 | | \u251c\u2500\u2500 ...\n\u2502 | | \u251c\u2500\u2500 WalkingWithDog\n\u2502 | | | \u251c\u2500\u2500 v_WalkingWithDog_g01_c01\n\u2502 | | | \u251c\u2500\u2500 ...\n\u2502 | | | \u251c\u2500\u2500 v_WalkingWithDog_g25_c04\n\u2502 | \u251c\u2500\u2500 splitfiles\n\u2502 | | \u251c\u2500\u2500 trainlist01.txt\n\u2502 | | |\u2500\u2500 testlist01.txt \n\u2502 | \u251c\u2500\u2500 trainlist.txt\n\u2502 | |\u2500\u2500 testlist.txt \n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8e1104d6-0fb2-4891-9ca2-0f69f5577975.json b/docs/doc/8e1104d6-0fb2-4891-9ca2-0f69f5577975.json
new file mode 100644
index 000000000..7db32c500
--- /dev/null
+++ b/docs/doc/8e1104d6-0fb2-4891-9ca2-0f69f5577975.json
@@ -0,0 +1,10 @@
+{
+ "summary": "These comments provide author, date, and copyright information, indicating the source should be cited if the code is reprinted.",
+ "details": [
+ {
+ "comment": "These comments provide author, date, and copyright information, indicating the source should be cited if the code is reprinted.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/setup.py\":0-3",
+ "content": "# Author: Acer Zhang\n# Datetime: 2022/1/11\n# Copyright belongs to the author.\n# Please indicate the source for reprinting."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8ea88484-a9f1-4daa-8c3d-5b3291deee68.json b/docs/doc/8ea88484-a9f1-4daa-8c3d-5b3291deee68.json
new file mode 100644
index 000000000..522c9d49b
--- /dev/null
+++ b/docs/doc/8ea88484-a9f1-4daa-8c3d-5b3291deee68.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code defines a custom exception \"ReaderNotFoundError\" and manages reader instances using a singleton ReaderZoo object. The `regist_reader` function registers new readers, while the `get_reader` function retrieves and returns an instance of the requested reader based on name, mode, and configuration.",
+ "details": [
+ {
+ "comment": "The code defines a class \"ReaderNotFoundError\" which is an exception to handle situations when a reader is not found. It takes the name of the missing reader and a list of available readers as arguments, and provides a formatted error message with the missing reader's name and a list of available readers. This can be useful for raising custom errors in cases where the required reader cannot be found or is not compatible with the provided options.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/reader_utils.py\":0-29",
+ "content": "\"\"\"\nreader utils\n\"\"\"\n# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nclass ReaderNotFoundError(Exception):\n \"Error: reader not found\"\n def __init__(self, reader_name, avail_readers):\n super(ReaderNotFoundError, self).__init__()\n self.reader_name = reader_name\n self.avail_readers = avail_readers\n def __str__(self):\n msg = \"Reader {} Not Found.\\nAvailiable readers:\\n\".format(\n self.reader_name)\n for reader in self.avail_readers:"
+ },
+ {
+ "comment": "This code defines classes for data readers and a reader registry. The `DataReader` class is a base class for different video input data readers, while the `ReaderZoo` class manages a dictionary of registered readers. The code snippet includes methods to register readers and retrieve them by name, mode, and configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/reader_utils.py\":30-72",
+ "content": " msg += \" {}\\n\".format(reader)\n return msg\nclass DataReader(object):\n \"\"\"data reader for video input\"\"\"\n def __init__(self, model_name, mode, cfg):\n self.name = model_name\n self.mode = mode\n self.cfg = cfg\n def create_reader(self):\n \"\"\"Not implemented\"\"\"\n pass\n def get_config_from_sec(self, sec, item, default=None):\n \"\"\"get_config_from_sec\n \"\"\"\n if sec.upper() not in self.cfg:\n return default\n return self.cfg[sec.upper()].get(item, default)\nclass ReaderZoo(object):\n \"\"\"ReaderZoo\n \"\"\"\n def __init__(self):\n self.reader_zoo = {}\n def regist(self, name, reader):\n \"\"\"regist\n \"\"\"\n assert reader.__base__ == DataReader, \"Unknow model type {}\".format(\n type(reader))\n self.reader_zoo[name] = reader\n def get(self, name, mode, cfg):\n \"\"\"get\n \"\"\"\n for k, v in self.reader_zoo.items():\n if k == name:\n return v(name, mode, cfg)"
+ },
+ {
+ "comment": "This code snippet is responsible for managing reader instances, using a singleton ReaderZoo object. The `regist_reader` function allows registration of new readers, while the `get_reader` function retrieves and returns an instance of the requested reader based on the provided name, mode, and configuration.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/MultimodalVideoTag/scenario_lib/datareader/reader_utils.py\":73-90",
+ "content": " raise ReaderNotFoundError(name, self.reader_zoo.keys())\n# singleton reader_zoo\nreader_zoo = ReaderZoo()\ndef regist_reader(name, reader):\n \"\"\"regist_reader\n \"\"\"\n reader_zoo.regist(name, reader)\ndef get_reader(name, mode, cfg):\n \"\"\"get_reader\n \"\"\"\n reader_model = reader_zoo.get(name, mode, cfg)\n return reader_model.create_reader()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8ed1f1ea-7e25-4cda-be10-917e93f1c559.json b/docs/doc/8ed1f1ea-7e25-4cda-be10-917e93f1c559.json
new file mode 100644
index 000000000..569bea0de
--- /dev/null
+++ b/docs/doc/8ed1f1ea-7e25-4cda-be10-917e93f1c559.json
@@ -0,0 +1,40 @@
+{
+ "summary": "ADDS-DepthNet code estimates depth using day and night images, requiring scikit-image and matplotlib, utilizes Oxford RobotCar dataset, offers Resnet18_Imagenet pre-trained model addition, and provides training, testing instructions, and download URL. It demonstrates PaddlePaddle's predict.py tool for inference and saves results as pseudo-colored depth maps with two input images (RGB and depth estimation).",
+ "details": [
+ {
+ "comment": "This code is for the ADDS-DepthNet model, which is based on a self-supervised monocular depth estimation paper by Baidu Robotics and Autonomous Driving Laboratory. The code utilizes day and night images to reproduce the model and achieve advanced depth estimation results on the Oxford RobotCar dataset, mitigating the impact of lighting changes between day and night images. Additional dependencies like scikit-image and matplotlib are required before using the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":0-22",
+ "content": "[Simplified Chinese](../../../zh-CN/model_zoo/estimation/adds.md) | English\n# ADDS-DepthNet model\n## content\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Inference](#Inference)\n- [Reference](#Reference)\nBefore getting started, you need to install additional dependencies as follows:\n```bash\npython -m pip install scikit-image\npython -m pip install matplotlib\n```\n## Introduction\nThis model is based on the ICCV 2021 paper **[Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation](https://arxiv.org/abs/2108.07628)** of Baidu Robotics and Autonomous Driving Laboratory,\nThe self-supervised monocular depth estimation model based on day and night images is reproduced, which utilizes the complementary nature of day and night image data, and slows down the large domain shift of day and night images and the accuracy of depth estimation caused by lighting changes. Impact, the most advanced depth estimation results of all-sky images have been achieved on the challenging Oxford RobotCar data set."
+ },
+ {
+ "comment": "This code provides instructions for downloading and adding a pre-trained model to the Oxford RobotCar dataset. It mentions the necessary steps to download the pre-training model, Resnet18_Imagenet.pdparams, using the wget command and specifying its path in the adds.yaml file. The code also highlights the importance of filling in the correct fields in the configuration file for proper association with the relevant model types and frameworks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":25-48",
+ "content": "## Data\nFor data download and preparation of Oxford RobotCar dataset, please refer to [Oxford RobotCar dataset data preparation](../../dataset/Oxford_RobotCar.md)\n## Train\n### Oxford RobotCar dataset training\n#### Download and add pre-trained models\n1. Download the image pre-training model [resnet18.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/Resnet18_Imagenet.pdparams) as Backbone initialization parameters, or download through the wget command\n ```bash\n wget -P ./data https://videotag.bj.bcebos.com/PaddleVideo-release2.2/Resnet18_Imagenet.pdparams\n ```\n2. Open `PaddleVideo/configs/estimation/adds/adds.yaml`, and fill in the downloaded weight storage path below `pretrained:`\n ```yaml\n MODEL: #MODEL field\n framework: \"DepthEstimator\" #Mandatory, indicate the type of network, associate to the'paddlevideo/modeling/framework/'.\n backbone: #Mandatory, indicate the type of backbone, associate to the'paddlevideo/modeling/backbones/'.\n name: 'ADDS_DepthNet'"
+ },
+ {
+ "comment": "This code snippet provides instructions for training and testing the ADDS-DepthNet model using the Oxford RobotCar dataset. The provided commands initiate the training process with specific configuration file (`configs/estimation/adds/adds.yaml`) and seed value (20). Testing involves running separate commands to test day and night data sets, then recording their respective indicators. A download URL for a pre-trained model is also provided.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":49-71",
+ "content": " pretrained: fill in the path here\n ```\n#### Start training\n- The Oxford RobotCar dataset uses a single card for training, and the starting command for the training method is as follows:\n ```bash\n python3.7 main.py --validate -c configs/estimation/adds/adds.yaml --seed 20\n ```\n## Test\n- The ADDS-DepthNet model is verified synchronously during training (only the day or night data is verified). You can find the keyword `best` in the training log to obtain the model test accuracy. The log example is as follows:\n ```bash\n Already save the best model (rmse)8.5531\n ```\n- Because the model can only test one day or night data set at a given path in the yaml file at a time, to get the complete test score at the beginning of this document, you need to run 4 test commands and record their indicators ( 40m during the day, 60m during the day, 40m at night, 60m at night)\n- Download URL of the trained model: [ADDS_car.pdparams](https://videotag.bj.bcebos.com/PaddleVideo-release2.2/ADDS_car.pdparams)"
+ },
+ {
+ "comment": "The code provides test commands for running the ADDS model on the Oxford RobotCar dataset with varying maximum ground truth depth limits and different light conditions (night and daytime). It uses Python 3.7 to execute the main.py file from the PaddleVideo library, configs/estimation/adds/adds.yaml configuration, and specific dataset files for testing.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":73-89",
+ "content": "- The test commands are as follows:\n ```bash\n # Night 40m\n python3.7 main.py --test -c configs/estimation/adds/adds.yaml -w \"output/ADDS/ADDS_best.pdparams\" -o DATASET.test.file_path=\"data/oxford/splits/oxford_day/val_night_files.txt\" -o MODEL.head.max_gt_depth=40\n # Night 60m\n python3.7 main.py --test -c configs/estimation/adds/adds.yaml -w \"output/ADDS/ADDS_best.pdparams\" -o DATASET.test.file_path=\"data/oxford/splits/oxford_day/val_night_files.txt\" -o MODEL.head.max_gt_depth=60\n # Daytime 40m\n python3.7 main.py --test -c configs/estimation/adds/adds.yaml -w \"output/ADDS/ADDS_best.pdparams\" -o DATASET.test.file_path=\"data/oxford/splits/oxford_day/val_day_files.txt\" -o MODEL.head.max_gt_depth=40\n # Daytime 60m\n python3.7 main.py --test -c configs/estimation/adds/adds.yaml -w \"output/ADDS/ADDS_best.pdparams\" -o DATASET.test.file_path=\"data/oxford/splits/oxford_day/val_day_files.txt\" -o MODEL.head.max_gt_depth=60\n ```\n The test indicators on the validation dataset of Oxford RobotCar dataset are as follows:"
+ },
+ {
+ "comment": "The code presents a table comparing performance metrics of different models under various conditions. It shows the version, maximum depth, and several error measures for each model. The table also includes whether or not the delta value is less than 1.25 raised to different powers. The text describes how to run a command to export an inference model using Python script with specific configuration file, pre-trained parameters, and output directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":91-106",
+ "content": " | version | Max Depth | Abs Rel | Sq Rel | RMSE | RMSE log | ![](\"https://latex.codecogs.com/svg.image?\\delta&space;<&space;1.25&space;\" "\\"\\delta")
| ![](\"https://latex.codecogs.com/svg.image?\\delta&space;<&space;1.25^2\" "\\"\\delta")
| ![](\"https://latex.codecogs.com/svg.image?\\delta&space;<&space;1.25^3\" "\\"\\delta")
|\n | ----------- | --------- | ------- | ------ | ----- | ------- | ----------------- |------------------- | ------------------- |\n | ours(night) | 40 | 0.209 | 1.741 | 6.031 | 0.243 | 0.708 | 0.923 | 0.975 |\n | ours(night) | 60 | 0.207 | 2.052 | 7.888 | 0.258 | 0.686 | 0.909 | 0.970 |\n | ours(day) | 40 | 0.114 | 0.574 | 3.411 | 0.157 | 0.860 | 0.977 | 0.993 |\n | ours(day) | 60 | 0.119 | 0.793 | 4.842 | 0.173 | 0.838 | 0.967 | 0.991 |\n## Inference\n### Export inference model\n```bash\npython3.7 tools/export_model.py -c configs/estimation/adds/adds.yaml -p data/ADDS_car.pdparams -o inference/ADDS\n```\nThe above command will"
+ },
+ {
+ "comment": "This code snippet demonstrates the usage of PaddlePaddle's predict.py tool for model inference. It uses a pre-trained model, ADDS, to estimate depth maps from input images. The model files are specified using the --model_file and --params_files parameters, while the input image file is provided with --input_file. The command also includes options for GPU usage (--use_gpu) and TensorRT acceleration (--use_tensorrt). The inference results will be saved as pseudo-colored depth maps by default.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":106-123",
+ "content": " generate the model structure file `ADDS.pdmodel` and model weight files `ADDS.pdiparams` and `ADDS.pdiparams.info` files needed for prediction, all of which are stored in the `inference/ADDS/` directory\nFor the meaning of each parameter in the above bash command, please refer to [Model Inference Method](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0/docs/en/start.md#2-%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86)\n### Use predictive engine inference\n```bash\npython3.7 tools/predict.py --input_file data/example.png \\\n --config configs/estimation/adds/adds.yaml \\\n --model_file inference/ADDS/ADDS.pdmodel \\\n --params_file inference/ADDS/ADDS.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False\n```\nAt the end of the inference, the depth map estimated by the model will be saved in pseudo-color by default.\nThe following is a sample picture and the corresponding predicted depth map\uff1a"
+ },
+ {
+ "comment": "The code includes two images, one for regular RGB image and the other for depth estimation from the paper \"Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation\" by Liu et al.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/estimation/adds.md\":125-132",
+ "content": "![\"image\"](\"../../../images/oxford_image.png\")
\n![\"depth\"](\"../../../images/oxford_image_depth.png\")
\n## Reference\n- [Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation](https://arxiv.org/abs/2108.07628), Liu, Lina and Song, Xibin and Wang, Mengmeng and Liu, Yong and Zhang, Liangjun"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/8fb27118-de58-4d64-9c32-3cf7c53614bd.json b/docs/doc/8fb27118-de58-4d64-9c32-3cf7c53614bd.json
new file mode 100644
index 000000000..0648e8b95
--- /dev/null
+++ b/docs/doc/8fb27118-de58-4d64-9c32-3cf7c53614bd.json
@@ -0,0 +1,10 @@
+{
+ "summary": "The function `is_eval_epoch` determines whether the model should be evaluated at a given epoch based on the provided configs, current epoch, and multigrid training schedule. If the current epoch is equal to the total number of epochs or if there's a non-null multigrid schedule, it checks if the current epoch is a time for evaluation based on the schedule intervals. The function returns True when an evaluation should occur and False otherwise.",
+ "details": [
+ {
+ "comment": "The function `is_eval_epoch` determines whether the model should be evaluated at a given epoch based on the provided configs, current epoch, and multigrid training schedule. If the current epoch is equal to the total number of epochs or if there's a non-null multigrid schedule, it checks if the current epoch is a time for evaluation based on the schedule intervals. The function returns True when an evaluation should occur and False otherwise.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/interval_helper.py\":0-18",
+ "content": "def is_eval_epoch(cfg, cur_epoch, total_epochs, multigrid_schedule):\n \"\"\"\n Determine if the model should be evaluated at the current epoch.\n Args:\n cfg (CfgNode): configs. Details can be found in\n slowfast/config/defaults.py\n cur_epoch (int): current epoch.\n multigrid_schedule (List): schedule for multigrid training.\n \"\"\"\n if cur_epoch + 1 == total_epochs:\n return True\n if multigrid_schedule is not None:\n prev_epoch = 0\n for s in multigrid_schedule:\n if cur_epoch < s[-1]:\n period = max(\n (s[-1] - prev_epoch) // cfg.MULTIGRID.EVAL_FREQ + 1, 1)\n return (s[-1] - 1 - cur_epoch) % period == 0\n prev_epoch = s[-1]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9096484c-68d0-4d1c-bb82-be7b63d078f7.json b/docs/doc/9096484c-68d0-4d1c-bb82-be7b63d078f7.json
new file mode 100644
index 000000000..c6289f791
--- /dev/null
+++ b/docs/doc/9096484c-68d0-4d1c-bb82-be7b63d078f7.json
@@ -0,0 +1,75 @@
+{
+ "summary": "This code introduces SampleFrames class for PaddleVideo's loader pipelines using OpenCV, supports various modes and training, includes storage backend classes, converts images to numpy arrays, defines pipeline, and provides SampleAVAFrames class for sampling video frames.",
+ "details": [
+ {
+ "comment": "This code is importing necessary libraries and registering a class SampleFrames under PaddleVideo's loader pipelines. The class appears to sample frames from video, supporting different reading modes (color/grayscale/unchanged). It uses OpenCV (cv2) as the image processing backend.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":0-34",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\nfrom PIL import Image\nfrom ..registry import PIPELINES\nimport os\nimport numpy as np\nimport io\nimport os.path as osp\nfrom abc import ABCMeta, abstractmethod\nimport cv2\nfrom cv2 import IMREAD_COLOR, IMREAD_GRAYSCALE, IMREAD_UNCHANGED\nimport inspect\nimread_backend = 'cv2'\nimread_flags = {\n 'color': IMREAD_COLOR,\n 'grayscale': IMREAD_GRAYSCALE,\n 'unchanged': IMREAD_UNCHANGED\n}\n@PIPELINES.register()\nclass SampleFrames:"
+ },
+ {
+ "comment": "The function `__init__` initializes the parameters for sampling frames from a video, including clip length, frame interval, number of clips, temporal jittering options, and out-of-bound handling. The `_get_train_clips` function calculates the clip offsets in training mode by determining the average interval between clips based on the total number of frames. It then generates random base offsets and adds random offsets to create the final clip offsets.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":35-60",
+ "content": " \"\"\"Sample frames from the video. \"\"\"\n def __init__(self,\n clip_len,\n frame_interval=1,\n num_clips=1,\n temporal_jitter=False,\n twice_sample=False,\n out_of_bound_opt='loop',\n test_mode=False):\n self.clip_len = clip_len\n self.frame_interval = frame_interval\n self.num_clips = num_clips\n self.temporal_jitter = temporal_jitter\n self.twice_sample = twice_sample\n self.out_of_bound_opt = out_of_bound_opt\n self.test_mode = test_mode\n assert self.out_of_bound_opt in ['loop', 'repeat_last']\n def _get_train_clips(self, num_frames):\n \"\"\"Get clip offsets in train mode. \"\"\"\n ori_clip_len = self.clip_len * self.frame_interval\n avg_interval = (num_frames - ori_clip_len + 1) // self.num_clips\n if avg_interval > 0:\n base_offsets = np.arange(self.num_clips) * avg_interval\n clip_offsets = base_offsets + np.random.randint("
+ },
+ {
+ "comment": "This code calculates clip offsets for video sampling based on the number of frames and other parameters. It handles different scenarios, such as when the number of frames exceeds or equals the original clip length, when average interval is 0, and in test mode. The clip_offsets are returned at the end.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":61-81",
+ "content": " avg_interval, size=self.num_clips)\n elif num_frames > max(self.num_clips, ori_clip_len):\n clip_offsets = np.sort(\n np.random.randint(\n num_frames - ori_clip_len + 1, size=self.num_clips))\n elif avg_interval == 0:\n ratio = (num_frames - ori_clip_len + 1.0) / self.num_clips\n clip_offsets = np.around(np.arange(self.num_clips) * ratio)\n else:\n clip_offsets = np.zeros((self.num_clips, ), dtype=np.int)\n return clip_offsets\n def _get_test_clips(self, num_frames):\n \"\"\"Get clip offsets in test mode. \"\"\"\n ori_clip_len = self.clip_len * self.frame_interval\n avg_interval = (num_frames - ori_clip_len + 1) / float(self.num_clips)\n if num_frames > ori_clip_len - 1:\n base_offsets = np.arange(self.num_clips) * avg_interval\n clip_offsets = (base_offsets + avg_interval / 2.0).astype(np.int)\n if self.twice_sample:\n clip_offsets = np.concatenate([clip_offsets, base_offsets])"
+ },
+ {
+ "comment": "This code defines a class that samples video clips and loads frames based on different modes, such as testing or training. It takes the total number of frames in a video and returns the corresponding clip offsets and frame indices for loading. The sampling mode, temporal jitter, and out-of-bound options can be specified to customize the sampling process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":82-106",
+ "content": " else:\n clip_offsets = np.zeros((self.num_clips, ), dtype=np.int)\n return clip_offsets\n def _sample_clips(self, num_frames):\n \"\"\"Choose clip offsets for the video in a given mode. \"\"\"\n if self.test_mode:\n clip_offsets = self._get_test_clips(num_frames)\n else:\n clip_offsets = self._get_train_clips(num_frames)\n return clip_offsets\n def __call__(self, results):\n \"\"\"Perform the SampleFrames loading. \"\"\"\n total_frames = results['total_frames']\n clip_offsets = self._sample_clips(total_frames)\n frame_inds = clip_offsets[:, None] + np.arange(\n self.clip_len)[None, :] * self.frame_interval\n frame_inds = np.concatenate(frame_inds)\n if self.temporal_jitter:\n perframe_offsets = np.random.randint(\n self.frame_interval, size=len(frame_inds))\n frame_inds += perframe_offsets\n frame_inds = frame_inds.reshape((-1, self.clip_len))\n if self.out_of_bound_opt == 'loop':"
+ },
+ {
+ "comment": "Code handles out-of-bound frame indices by wrapping them around, repeating the last frame, or throwing an error. It then updates results with frame indices, clip length, and number of clips.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":107-128",
+ "content": " frame_inds = np.mod(frame_inds, total_frames)\n elif self.out_of_bound_opt == 'repeat_last':\n safe_inds = frame_inds < total_frames\n unsafe_inds = 1 - safe_inds\n last_ind = np.max(safe_inds * frame_inds, axis=1)\n new_inds = (safe_inds * frame_inds + (unsafe_inds.T * last_ind).T)\n frame_inds = new_inds\n else:\n raise ValueError('Illegal out_of_bound option.')\n start_index = results['start_index']\n frame_inds = np.concatenate(frame_inds) + start_index\n results['frame_inds'] = frame_inds.astype(np.int)\n results['clip_len'] = self.clip_len\n results['frame_interval'] = self.frame_interval\n results['num_clips'] = self.num_clips\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'clip_len={self.clip_len}, '\n f'frame_interval={self.frame_interval}, '\n f'num_clips={self.num_clips}, '"
+ },
+ {
+ "comment": "This code defines three classes: `BaseStorageBackend` (abstract class), `HardDiskBackend`, and a generic file client called `FileClient`. The `BaseStorageBackend` is an abstract class that provides two methods, `get()` and `get_text()`, which are expected to be implemented by subclasses. The `HardDiskBackend` implements these methods for handling files stored on the hard disk. Finally, the `FileClient` serves as a generic file client to access files in different backends.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":129-165",
+ "content": " f'temporal_jitter={self.temporal_jitter}, '\n f'twice_sample={self.twice_sample}, '\n f'out_of_bound_opt={self.out_of_bound_opt}, '\n f'test_mode={self.test_mode})')\n return repr_str\nclass BaseStorageBackend(metaclass=ABCMeta):\n \"\"\"Abstract class of storage backends. \"\"\"\n @abstractmethod\n def get(self, filepath):\n pass\n @abstractmethod\n def get_text(self, filepath):\n pass\nclass HardDiskBackend(BaseStorageBackend):\n \"\"\"Raw hard disks storage backend.\"\"\"\n def get(self, filepath):\n filepath = str(filepath)\n with open(filepath, 'rb') as f:\n value_buf = f.read()\n return value_buf\n def get_text(self, filepath):\n filepath = str(filepath)\n with open(filepath, 'r') as f:\n value_buf = f.read()\n return value_buf\nclass FileClient:\n \"\"\"A general file client to access files in different backend. \"\"\"\n _backends = {\n 'disk': HardDiskBackend,"
+ },
+ {
+ "comment": "This code defines a class with an initializer and a class method for registering backends. The initializer takes a backend argument, checks if it is supported, and initializes the client object with that backend. If the name or backend type is incorrect, it raises TypeError. The _register_backend method allows for backend registration by name, checking if it is a string and if the backend is a subclass of BaseStorageBackend. Raises KeyError if already registered.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":166-189",
+ "content": " }\n def __init__(self, backend='disk', **kwargs):\n if backend not in self._backends:\n raise ValueError(\n f'Backend {backend} is not supported. Currently supported ones'\n f' are {list(self._backends.keys())}')\n self.backend = backend\n self.client = self._backends[backend](**kwargs)\n @classmethod\n def _register_backend(cls, name, backend, force=False):\n if not isinstance(name, str):\n raise TypeError('the backend name should be a string, '\n f'but got {type(name)}')\n if not inspect.isclass(backend):\n raise TypeError(\n f'backend should be a class but got {type(backend)}')\n if not issubclass(backend, BaseStorageBackend):\n raise TypeError(\n f'backend {backend} is not a subclass of BaseStorageBackend')\n if not force and name in cls._backends:\n raise KeyError(\n f'{name} is already registered as a storage backend, '"
+ },
+ {
+ "comment": "This code defines a class called FileClient, which handles file operations like registration and retrieval. It also registers a pipeline named RawFrameDecode for loading and decoding frames using specified backends for I/O and decoding. The class has an _pillow2array method to convert PIL image to numpy array in specific channel order.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":190-224",
+ "content": " 'add \"force=True\" if you want to override it')\n cls._backends[name] = backend\n @classmethod\n def register_backend(cls, name, backend=None, force=False):\n \"\"\"Register a backend to FileClient. \"\"\"\n if backend is not None:\n cls._register_backend(name, backend, force=force)\n return\n def _register(backend_cls):\n cls._register_backend(name, backend_cls, force=force)\n return backend_cls\n return _register\n def get(self, filepath):\n return self.client.get(filepath)\n def get_text(self, filepath):\n return self.client.get_text(filepath)\n@PIPELINES.register()\nclass RawFrameDecode:\n \"\"\"Load and decode frames with given indices. \"\"\"\n def __init__(self, io_backend='disk', decoding_backend='cv2', **kwargs):\n self.io_backend = io_backend\n self.decoding_backend = decoding_backend\n self.kwargs = kwargs\n self.file_client = None\n def _pillow2array(self,img, flag='color', channel_order='bgr'):"
+ },
+ {
+ "comment": "This code converts a Pillow image to a numpy array. It checks the channel order and flag, then either keeps the array unchanged or converts it from RGB to BGR if necessary. If the image mode is not RGB, it converts it to RGB first using convert('RGB'). If the mode is LA, a random color is used for the canvas to avoid shadowing black objects in the foreground.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":225-246",
+ "content": " \"\"\"Convert a pillow image to numpy array. \"\"\"\n channel_order = channel_order.lower()\n if channel_order not in ['rgb', 'bgr']:\n raise ValueError('channel order must be either \"rgb\" or \"bgr\"')\n if flag == 'unchanged':\n array = np.array(img)\n if array.ndim >= 3 and array.shape[2] >= 3: # color image\n array[:, :, :3] = array[:, :, (2, 1, 0)] # RGB to BGR\n else:\n # If the image mode is not 'RGB', convert it to 'RGB' first.\n if img.mode != 'RGB':\n if img.mode != 'LA':\n # Most formats except 'LA' can be directly converted to RGB\n img = img.convert('RGB')\n else:\n # When the mode is 'LA', the default conversion will fill in\n # the canvas with black, which sometimes shadows black objects\n # in the foreground.\n #\n # Therefore, a random color (124, 117, 104) is used for canvas"
+ },
+ {
+ "comment": "The code reads an image from bytes and converts it into a numpy array based on the provided flag (color or grayscale) and channel order. It first checks if the flag is valid, then decodes the image using OpenCV's imdecode function. If the flag is color and channel order is rgb, it returns the image as is. For other combinations, it converts the image to RGB or grayscale before returning the numpy array.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":247-269",
+ "content": " img_rgba = img.convert('RGBA')\n img = Image.new('RGB', img_rgba.size, (124, 117, 104))\n img.paste(img_rgba, mask=img_rgba.split()[3]) # 3 is alpha\n if flag == 'color':\n array = np.array(img)\n if channel_order != 'rgb':\n array = array[:, :, ::-1] # RGB to BGR\n elif flag == 'grayscale':\n img = img.convert('L')\n array = np.array(img)\n else:\n raise ValueError(\n 'flag must be \"color\", \"grayscale\" or \"unchanged\", '\n f'but got {flag}')\n return array\n def _imfrombytes(self,content, flag='color', channel_order='bgr'):#, backend=None):\n \"\"\"Read an image from bytes. \"\"\"\n img_np = np.frombuffer(content, np.uint8)\n flag = imread_flags[flag] if isinstance(flag, str) else flag\n img = cv2.imdecode(img_np, flag)\n if flag == IMREAD_COLOR and channel_order == 'rgb':"
+ },
+ {
+ "comment": "This code defines a pipeline for decoding frames using the RawFrameDecode transform. It reads image files from the specified directory and suffix, handles different frame indices, and utilizes a file client to retrieve images in binary format. The cv2.cvtColor function is used to convert the color of images from BGR to RGB. The code also checks if the frame indices have the correct dimensions and squeezes them if necessary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":270-300",
+ "content": " cv2.cvtColor(img, cv2.COLOR_BGR2RGB, img)\n return img\n def __call__(self, results):\n \"\"\"Perform the ``RawFrameDecode`` to pick frames given indices.\n Args:\n results (dict): The resulting dict to be modified and passed\n to the next transform in pipeline.\n \"\"\"\n # mmcv.use_backend(self.decoding_backend)\n directory = results['frame_dir']\n suffix = results['suffix']\n #modality = results['modality']\n if self.file_client is None:\n self.file_client = FileClient(self.io_backend, **self.kwargs)\n imgs = list()\n if results['frame_inds'].ndim != 1:\n results['frame_inds'] = np.squeeze(results['frame_inds'])\n offset = results.get('offset', 0)\n for frame_idx in results['frame_inds']:\n frame_idx += offset\n filepath = osp.join(directory, suffix.format(frame_idx))\n img_bytes = self.file_client.get(filepath) #\u4ee5\u4e8c\u8fdb\u5236\u65b9\u5f0f\u8bfb\u53d6\u56fe\u7247\n # Get frame with channel order RGB directly."
+ },
+ {
+ "comment": "Function applies image processing and resizing to input, appends frames to a list, and scales gt_bboxes and proposals accordingly. It then returns the results. The __repr__ function provides a string representation of the object's class and arguments.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":302-325",
+ "content": " cur_frame = self._imfrombytes(img_bytes, channel_order='rgb')\n imgs.append(cur_frame)\n results['imgs'] = imgs\n results['original_shape'] = imgs[0].shape[:2]\n results['img_shape'] = imgs[0].shape[:2]\n # we resize the gt_bboxes and proposals to their real scale\n h, w = results['img_shape']\n scale_factor = np.array([w, h, w, h])\n if 'gt_bboxes' in results:\n gt_bboxes = results['gt_bboxes']\n gt_bboxes_new = (gt_bboxes * scale_factor).astype(np.float32)\n results['gt_bboxes'] = gt_bboxes_new\n if 'proposals' in results and results['proposals'] is not None:\n proposals = results['proposals']\n proposals = (proposals * scale_factor).astype(np.float32)\n results['proposals'] = proposals\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'io_backend={self.io_backend}, '\n f'decoding_backend={self.decoding_backend})')"
+ },
+ {
+ "comment": "The code defines a class called SampleAVAFrames, which inherits from SampleFrames. It takes clip length, frame interval, and test mode as arguments during initialization. The _get_clips method calculates the start and end indices for a given center index, taking into account skip offsets and shot information. The __call__ method retrieves fps, timestamp, timestamp_start, and shot_info from the results dictionary, and then calculates the center index to sample video frames around that index.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":326-353",
+ "content": " return repr_str\n@PIPELINES.register()\nclass SampleAVAFrames(SampleFrames):\n def __init__(self, clip_len, frame_interval=2, test_mode=False):\n super().__init__(clip_len, frame_interval, test_mode=test_mode)\n def _get_clips(self, center_index, skip_offsets, shot_info):\n start = center_index - (self.clip_len // 2) * self.frame_interval\n end = center_index + ((self.clip_len + 1) // 2) * self.frame_interval\n frame_inds = list(range(start, end, self.frame_interval))\n frame_inds = frame_inds + skip_offsets\n frame_inds = np.clip(frame_inds, shot_info[0], shot_info[1] - 1)\n return frame_inds\n def __call__(self, results):\n fps = results['fps']\n timestamp = results['timestamp']\n timestamp_start = results['timestamp_start']\n shot_info = results['shot_info']\n #delta=(timestamp - timestamp_start) \u4e3a\u8be5\u5e27\u8ddd\u79bb15min\u89c6\u9891\u5f00\u5934\u6709\u51e0\u79d2\n #center_index=fps*delta\u4e3a\u8be5\u5e27\u8ddd\u79bb15min\u89c6\u9891\u5f00\u5934\u6709\u51e0\u5e27\n #center_index+1\u662f\u4e3a\u4e86\u907f\u514d\u540e\u7eed\u91c7\u6837\u65f6\u51fa\u73b0\u8d1f\u6570? \n #\u540e\u7eed\u9700\u8981\u4ee5center_index\u4e3a\u4e2d\u5fc3\u524d\u540e\u91c7\u6837\u89c6\u9891\u5e27\u7247\u6bb5"
+ },
+ {
+ "comment": "This function samples a video clip by calculating the center index and generating random skip offsets to select frames. It returns frame indices, clip length, frame interval, number of clips, and crop quadruple in a dictionary format for further processing. The `__repr__` method provides a concise string representation of the object's attributes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/loader/pipelines/sample_ava.py\":354-373",
+ "content": " center_index = fps * (timestamp - timestamp_start) + 1\n skip_offsets = np.random.randint(\n -self.frame_interval // 2, (self.frame_interval + 1) // 2,\n size=self.clip_len)\n frame_inds = self._get_clips(center_index, skip_offsets, shot_info)\n results['frame_inds'] = np.array(frame_inds, dtype=np.int)\n results['clip_len'] = self.clip_len\n results['frame_interval'] = self.frame_interval\n results['num_clips'] = 1\n results['crop_quadruple'] = np.array([0, 0, 1, 1], dtype=np.float32)\n return results\n def __repr__(self):\n repr_str = (f'{self.__class__.__name__}('\n f'clip_len={self.clip_len}, '\n f'frame_interval={self.frame_interval}, '\n f'test_mode={self.test_mode})')\n return repr_str"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/909a3d87-fe65-4b1b-ba04-656643af5c4a.json b/docs/doc/909a3d87-fe65-4b1b-ba04-656643af5c4a.json
new file mode 100644
index 000000000..bd93c720e
--- /dev/null
+++ b/docs/doc/909a3d87-fe65-4b1b-ba04-656643af5c4a.json
@@ -0,0 +1,25 @@
+{
+ "summary": "This Python file utilizes PaddleVideo and PaddlePaddle library to construct video pipelines, defining functions for dataset, pipeline, and dataloader creation. It also includes signal handlers to terminate child processes upon receiving specific signals.",
+ "details": [
+ {
+ "comment": "This code is a Python file for building video pipeline in PaddleVideo, which uses PaddlePaddle library. It imports necessary modules and defines a function to build the pipeline according to the provided configuration. The logger is used for logging purposes, and numpy is imported for numerical operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/builder.py\":0-32",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nimport signal\nimport os\nimport paddle\nfrom paddle.io import DataLoader, DistributedBatchSampler\nfrom .registry import DATASETS, PIPELINES\nfrom ..utils.build_utils import build\nfrom .pipelines.compose import Compose\nfrom paddlevideo.utils import get_logger\nimport numpy as np\nlogger = get_logger(\"paddlevideo\")\ndef build_pipeline(cfg):\n \"\"\"Build pipeline.\n Args:\n cfg (dict): root config dict.\n \"\"\""
+ },
+ {
+ "comment": "This code defines functions to build a dataset, batch pipeline, and dataloader for PaddleVideo's Video Quality Assessment application. The build_dataset function constructs the dataset using cfg config dictionary. The build_batch_pipeline function builds the batch pipeline. Lastly, the build_dataloader function creates a Paddle Dataloader using the constructed dataset and other parameters like batch size, num_workers, etc.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/builder.py\":33-73",
+ "content": " return Compose(cfg)\ndef build_dataset(cfg):\n \"\"\"Build dataset.\n Args:\n cfg (dict): root config dict.\n Returns:\n dataset: dataset.\n \"\"\"\n #XXX: ugly code here!\n cfg_dataset, cfg_pipeline = cfg\n cfg_dataset.pipeline = build_pipeline(cfg_pipeline)\n dataset = build(cfg_dataset, DATASETS, key=\"format\")\n return dataset\ndef build_batch_pipeline(cfg):\n \"\"\"build batch pipeline\"\"\"\n batch_pipeline = build(cfg, PIPELINES)\n return batch_pipeline\ndef build_dataloader(dataset,\n batch_size,\n num_workers,\n places,\n shuffle=True,\n drop_last=True,\n multigrid=False,\n collate_fn_cfg=None,\n **kwargs):\n \"\"\"Build Paddle Dataloader.\n XXX explain how the dataloader work!\n Args:\n dataset (paddle.dataset): A PaddlePaddle dataset object.\n batch_size (int): batch size on single card.\n num_worker (int): num_worker"
+ },
+ {
+ "comment": "The code creates a DistributedBatchSampler for dataset with optional shuffle and drop_last parameters, and defines a mix_collate_fn function that applies a predefined collate_fn_cfg to batch data and returns it in a specific format using build_batch_pipeline.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/builder.py\":74-96",
+ "content": " shuffle(bool): whether to shuffle the data at every epoch.\n \"\"\"\n sampler = DistributedBatchSampler(dataset,\n batch_size=batch_size,\n shuffle=shuffle,\n drop_last=drop_last)\n #NOTE(shipping): when switch the mix operator on, such as: mixup, cutmix.\n # batch like: [[img, label, attibute, ...], [imgs, label, attribute, ...], ...] will recollate to:\n # [[img, img, ...], [label, label, ...], [attribute, attribute, ...], ...] as using numpy.transpose.\n def mix_collate_fn(batch):\n \"\"\"mix collate fn\"\"\"\n pipeline = build_batch_pipeline(collate_fn_cfg)\n batch = pipeline(batch)\n slots = []\n for items in batch:\n for i, item in enumerate(items):\n if len(slots) < len(items):\n slots.append([item])\n else:\n slots[i].append(item)\n return [np.stack(slot, axis=0) for slot in slots]"
+ },
+ {
+ "comment": "The code defines a function that returns a DataLoader object. If collate_fn_cfg is not None, it creates a mix_collate_fn and assigns it to the collate_fn variable. The returned DataLoader has its collate_fn set according to the value of collate_fn_cfg. The code also sets up signal handlers for SIGINT and SIGTERM, calling the term_mp function on receipt of either signal. The term_mp function kills all child processes in the current process group.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/builder.py\":98-125",
+ "content": " #if collate_fn_cfg is not None:\n #ugly code here. collate_fn is mix op config\n # collate_fn = mix_collate_fn(collate_fn_cfg)\n data_loader = DataLoader(\n dataset,\n batch_sampler=sampler,\n places=places,\n num_workers=num_workers,\n collate_fn=mix_collate_fn if collate_fn_cfg is not None else None,\n return_list=True,\n **kwargs)\n return data_loader\ndef term_mp(sig_num, frame):\n \"\"\" kill all child processes\n \"\"\"\n pid = os.getpid()\n pgid = os.getpgid(os.getpid())\n logger.info(\"main proc {} exit, kill process group \" \"{}\".format(pid, pgid))\n os.killpg(pgid, signal.SIGKILL)\n return\nsignal.signal(signal.SIGINT, term_mp)\nsignal.signal(signal.SIGTERM, term_mp)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/911f800a-47b0-41b0-8261-06017d9187f6.json b/docs/doc/911f800a-47b0-41b0-8261-06017d9187f6.json
new file mode 100644
index 000000000..f0d6bf54f
--- /dev/null
+++ b/docs/doc/911f800a-47b0-41b0-8261-06017d9187f6.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The code imports libraries and sets up an argument parser for PaddleVideo's Ma-Net application. It defines a function, configures model parameters, and trains video object detection tasks. The `--TEST_CHECKPOINT` and `--TEST_MODE` arguments are initialized with default values, and the training epoch count is calculated based on batch size and total steps.",
+ "details": [
+ {
+ "comment": "This code is importing necessary libraries and defining a function. It also sets up an argument parser, and provides default values for various parameters including the root directory, experiment name, save result directories, number of workers, KNNs, and pre-trained model path.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/config.py\":0-31",
+ "content": "import paddle\nimport argparse\nimport os\nimport sys\nimport cv2\nimport time\ndef str2bool(v):\n if isinstance(v, bool):\n return v\n if v.lower() in ('yes', 'true', 't', 'y', '1'):\n return True\n elif v.lower() in ('no', 'false', 'f', 'n', '0'):\n return False\n else:\n raise argparse.ArgumentTypeError('Boolean value expected.')\nparser = argparse.ArgumentParser(description='intvos config')\nparser.add_argument('--ROOT_DIR',\n type=str,\n default=os.path.abspath(\n os.path.join(os.path.dirname(\"__file__\"))))\nparser.add_argument('--EXP_NAME', type=str, default='deeplabv3+coco')\nparser.add_argument('--SAVE_RESULT_DIR', type=str, default='../afs/result/')\nparser.add_argument('--SAVE_VOS_RESULT_DIR', type=str, default='')\nparser.add_argument('--NUM_WORKER', type=int, default=4)\nparser.add_argument('--KNNS', type=int, default=1)\nparser.add_argument('--PRETRAINED_MODEL',\n type=str,\n default='./model_best.pth.tar')"
+ },
+ {
+ "comment": "This code snippet is from the 'config.py' file in the PaddleVideo/applications/Ma-Net directory, and it defines command line arguments for the application. It sets default values for parameters related to result storage location, data configuration, and model configuration. These arguments can be overridden when running the application by specifying them on the command line.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/config.py\":32-52",
+ "content": "parser.add_argument(\n '--RESULT_ROOT',\n type=str,\n default=os.path.join('../afs/vos_result/result_total_80000'))\n######DATA_CONFIG\nparser.add_argument('--DATA_NAME', type=str, default='COCO2017')\nparser.add_argument('--DATA_AUG', type=str2bool, default=True)\nparser.add_argument('--DATA_WORKERS', type=int, default=4)\nparser.add_argument('--DATA_RESCALE', type=int, default=416)\nparser.add_argument('--DATA_RANDOMCROP', type=int, default=416)\nparser.add_argument('--DATA_RANDOMROTATION', type=int, default=0)\nparser.add_argument('--DATA_RANDOM_H', type=int, default=10)\nparser.add_argument('--DATA_RANDOM_S', type=int, default=10)\nparser.add_argument('--DATA_RANDOM_V', type=int, default=10)\nparser.add_argument('--DATA_RANDOMFLIP', type=float, default=0.5)\nparser.add_argument('--DATA_ROOT', type=str, default='../data/DAVIS')\n######MODEL_CONFIG\nparser.add_argument('--MODEL_NAME', type=str, default='deeplabv3plus')\nparser.add_argument('--MODEL_BACKBONE', type=str, default='res101_atrous')\nparser.add_argument('--MODEL_OUTPUT_STRIDE', type=int, default=16)"
+ },
+ {
+ "comment": "This code snippet is from the \"config.py\" file in PaddleVideo's Ma-Net application, and it sets various model parameters like output dimension, shortcut dimensions, kernel size, number of classes, embedding dimensions, downsampling method, selection percentage, and training parameters such as learning rate, gamma, momentum, weight decay, and power. These parameters are used to configure and train the Ma-Net model for video object detection tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/config.py\":53-69",
+ "content": "parser.add_argument('--MODEL_ASPP_OUTDIM', type=int, default=256)\nparser.add_argument('--MODEL_SHORTCUT_DIM', type=int, default=48)\nparser.add_argument('--MODEL_SHORTCUT_KERNEL', type=int, default=1)\nparser.add_argument('--MODEL_NUM_CLASSES', type=int, default=21)\nparser.add_argument('--MODEL_SEMANTIC_EMBEDDING_DIM', type=int, default=100)\nparser.add_argument('--MODEL_HEAD_EMBEDDING_DIM', type=int, default=256)\nparser.add_argument('--MODEL_LOCAL_DOWNSAMPLE', type=str2bool, default=True)\nparser.add_argument('--MODEL_MAX_LOCAL_DISTANCE', type=int, default=12)\nparser.add_argument('--MODEL_SELECT_PERCENT', type=float, default=0.8)\nparser.add_argument('--MODEL_USEIntSeg', type=str2bool, default=False)\n######TRAIN_CONFIG\nparser.add_argument('--TRAIN_LR', type=float, default=0.0007)\nparser.add_argument('--TRAIN_LR_GAMMA', type=float, default=0.1)\nparser.add_argument('--TRAIN_MOMENTUM', type=float, default=0.9)\nparser.add_argument('--TRAIN_WEIGHT_DECAY', type=float, default=0.00004)\nparser.add_argument('--TRAIN_POWER', type=float, default=0.9)"
+ },
+ {
+ "comment": "This code snippet is part of the configuration file for the Ma-Net application in PaddleVideo. It includes various arguments and their default values for training the model, such as batch size, shuffling, gradient norm, number of epochs, total steps, loss lambda, logging settings, BN momentum, top K percent pixels, hard mining step, LR step size, and resuming from a specific directory.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/config.py\":70-87",
+ "content": "parser.add_argument('--TRAIN_BATCH_SIZE', type=int, default=2)\nparser.add_argument('--TRAIN_SHUFFLE', type=str2bool, default=True)\nparser.add_argument('--TRAIN_CLIP_GRAD_NORM', type=float, default=5.)\nparser.add_argument('--TRAIN_MINEPOCH', type=int, default=9)\nparser.add_argument('--TRAIN_TOTAL_STEPS', type=int, default=101000)\nparser.add_argument('--TRAIN_LOSS_LAMBDA', type=int, default=0)\nparser.add_argument('--TRAIN_TBLOG', type=str2bool, default=False)\nparser.add_argument('--TRAIN_BN_MOM', type=float,\n default=0.9997) # fixed. difs between paddle and torch.\nparser.add_argument('--TRAIN_TOP_K_PERCENT_PIXELS', type=float, default=0.15)\nparser.add_argument('--TRAIN_HARD_MINING_STEP', type=int, default=50000)\nparser.add_argument('--TRAIN_LR_STEPSIZE', type=int, default=2000)\nparser.add_argument('--TRAIN_INTER_USE_TRUE_RESULT',\n type=str2bool,\n default=True)\nparser.add_argument('--TRAIN_RESUME_DIR', type=str, default='')\nparser.add_argument('--LOG_DIR', type=str, default=os.path.join('./log'))"
+ },
+ {
+ "comment": "This code snippet initializes the `--TEST_CHECKPOINT` and `--TEST_MODE` arguments using default values, then calculates the number of training epochs based on the batch size and the total number of steps.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/config.py\":89-95",
+ "content": "parser.add_argument('--TEST_CHECKPOINT',\n type=str,\n default='save_step_100000.pth')\nparser.add_argument('--TEST_MODE', type=str2bool, default=False)\ncfg = parser.parse_args()\ncfg.TRAIN_EPOCHS = int(200000 * cfg.TRAIN_BATCH_SIZE / 60.)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/916a4e33-c5fa-4891-a1bc-0700e02e5986.json b/docs/doc/916a4e33-c5fa-4891-a1bc-0700e02e5986.json
new file mode 100644
index 000000000..f1e26bc14
--- /dev/null
+++ b/docs/doc/916a4e33-c5fa-4891-a1bc-0700e02e5986.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This introduction explains how to install PaddlePaddle and PaddleVideo, their requirements like Python 3.7 and CUDA 10.1, enabling distribution feature, setting shared memory in Docker, cloning PaddleVideo repo, upgrading pip and requirements, installing ppvideo package, and usage example specifying model, disabling GPU, and input video file.",
+ "details": [
+ {
+ "comment": "Introduction: Describes how to install PaddlePaddle, PaddleVideo, and their requirements.\nInstall PaddlePaddle: Requires Python 3.7, CUDA 10.1, CUDNN7.6.4 nccl2.1.2 and supports GPU training only. Follow the instructions on the website if PaddlePaddle on the device is lower than v2.0.\nInstallation commands: Use pip3 to install paddlepaddle-gpu or compile from source code, following instructions on the website.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/install.md\":0-40",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../zh-CN/install.md) | English\n# Installation\n---\n- [Introduction](#Introduction)\n- [Install PaddlePaddle](#Install-PaddlePaddle)\n- [Install PaddleVideo](#Install-PaddleVideo)\n## Introduction\nThis document introduces how to install PaddlePaddle\u3001PaddleVideo and its requirements.\n## Install PaddlePaddle\nPython 3.7, CUDA 10.1, CUDNN7.6.4 nccl2.1.2 and later version are required at first, For now, PaddleVideo only support training on the GPU device. Please follow the instructions in the [Installation](http://www.paddlepaddle.org.cn/install/quick) if the PaddlePaddle on the device is lower than v2.0\n**Install PaddlePaddle**\n```bash\npip3 install paddlepaddle-gpu --upgrade\n```\nor compile from source code, please refer to [Installation](http://www.paddlepaddle.org.cn/install/quick).\nVerify Installation\n```python\nimport paddle\npaddle.utils.run_check()\n```\nCheck PaddlePaddle version\uff1a\n```bash\npython3 -c \"import paddle; print(paddle.__version__)\"\n```\nNote:\n- Make sure the compiled version is later than PaddlePaddle2.0."
+ },
+ {
+ "comment": "WITH_DISTRIBUTE=ON: Enables the distribution feature in PaddleVideo, refer to Instruction for more details.\nDocker shm_size: Set --shm_size=32g when creating a docker container for enough shared memory.\nClone PaddleVideo: Navigate to desired path and clone the repository from GitHub.\nRequirements upgrade: Ensure pip is up-to-date before installing requirements.txt.\nInstall python package: Use pip3.7 to install specific version of ppvideo package.\nUsage example: Specify model, disable GPU usage, and input video file when running ppvideo script.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/install.md\":41-71",
+ "content": "- Indicate **WITH_DISTRIBUTE=ON** when compiling, Please refer to [Instruction](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/install/Tables.html#id3) for more details.\n- When running in the docker, in order to ensure that the container has enough shared memory for data read acceleration of Paddle, please set the parameter `--shm_size=32g` at creating a docker container, if conditions permit, you can set it to a larger value.\n---\n## Install PaddleVideo\n**Clone PaddleVideo:**\n```bash\ncd path_to_clone_PaddleVideo\ngit clone https://github.com/PaddlePaddle/PaddleVideo.git\n```\n**Install requirements**\n```bash\npython3.7 -m pip install --upgrade pip\npip3.7 install --upgrade -r requirements.txt\n```\n**Install python package**\n```bash\npip3.7 install ppvideo==2.3.0\n```\nuse scripts:\n```bash\nppvideo --model_name='ppTSM_v2' --use_gpu=False --video_file='data/example.avi'\n```"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/916b43f1-7297-4ac5-9be5-e1b9bc622090.json b/docs/doc/916b43f1-7297-4ac5-9be5-e1b9bc622090.json
new file mode 100644
index 000000000..602f5c9a8
--- /dev/null
+++ b/docs/doc/916b43f1-7297-4ac5-9be5-e1b9bc622090.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The code defines an AttentionLSTM class for a video tagging model, extending ModelBase. It initializes properties, retrieves configurations and dimensions, builds the LSTM attention model, applies fully connected layers, and uses an optimizer with piecewise learning rate decay and L2 regularization.",
+ "details": [
+ {
+ "comment": "The code is importing necessary modules and defining a class called AttentionLSTM. It extends the ModelBase class and has an __init__ method to initialize its properties such as name, configuration, mode, and is_videotag flag. The get_config method is also defined for retrieving configuration from a file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":0-30",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n#Licensed under the Apache License, Version 2.0 (the \"License\");\n#you may not use this file except in compliance with the License.\n#You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n#Unless required by applicable law or agreed to in writing, software\n#distributed under the License is distributed on an \"AS IS\" BASIS,\n#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n#See the License for the specific language governing permissions and\n#limitations under the License.\nfrom ..model import ModelBase\nfrom .lstm_attention import LSTMAttentionModel\nimport logging\nimport paddle\nimport paddle.static as static\nlogger = logging.getLogger(__name__)\n__all__ = [\"AttentionLSTM\"]\nclass AttentionLSTM(ModelBase):\n def __init__(self, name, cfg, mode='train', is_videotag=False):\n super(AttentionLSTM, self).__init__(name, cfg, mode)\n self.is_videotag = is_videotag\n self.get_config()"
+ },
+ {
+ "comment": "The code defines a model's configuration method, retrieving feature names, dimensions, number of classes, embedding size, LSTM size, and drop rate. It also gets mode-specific configurations such as batch size, number of GPUs, learning rate, weight decay, total training samples, and epochs for learning rate decay.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":32-53",
+ "content": " def get_config(self):\n # get model configs\n self.feature_names = self.cfg.MODEL.feature_names\n self.feature_dims = self.cfg.MODEL.feature_dims\n self.num_classes = self.cfg.MODEL.num_classes\n self.embedding_size = self.cfg.MODEL.embedding_size\n self.lstm_size = self.cfg.MODEL.lstm_size\n self.drop_rate = self.cfg.MODEL.drop_rate\n # get mode configs\n self.batch_size = self.get_config_from_sec(self.mode, 'batch_size', 1)\n self.num_gpus = self.get_config_from_sec(self.mode, 'num_gpus', 1)\n if self.mode == 'train':\n self.learning_rate = self.get_config_from_sec(\n 'train', 'learning_rate', 1e-3)\n self.weight_decay = self.get_config_from_sec(\n 'train', 'weight_decay', 8e-4)\n self.num_samples = self.get_config_from_sec('train', 'num_samples',\n 5000000)\n self.decay_epochs = self.get_config_from_sec(\n 'train', 'decay_epochs', [5])"
+ },
+ {
+ "comment": "The code initializes feature and label inputs for the model, depending on the mode. It also builds a dataloader if use_dataloader is True, but not recommended in infer mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":54-75",
+ "content": " self.decay_gamma = self.get_config_from_sec('train', 'decay_gamma',\n 0.1)\n def build_input(self, use_dataloader):\n self.feature_input = []\n for name, dim in zip(self.feature_names, self.feature_dims):\n self.feature_input.append(\n static.data(shape=[None, dim],\n lod_level=1,\n dtype='float32',\n name=name))\n if self.mode != 'infer':\n self.label_input = static.data(shape=[None, self.num_classes],\n dtype='float32',\n name='label')\n else:\n self.label_input = None\n if use_dataloader:\n assert self.mode != 'infer', \\\n 'dataloader is not recommendated when infer, please set use_dataloader to be false.'\n self.dataloader = paddle.io.DataLoader.from_generator(\n feed_list=self.feature_input + [self.label_input],"
+ },
+ {
+ "comment": "This code defines a LSTM attention model with multiple input features. It concatenates output of each feature, applies fully connected layers, and returns the result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":76-102",
+ "content": " capacity=8,\n iterable=True)\n def build_model(self):\n att_outs = []\n for i, (input_dim,\n feature) in enumerate(zip(self.feature_dims,\n self.feature_input)):\n att = LSTMAttentionModel(input_dim, self.embedding_size,\n self.lstm_size, self.drop_rate)\n att_out = att.forward(feature, is_training=(self.mode == 'train'))\n att_outs.append(att_out)\n if len(att_outs) > 1:\n out = paddle.concat(x=att_outs, axis=1)\n else:\n out = att_outs[0] # video only, without audio in videoTag\n fc1 = static.nn.fc(\n x=out,\n size=8192,\n activation='relu',\n bias_attr=paddle.ParamAttr(\n regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)),\n name='fc1')\n fc2 = static.nn.fc(\n x=fc1,"
+ },
+ {
+ "comment": "This code defines an attention LSTM model for video tagging. It uses a tanh activation function, L2 decay regularizer, and normal initializer for the fully connected layers. The logit layer applies sigmoid activation to output probabilities for each class. The optimizer function sets up a learning rate schedule using RMSProp optimizer with decay epochs and boundaries.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":103-124",
+ "content": " size=4096,\n activation='tanh',\n bias_attr=paddle.ParamAttr(\n regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)),\n name='fc2')\n self.logit = static.nn.fc(x=fc2, size=self.num_classes, activation=None, \\\n bias_attr=paddle.ParamAttr(regularizer=paddle.regularizer.L2Decay(coeff=0.0),\n initializer=paddle.nn.initializer.Normal(std=0.0)), name='output')\n self.output = paddle.nn.functional.sigmoid(self.logit)\n def optimizer(self):\n assert self.mode == 'train', \"optimizer only can be get in train mode\"\n values = [\n self.learning_rate * (self.decay_gamma**i)\n for i in range(len(self.decay_epochs) + 1)\n ]\n iter_per_epoch = self.num_samples / self.batch_size\n boundaries = [e * iter_per_epoch for e in self.decay_epochs]\n return paddle.optimizer.RMSProp("
+ },
+ {
+ "comment": "This code defines a model with an LSTM layer and attention mechanism. It uses piecewise learning rate decay, L2 weight decay regularization, calculates binary cross-entropy loss for classification tasks, and supports both training, validation, and inference modes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":125-150",
+ "content": " learning_rate=paddle.optimizer.lr.PiecewiseDecay(values=values,\n boundaries=boundaries),\n centered=True,\n weight_decay=paddle.regularizer.L2Decay(coeff=self.weight_decay))\n def loss(self):\n assert self.mode != 'infer', \"invalid loss calculationg in infer mode\"\n cost = paddle.nn.functional.binary_cross_entropy(\n input=self.logit, label=self.label_input, reduction=None)\n cost = paddle.sum(x=cost, axis=-1)\n sum_cost = paddle.sum(x=cost)\n self.loss_ = paddle.scale(sum_cost,\n scale=self.num_gpus,\n bias_after_scale=False)\n return self.loss_\n def outputs(self):\n return [self.output, self.logit]\n def feeds(self):\n return self.feature_input if self.mode == 'infer' else self.feature_input + [\n self.label_input\n ]\n def fetches(self):\n if self.mode == 'train' or self.mode == 'valid':"
+ },
+ {
+ "comment": "This code defines a class with three methods. The first method, `fetch_list()`, returns the fetch list for different modes ('train', 'test', or 'infer'). In 'train' mode, it calculates losses and includes them in the fetch list. In 'test' mode, it does the same. In 'infer' mode, only the output is included in the fetch list. If an unrecognized mode is provided, a `NotImplementedError` is raised. The `weights_info()` method returns no information as it is not implemented yet. Lastly, the `load_pretrain_params()` method loads pretrained weights from a given file, excluding any \"fc_0\" layer parameters, and logs a message confirming this action.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoTag/models/attention_lstm/attention_lstm.py\":151-179",
+ "content": " losses = self.loss()\n fetch_list = [losses, self.output, self.label_input]\n elif self.mode == 'test':\n losses = self.loss()\n fetch_list = [losses, self.output, self.label_input]\n elif self.mode == 'infer':\n fetch_list = [self.output]\n else:\n raise NotImplementedError('mode {} not implemented'.format(\n self.mode))\n return fetch_list\n def weights_info(self):\n return None, None\n def load_pretrain_params(self, exe, pretrain, prog):\n logger.info(\n \"Load pretrain weights from {}, exclude fc layer.\".format(pretrain))\n state_dict = paddle.static.load_program_state(pretrain)\n dict_keys = list(state_dict.keys())\n for name in dict_keys:\n if \"fc_0\" in name:\n del state_dict[name]\n logger.info(\n 'Delete {} from pretrained parameters. Do not load it'.\n format(name))\n paddle.static.set_program_state(prog, state_dict)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9184c97c-2073-483f-84da-9aff0d4e4d06.json b/docs/doc/9184c97c-2073-483f-84da-9aff0d4e4d06.json
new file mode 100644
index 000000000..137ecfc90
--- /dev/null
+++ b/docs/doc/9184c97c-2073-483f-84da-9aff0d4e4d06.json
@@ -0,0 +1,25 @@
+{
+ "summary": "This Python code implements a PaddlePaddle neural network head for classification tasks using ppTSN Head, initializing the base class and applying dropout regularization with an FC layer. The init_weights function sets the FC layer's initial weights.",
+ "details": [
+ {
+ "comment": "This code is the Python implementation of ppTSN Head, a classification model head used in PaddleVideo. The class has the number of classes and input channels as arguments. It inherits from BaseHead and is registered to the HEADS registry using @HEADS.register(). The code also imports necessary libraries and functions for its operations such as Linear, AdaptiveAvgPool2D, Dropout, ParamAttr, L2Decay, paddle.nn, and PaddleVideo's base and weight_init modules.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptsn_head.py\":0-29",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport paddle\nfrom paddle import ParamAttr\nfrom paddle.nn import AdaptiveAvgPool2D, Linear, Dropout\nfrom paddle.regularizer import L2Decay\nfrom .base import BaseHead\nfrom ..registry import HEADS\nfrom ..weight_init import weight_init_\n@HEADS.register()\nclass ppTSNHead(BaseHead):\n \"\"\"ppTSN Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature."
+ },
+ {
+ "comment": "This code defines a class with an __init__ method, taking parameters such as num_classes, in_channels, loss_cfg, drop_ratio, std, data_format, and fclr5. It initializes the base class and sets the drop_ratio, std, and creates an AdaptiveAvgPool2D object for global pooling performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptsn_head.py\":30-53",
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n drop_ratio(float): drop ratio. Default: 0.4.\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n data_format(str): data format of input tensor in ['NCHW', 'NHWC']. Default: 'NCHW'.\n fclr5(bool): Whether to increase the learning rate of the fully connected layer. Default: True\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n drop_ratio=0.4,\n std=0.01,\n data_format=\"NCHW\",\n fclr5=True,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.drop_ratio = drop_ratio\n self.std = std\n # NOTE: global pool performance\n self.avgpool2d = AdaptiveAvgPool2D((1, 1), data_format=data_format)"
+ },
+ {
+ "comment": "This code initializes and defines a PaddlePaddle neural network head for classification tasks. It includes optional dropout regularization, an FC layer with learnable parameters, and a forward function to process input data. The init_weights function is used to set the initial weights of the FC layer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptsn_head.py\":55-83",
+ "content": " if self.drop_ratio != 0:\n self.dropout = Dropout(p=self.drop_ratio)\n else:\n self.dropout = None\n self.fc = Linear(\n self.in_channels,\n self.num_classes,\n weight_attr=ParamAttr(learning_rate=5.0 if fclr5 else 1.0,\n regularizer=L2Decay(1e-4)),\n bias_attr=ParamAttr(learning_rate=10.0 if fclr5 else 1.0,\n regularizer=L2Decay(0.0)))\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'Normal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.,\n std=self.std)\n def forward(self, x, num_seg=8):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n num_segs (int): Number of segments.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples."
+ },
+ {
+ "comment": "This code snippet is responsible for processing the input and output of a PPTSN head model. It performs average pooling, reshapes the tensor, calculates the mean along an axis, applies dropout if applicable, reshapes again, and finally passes the result through a fully connected layer to produce scores.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptsn_head.py\":84-102",
+ "content": " \"\"\"\n # XXX: check dropout location!\n # [N * num_segs, in_channels, 7, 7]\n x = self.avgpool2d(x)\n # [N * num_segs, in_channels, 1, 1]\n x = paddle.reshape(x, [-1, num_seg, x.shape[1]])\n # [N, num_seg, in_channels]\n x = paddle.mean(x, axis=1)\n # [N, in_channels]\n if self.dropout is not None:\n x = self.dropout(x)\n # [N, in_channels]\n x = paddle.reshape(x, shape=[-1, self.in_channels])\n # [N, in_channels]\n score = self.fc(x)\n # [N, num_class]\n # x = F.softmax(x) # NOTE remove\n return score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/918d2d55-8374-4222-b2e7-d252e0bbbd0f.json b/docs/doc/918d2d55-8374-4222-b2e7-d252e0bbbd0f.json
new file mode 100644
index 000000000..904e96a97
--- /dev/null
+++ b/docs/doc/918d2d55-8374-4222-b2e7-d252e0bbbd0f.json
@@ -0,0 +1,150 @@
+{
+ "summary": "The code trains Ma-Net stage 2 models with adjustable learning rates, applies binary cross-entropy loss, and evaluates performance. It also performs image processing, ROI operations, and video analysis using pretrained network weights.",
+ "details": [
+ {
+ "comment": "Import necessary libraries and modules, define custom data loader class, set device to GPU0, and disable static mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":0-33",
+ "content": "import cv2\nimport paddle\nimport paddle.nn as nn\nimport os\nimport numpy as np\n# from paddle.io import DataLoader\nimport paddle.optimizer as optim\nfrom paddle.vision import transforms\nfrom dataloaders.davis_2017_f import DAVIS2017_Train\nimport dataloaders.custom_transforms_f as tr\nfrom dataloaders.samplers import RandomIdentitySampler\nfrom networks.deeplab import DeepLab\nfrom networks.IntVOS import IntVOS\nfrom networks.loss import Added_BCEWithLogitsLoss, Added_CrossEntropyLoss\nfrom config import cfg\nfrom utils.api import float_, long_, byte_\nfrom utils.meters import AverageMeter\nfrom utils.mask_damaging import damage_masks, mask_damager\nfrom utils.utils import label2colormap\nfrom PIL import Image\nimport random\nimport scipy.misc as sm\nimport time\nimport davisinteractive.robot.interactive_robot as interactive_robot\npaddle.disable_static()\npaddle.device.set_device(\"gpu:0\")\nclass DataLoader(paddle.io.DataLoader):\n def __init__(self,\n dataset,\n batch_size=1,\n shuffle=False,"
+ },
+ {
+ "comment": "The code initializes a DataLoader with parameters. It checks if the dataset contains tuples or lists, then sets return_list accordingly. It initializes the DataLoader using the dataset, batch size, shuffle, etc., and returns a DataLoader object for loading data efficiently.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":34-60",
+ "content": " sampler=None,\n batch_sampler=None,\n num_workers=0,\n collate_fn=None,\n pin_memory=False,\n drop_last=False,\n timeout=0,\n worker_init_fn=None,\n multiprocessing_context=None,\n generator=None):\n if isinstance(dataset[0], (tuple, list)):\n return_list = True\n else:\n return_list = False\n super().__init__(dataset,\n feed_list=None,\n places=None,\n return_list=return_list,\n batch_sampler=batch_sampler,\n batch_size=batch_size,\n shuffle=shuffle,\n drop_last=drop_last,\n collate_fn=collate_fn,\n num_workers=num_workers,\n use_buffer_reader=True,\n use_shared_memory=False,"
+ },
+ {
+ "comment": "This code initializes a Manager object with options for GPU usage, time budget, result directory, pretrained model, and interactive testing. It loads the feature extractor, DeepLab, and the VOS model using provided parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":61-88",
+ "content": " timeout=timeout,\n worker_init_fn=worker_init_fn)\n if sampler is not None:\n self.batch_sampler.sampler = sampler\nclass Manager(object):\n def __init__(self,\n use_gpu=True,\n time_budget=None,\n save_result_dir=cfg.SAVE_RESULT_DIR,\n pretrained=True,\n interactive_test=False):\n self.save_res_dir = save_result_dir\n self.time_budget = time_budget\n self.feature_extracter = DeepLab(backbone='resnet')\n if pretrained:\n pretrained_dict = paddle.load(cfg.PRETRAINED_MODEL)\n pretrained_dict = pretrained_dict['state_dict']\n self.load_network(self.feature_extracter, pretrained_dict)\n print('load pretrained model successfully.')\n self.model = IntVOS(cfg, self.feature_extracter)\n model_filename = cfg.SAVE_VOS_RESULT_DIR\n pd = paddle.load(model_filename)\n self.load_network(self.model, pd)"
+ },
+ {
+ "comment": "This code initializes a model and optimizer for training stage 2 of the Ma-Net application. It uses a GPU if specified, sets up training parameters, and initializes transforms to apply data augmentation during training. The model's segment head is trained using Momentum optimization with specified learning rate, momentum, and weight decay values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":90-118",
+ "content": " print('load stage 1 model from', model_filename)\n self.use_gpu = use_gpu\n if use_gpu:\n self.model = self.model\n ##################################\n def train(self,\n damage_initial_previous_frame_mask=True,\n lossfunc='cross_entropy',\n model_resume=False,\n eval_total=False,\n init_prev=False):\n ###################\n interactor = interactive_robot.InteractiveScribblesRobot()\n self.model.train()\n running_loss = AverageMeter()\n optimizer = optim.Momentum(parameters=[{\n 'params':\n self.model.inter_seghead.parameters()\n }],\n learning_rate=cfg.TRAIN_LR,\n momentum=cfg.TRAIN_MOMENTUM,\n weight_decay=cfg.TRAIN_WEIGHT_DECAY)\n ###################\n composed_transforms = transforms.Compose([\n tr.RandomHorizontalFlip(cfg.DATA_RANDOMFLIP),\n tr.RandomScale(),"
+ },
+ {
+ "comment": "The code initializes a dataset, applies transformations to the images, and selects the loss function. It then sets up the maximum number of iterations, and keeps track of current iteration and round numbers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":119-144",
+ "content": " tr.RandomCrop((cfg.DATA_RANDOMCROP, cfg.DATA_RANDOMCROP), 10),\n tr.Resize(cfg.DATA_RESCALE),\n tr.ToTensor()\n ])\n print('dataset processing...')\n train_dataset = DAVIS2017_Train(root=cfg.DATA_ROOT,\n transform=composed_transforms)\n train_list = train_dataset.seqs\n print('dataset processing finished.')\n if lossfunc == 'bce':\n criterion = Added_BCEWithLogitsLoss(cfg.TRAIN_TOP_K_PERCENT_PIXELS,\n cfg.TRAIN_HARD_MINING_STEP)\n elif lossfunc == 'cross_entropy':\n criterion = Added_CrossEntropyLoss(cfg.TRAIN_TOP_K_PERCENT_PIXELS,\n cfg.TRAIN_HARD_MINING_STEP)\n else:\n print(\n 'unsupported loss funciton. Please choose from [cross_entropy,bce]'\n )\n max_itr = cfg.TRAIN_TOTAL_STEPS\n step = 0\n round_ = 3\n epoch_per_round = 30"
+ },
+ {
+ "comment": "The code checks if model resuming is enabled, and if so, loads a saved model from a specific step and updates the current model. It then enters a loop where it trains the interaction branch for each round, performing various data transformations like random flipping, scaling, cropping, resizing, and converting to tensor. The training stops after 80,001 steps or if r is not equal to 0 (first round).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":145-169",
+ "content": " if model_resume:\n saved_model_ = os.path.join(self.save_res_dir,\n 'save_step_75000.pth')\n saved_model_ = paddle.load(saved_model_)\n self.model = self.load_network(self.model, saved_model_)\n step = 75000\n print('resume from step {}'.format(step))\n while step < cfg.TRAIN_TOTAL_STEPS:\n if step > 80001:\n break\n for r in range(round_):\n if r == 0: #### r==0: Train the interaction branch in the first round\n print('start new')\n global_map_tmp_dic = {}\n train_dataset.transform = transforms.Compose([\n tr.RandomHorizontalFlip(cfg.DATA_RANDOMFLIP),\n tr.RandomScale(),\n tr.RandomCrop(\n (cfg.DATA_RANDOMCROP, cfg.DATA_RANDOMCROP)),\n tr.Resize(cfg.DATA_RESCALE),\n tr.ToTensor()"
+ },
+ {
+ "comment": "The code initializes a dataset, creates a data loader with a random sampler, and adjusts the learning rate. It then loops through the dataset for a specified number of epochs, accessing relevant sample features and labels. The length of the dataset and data loader are printed before training begins.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":170-190",
+ "content": " ])\n train_dataset.init_ref_frame_dic()\n trainloader = DataLoader(train_dataset,\n sampler=RandomIdentitySampler(\n train_dataset.sample_list),\n shuffle=False,\n batch_size=cfg.TRAIN_BATCH_SIZE,\n num_workers=0)\n print('round:{} start'.format(r))\n print(len(train_dataset))\n print(len(trainloader))\n for epoch in range(epoch_per_round):\n for ii, sample in enumerate(trainloader):\n now_lr = self._adjust_lr(optimizer, step, max_itr)\n ref_imgs = sample['ref_img'] # batch_size * 3 * h * w\n ref_scribble_labels = sample[\n 'ref_scribble_label'] # batch_size * 1 * h * w\n seq_names = sample['meta']['seq_name']"
+ },
+ {
+ "comment": "The code initializes variables and sets up the model for training stage 2. It handles GPU usage, evaluates the model's feature extractor and semantic embedding, and extracts feature embeddings from reference frame images. It then checks if it's processing the first inter-frame instance and calls int_seghead function with reference frame embeddings as input.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":191-211",
+ "content": " obj_nums = sample['meta']['obj_num']\n ref_frame_nums = sample['meta']['ref_frame_num']\n ref_frame_gts = sample['ref_frame_gt']\n bs, _, h, w = ref_imgs.shape\n ##########\n if self.use_gpu:\n inputs = ref_imgs\n ref_scribble_labels = ref_scribble_labels\n ref_frame_gts = ref_frame_gts\n ##########\n with paddle.no_grad():\n self.model.feature_extracter.eval()\n self.model.semantic_embedding.eval()\n ref_frame_embedding = self.model.extract_feature(\n inputs)\n if r == 0:\n first_inter = True\n tmp_dic = self.model.int_seghead(\n ref_frame_embedding=ref_frame_embedding,"
+ },
+ {
+ "comment": "This code snippet seems to be a part of a larger function and appears to involve image classification tasks. The code initializes variables such as `ref_scribble_labels`, `prev_round_label`, `normalize_nearest_neighbor_distances`, `global_map_tmp_dic`, `seq_names`, `gt_ids`, `k_nearest_neighbors`, and `frame_num`. The code also checks if a variable named `first_inter` exists, and if not, initializes it as `False` along with `prev_round_label` and performs some operations using the `model.int_seghead()` method.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":212-228",
+ "content": " ref_scribble_label=ref_scribble_labels,\n prev_round_label=None,\n normalize_nearest_neighbor_distances=True,\n global_map_tmp_dic={},\n seq_names=seq_names,\n gt_ids=obj_nums,\n k_nearest_neighbors=cfg.KNNS,\n frame_num=ref_frame_nums,\n first_inter=first_inter)\n else:\n first_inter = False\n prev_round_label = sample['prev_round_label']\n prev_round_label = prev_round_label\n tmp_dic = self.model.int_seghead(\n ref_frame_embedding=ref_frame_embedding,\n ref_scribble_label=ref_scribble_labels,\n prev_round_label=prev_round_label,"
+ },
+ {
+ "comment": "The code initializes an empty dictionary for label and object dictionaries. It then iterates through the sequence names, assigning the corresponding ground truth frame and object number to each sequence in the label_and_obj_dic dictionary. Next, it iterates through the temporary dictionary keys, interpolating the predicted logits of each sequence to a fixed size (h, w) using bilinear interpolation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":229-246",
+ "content": " normalize_nearest_neighbor_distances=True,\n global_map_tmp_dic={},\n seq_names=seq_names,\n gt_ids=obj_nums,\n k_nearest_neighbors=cfg.KNNS,\n frame_num=ref_frame_nums,\n first_inter=first_inter)\n label_and_obj_dic = {}\n label_dic = {}\n for i, seq_ in enumerate(seq_names):\n label_and_obj_dic[seq_] = (ref_frame_gts[i],\n obj_nums[i])\n for seq_ in tmp_dic.keys():\n tmp_pred_logits = tmp_dic[seq_]\n tmp_pred_logits = nn.functional.interpolate(\n tmp_pred_logits,\n size=(h, w),\n mode='bilinear',"
+ },
+ {
+ "comment": "This code section is responsible for handling label and object dictionaries, preparing the data for different loss functions, and calculating the loss based on the provided data. It also performs necessary tensor conversions and optimizer operations.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":247-264",
+ "content": " align_corners=True)\n tmp_dic[seq_] = tmp_pred_logits\n label_tmp, obj_num = label_and_obj_dic[seq_]\n obj_ids = np.arange(0, obj_num + 1)\n obj_ids = paddle.to_tensor(obj_ids)\n obj_ids = paddle.to_tensor(obj_ids, dtype='int64')\n if lossfunc == 'bce':\n label_tmp = label_tmp.permute(1, 2, 0)\n label = (float_(label_tmp) == float_(obj_ids))\n label = label.unsqueeze(-1).permute(3, 2, 0, 1)\n label_dic[seq_] = float_(label)\n elif lossfunc == 'cross_entropy':\n label_dic[seq_] = long_(label_tmp)\n loss = criterion(tmp_dic, label_dic, step)\n loss = loss / bs\n optimizer.clear_grad()"
+ },
+ {
+ "comment": "Updating the running loss and printing details, including step, current learning rate, and loss values. Visualizing reference image and ground truth frame, and converting scribble labels to color maps.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":265-286",
+ "content": " loss.backward()\n optimizer.step()\n running_loss.update(loss.item(), bs)\n if step % 50 == 0:\n print(\n 'step:{},now_lr:{} ,loss:{:.4f}({:.4f})'.format(\n step, now_lr, running_loss.val,\n running_loss.avg))\n show_ref_img = ref_imgs.numpy()[0]\n mean = np.array([[[0.485]], [[0.456]], [[0.406]]])\n sigma = np.array([[[0.229]], [[0.224]], [[0.225]]])\n show_ref_img = show_ref_img * sigma + mean\n show_gt = ref_frame_gts[0].squeeze(0).numpy()\n show_gtf = label2colormap(show_gt).transpose(\n (2, 0, 1))\n show_scrbble = ref_scribble_labels[0].squeeze(\n 0).numpy()"
+ },
+ {
+ "comment": "This code is handling the visualization of labels and predictions. If r is not zero, it retrieves the previous round label, maps it to a color map, and transposes it for visualization. If r is zero, it creates a zero-filled array for the previous round label. The final step is getting the predictions for the first sequence name, interpolating them to fit the image size, and preparing them for visualization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":287-305",
+ "content": " show_scrbble = label2colormap(\n show_scrbble).transpose((2, 0, 1))\n if r != 0:\n show_prev_round_label = prev_round_label[\n 0].squeeze(0).numpy()\n show_prev_round_label = label2colormap(\n show_prev_round_label).transpose((2, 0, 1))\n else:\n show_prev_round_label = np.zeros_like(show_gt)\n show_prev_round_label = label2colormap(\n show_prev_round_label).transpose((2, 0, 1))\n ##########\n show_preds = tmp_dic[seq_names[0]]\n show_preds = nn.functional.interpolate(\n show_preds,\n size=(h, w),\n mode='bilinear',"
+ },
+ {
+ "comment": "This code is segmenting an image by applying a binary cross-entropy or cross-entropy loss function to the output of a PaddlePaddle neural network. The resulting segmentation map is stored in 'show_preds_s' after being converted to a numpy array and then transformed using the 'label2colormap' function.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":306-323",
+ "content": " align_corners=True)\n show_preds = show_preds.squeeze(0)\n if lossfunc == 'bce':\n show_preds = show_preds[1:]\n show_preds = (\n paddle.nn.functional.sigmoid(show_preds) >\n 0.5)\n marker = paddle.argmax(show_preds, axis=0)\n show_preds_s = paddle.zeros((h, w))\n for i in range(show_preds.size(0)):\n tmp_mask = (marker\n == i) & (show_preds[i] > 0.5)\n show_preds_s[tmp_mask] = i + 1\n elif lossfunc == 'cross_entropy':\n show_preds_s = paddle.argmax(show_preds, axis=0)\n show_preds_s = show_preds_s.numpy()\n show_preds_sf = label2colormap("
+ },
+ {
+ "comment": "This code block is responsible for saving the network at certain intervals during training. It checks if the current step is a multiple of 20,000 and not the first step, then calls save_network function to store the model's parameters. The model is also evaluated on the trainset and its performance might be influenced by the cfg.TRAIN_INTER_USE_TRUE_RESULT flag which determines whether to use the true result for evaluation. This block also resets transforms of the traindataset at specific rounds (r != round_-1).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":324-349",
+ "content": " show_preds_s).transpose((2, 0, 1))\n pix_acc = np.sum(show_preds_s == show_gt) / (h * w)\n ###########TODO\n if step % 20000 == 0 and step != 0:\n self.save_network(self.model, step)\n step += 1\n print('trainset evaluating...')\n print('*' * 100)\n if cfg.TRAIN_INTER_USE_TRUE_RESULT:\n if r != round_ - 1:\n if r == 0:\n prev_round_label_dic = {}\n self.model.eval()\n with paddle.no_grad():\n round_scribble = {}\n frame_num_dic = {}\n train_dataset.transform = transforms.Compose(\n [tr.Resize(cfg.DATA_RESCALE),\n tr.ToTensor()])\n trainloader = DataLoader("
+ },
+ {
+ "comment": "The code is initializing a data loader for training stage 2. It uses the RandomIdentitySampler with the train_dataset, sets shuffle to False, batch size to 1, and num_workers to 0. Then it iterates through the trainloader, extracting ref_imgs, img1s, img2s, ref_scribble_labels, label1s, label2s, and seq_names from each sample.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":350-366",
+ "content": " train_dataset,\n sampler=RandomIdentitySampler(\n train_dataset.sample_list),\n shuffle=False,\n batch_size=1,\n num_workers=0)\n for ii, sample in enumerate(trainloader):\n ref_imgs = sample[\n 'ref_img'] # batch_size * 3 * h * w\n img1s = sample['img1']\n img2s = sample['img2']\n ref_scribble_labels = sample[\n 'ref_scribble_label'] # batch_size * 1 * h * w\n label1s = sample['label1']\n label2s = sample['label2']\n seq_names = sample['meta']['seq_name']\n obj_nums = sample['meta']['obj_num']"
+ },
+ {
+ "comment": "This code segment is a part of an image processing and scribble labeling task. It extracts the frame numbers from sample metadata, concatenates reference images, img1, and img2. It applies rough ROI (region of interest) operation on ref_scribble_labels if r equals 0. Then, it processes the label1s, creating a tensor for scribble labels to be used in the model's input.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":367-385",
+ "content": " frame_nums = sample['meta']['frame_num']\n bs, _, h, w = img2s.shape\n inputs = paddle.concat((ref_imgs, img1s, img2s),\n 0)\n if r == 0:\n ref_scribble_labels = self.rough_ROI(\n ref_scribble_labels)\n print(seq_names[0])\n label1s_tocat = None\n for i in range(bs):\n l = label1s[i]\n l = l.unsqueeze(0)\n l = mask_damager(l, 0.0)\n l = paddle.to_tensor(l)\n l = l.unsqueeze(0).unsqueeze(0)\n if label1s_tocat is None:\n label1s_tocat = float_(l)"
+ },
+ {
+ "comment": "This code is part of a machine learning model training process. It appears to be concatenating label data (label1s_tocat) and checking if GPU usage is required. The model then processes input data, reference scribble labels, and labels (label1s) to produce outputs (tmp_dic). The specific output used is determined by the variable 'pred_label'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":386-405",
+ "content": " else:\n label1s_tocat = paddle.concat(\n (label1s_tocat, float_(l)), 0)\n label1s = label1s_tocat\n if self.use_gpu:\n inputs = inputs\n ref_scribble_labels = ref_scribble_labels\n label1s = label1s\n tmp_dic, global_map_tmp_dic = self.model(\n inputs,\n ref_scribble_labels,\n label1s,\n seq_names=seq_names,\n gt_ids=obj_nums,\n k_nearest_neighbors=cfg.KNNS,\n global_map_tmp_dic=global_map_tmp_dic,\n frame_num=frame_nums)\n pred_label = tmp_dic["
+ },
+ {
+ "comment": "The code is performing inference for an image classification task. It detaches, interpolates and converts the predicted label tensor to obtain the final prediction. The try-except block handles potential errors when applying a function called \"damage_masks\" on the prediction label. Finally, it applies the \"interact\" function from a class called \"interactor\" on the sequence with name \"seq_names[0]\" using numpy arrays for prediction and ground truth labels.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":406-422",
+ "content": " seq_names[0]].detach().cpu()\n pred_label = nn.functional.interpolate(\n pred_label,\n size=(h, w),\n mode='bilinear',\n align_corners=True)\n pred_label = paddle.argmax(pred_label, axis=1)\n pred_label = pred_label.unsqueeze(0)\n try:\n pred_label = damage_masks(pred_label)\n except:\n pred_label = pred_label\n pred_label = pred_label.squeeze(0)\n round_scribble[\n seq_names[0]] = interactor.interact(\n seq_names[0], pred_label.numpy(),\n float_(label2s).squeeze(0).numpy(),"
+ },
+ {
+ "comment": "This code opens an image and resizes the prediction label to match the image's height and width. Then, it updates the reference frame and label in the training dataset for a specific sequence name.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":423-438",
+ "content": " obj_nums)\n frame_num_dic[seq_names[0]] = frame_nums[0]\n pred_label = pred_label.unsqueeze(0)\n img_ww = Image.open(\n os.path.join(cfg.DATA_ROOT,\n 'JPEGImages/480p/',\n seq_names[0], '00000.jpg'))\n img_ww = np.array(img_ww)\n or_h, or_w = img_ww.shape[:2]\n pred_label = paddle.nn.functional.interpolate(\n float_(pred_label), (or_h, or_w),\n mode='nearest')\n prev_round_label_dic[\n seq_names[0]] = pred_label.squeeze(0)\n train_dataset.update_ref_frame_and_label(\n round_scribble, frame_num_dic, prev_round_label_dic)"
+ },
+ {
+ "comment": "This code segment appears to be a part of a training process for a video analysis model. It's updating the reference frame and label based on the current round, possibly in a round-based training loop. The `RandomIdentitySampler` seems to be used to load the data for this specific round.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":440-461",
+ "content": " print(f'round {r}', 'trainset evaluating finished!')\n print('*' * 100)\n self.model.train()\n print('updating ref frame and label')\n train_dataset.transform = composed_transforms\n print('updating ref frame and label finished!')\n else:\n if r != round_ - 1:\n round_scribble = {}\n if r == 0:\n prev_round_label_dic = {}\n frame_num_dic = {}\n train_dataset.transform = tr.ToTensor()\n trainloader = DataLoader(train_dataset,\n sampler=RandomIdentitySampler(\n train_dataset.sample_list),\n shuffle=False,\n batch_size=1,\n num_workers=0)"
+ },
+ {
+ "comment": "This code is in the \"train_stage2.py\" file of PaddleVideo's Ma-Net application, and it prepares for training by setting the model to evaluation mode, disabling gradient tracking with paddle.no_grad(), iterating over training data using trainloader, and extracting necessary samples. It also applies a mask_damager to label2s with 0.1 intensity to potentially improve model performance.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":463-480",
+ "content": " self.model.eval()\n with paddle.no_grad():\n for ii, sample in enumerate(trainloader):\n ref_imgs = sample[\n 'ref_img'] # batch_size * 3 * h * w\n img1s = sample['img1']\n img2s = sample['img2']\n ref_scribble_labels = sample[\n 'ref_scribble_label'] # batch_size * 1 * h * w\n label1s = sample['label1']\n label2s = sample['label2']\n seq_names = sample['meta']['seq_name']\n obj_nums = sample['meta']['obj_num']\n frame_nums = sample['meta']['frame_num']\n bs, _, h, w = img2s.shape\n print(seq_names[0])\n label2s_ = mask_damager(label2s, 0.1)"
+ },
+ {
+ "comment": "This code updates the reference frame and label for the train_dataset, sets the model to training mode, and prints progress messages. The interactor is used to interact with the first sequence's data and update the round_scribble variable. Label2s and frame_nums are used in this process, and prev_round_label_dic stores the previous round's label for future reference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":481-499",
+ "content": " round_scribble[\n seq_names[0]] = interactor.interact(\n seq_names[0],\n np.expand_dims(label2s_, axis=0),\n float_(label2s).squeeze(0).numpy(),\n obj_nums)\n label2s__ = paddle.to_tensor(label2s_)\n frame_num_dic[seq_names[0]] = frame_nums[0]\n prev_round_label_dic[seq_names[0]] = label2s__\n print(f'round {r}', 'trainset evaluating finished!')\n print('*' * 100)\n print('updating ref frame and label')\n train_dataset.update_ref_frame_and_label(\n round_scribble, frame_num_dic, prev_round_label_dic)\n self.model.train()\n train_dataset.transform = composed_transforms"
+ },
+ {
+ "comment": "This function rough_ROI takes ref_scribble_labels as input and performs a region of interest (ROI) operation. It iterates over each batch element, identifies the valid non-background regions, calculates the minimum and maximum coordinates within these regions, and then creates a filter mask. The filter mask is used to selectively copy the ref_scribble_labels into final_scribble_labels for further processing. The operation ensures that only relevant regions are considered, improving efficiency.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":500-524",
+ "content": " print('updating ref frame and label finished!')\n #############################################\n def rough_ROI(self, ref_scribble_labels):\n #### b*1*h*w\n dist = 15\n b, _, h, w = ref_scribble_labels.shape\n filter_ = paddle.zeros_like(ref_scribble_labels)\n to_fill = paddle.zeros_like(ref_scribble_labels)\n for i in range(b):\n no_background = (ref_scribble_labels[i] != -1)\n no_background = no_background.squeeze(0)\n no_b = no_background.nonzero()\n h_min, w_min = paddle.min(no_b, 0) # fixed\n h_max, w_max = paddle.max(no_b, 0) # fixed\n filter_[i, 0,\n max(h_min - dist, 0):min(h_max + dist, h - 1),\n max(w_min - dist, 0):min(w_max + dist, w - 1)] = 1\n final_scribble_labels = paddle.where(byte_(filter_),\n ref_scribble_labels,\n to_fill) # uint8_ fixed."
+ },
+ {
+ "comment": "The code defines three functions: \n1. `train_stage2.py:525-555`: The `return final_scribble_labels` statement concludes the function, returning the final scribble labels after some operations on them.\n2. `load_network`: Loads pretrained weights into a network by matching the keys in the pretrained dictionary with those in the model dictionary and updating the state dict accordingly.\n3. `save_network`: Saves the current state of the network at a specified step to a given directory, creating the directory if it doesn't exist.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":525-555",
+ "content": " return final_scribble_labels\n def load_network(self, net, pretrained_dict):\n # pretrained_dict = pretrained_dict\n model_dict = net.state_dict()\n # 1. filter out unnecessary keys\n pretrained_dict = {\n k: v\n for k, v in pretrained_dict.items() if k in model_dict\n }\n # 2. overwrite entries in the existing state dict\n # for k in model_dict:\n # if k not in pretrained_dict:\n # print(k, 'not in loaded weights.')\n model_dict.update(pretrained_dict)\n net.set_state_dict(model_dict)\n return net\n def save_network(self, net, step):\n save_path = self.save_res_dir\n if not os.path.exists(save_path):\n os.makedirs(save_path)\n save_file = 'save_step_%s.pth' % (step)\n paddle.save(net.state_dict(), os.path.join(save_path, save_file))\n def _adjust_lr(self, optimizer, itr, max_itr):\n now_lr = cfg.TRAIN_LR * (1 - itr / (max_itr + 1))**cfg.TRAIN_POWER\n optimizer._param_groups[0]['lr'] = now_lr"
+ },
+ {
+ "comment": "The code defines a color palette with 81 RGB colors, ranging from black (0, 0, 0) to white (255, 255, 255). Each value represents the color's red, green, and blue components.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":556-572",
+ "content": " return now_lr\n_palette = [\n 0, 0, 0, 128, 0, 0, 0, 128, 0, 128, 128, 0, 0, 0, 128, 128, 0, 128, 0, 128,\n 128, 128, 128, 128, 64, 0, 0, 191, 0, 0, 64, 128, 0, 191, 128, 0, 64, 0,\n 128, 191, 0, 128, 64, 128, 128, 191, 128, 128, 0, 64, 0, 128, 64, 0, 0, 191,\n 0, 128, 191, 0, 0, 64, 128, 128, 64, 128, 22, 22, 22, 23, 23, 23, 24, 24,\n 24, 25, 25, 25, 26, 26, 26, 27, 27, 27, 28, 28, 28, 29, 29, 29, 30, 30, 30,\n 31, 31, 31, 32, 32, 32, 33, 33, 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 37,\n 37, 37, 38, 38, 38, 39, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 43, 43,\n 43, 44, 44, 44, 45, 45, 45, 46, 46, 46, 47, 47, 47, 48, 48, 48, 49, 49, 49,\n 50, 50, 50, 51, 51, 51, 52, 52, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56,\n 56, 56, 57, 57, 57, 58, 58, 58, 59, 59, 59, 60, 60, 60, 61, 61, 61, 62, 62,\n 62, 63, 63, 63, 64, 64, 64, 65, 65, 65, 66, 66, 66, 67, 67, 67, 68, 68, 68,\n 69, 69, 69, 70, 70, 70, 71, 71, 71, 72, 72, 72, 73, 73, 73, 74, 74, 74, 75,\n 75, 75, 76, 76, 76, 77, 77, 77, 78, 78, 78, 79, 79, 79, 80, 80, 80, 81, 81,"
+ },
+ {
+ "comment": "This code represents a list of numbers ranging from 81 to 150.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":573-585",
+ "content": " 81, 82, 82, 82, 83, 83, 83, 84, 84, 84, 85, 85, 85, 86, 86, 86, 87, 87, 87,\n 88, 88, 88, 89, 89, 89, 90, 90, 90, 91, 91, 91, 92, 92, 92, 93, 93, 93, 94,\n 94, 94, 95, 95, 95, 96, 96, 96, 97, 97, 97, 98, 98, 98, 99, 99, 99, 100,\n 100, 100, 101, 101, 101, 102, 102, 102, 103, 103, 103, 104, 104, 104, 105,\n 105, 105, 106, 106, 106, 107, 107, 107, 108, 108, 108, 109, 109, 109, 110,\n 110, 110, 111, 111, 111, 112, 112, 112, 113, 113, 113, 114, 114, 114, 115,\n 115, 115, 116, 116, 116, 117, 117, 117, 118, 118, 118, 119, 119, 119, 120,\n 120, 120, 121, 121, 121, 122, 122, 122, 123, 123, 123, 124, 124, 124, 125,\n 125, 125, 126, 126, 126, 127, 127, 127, 128, 128, 128, 129, 129, 129, 130,\n 130, 130, 131, 131, 131, 132, 132, 132, 133, 133, 133, 134, 134, 134, 135,\n 135, 135, 136, 136, 136, 137, 137, 137, 138, 138, 138, 139, 139, 139, 140,\n 140, 140, 141, 141, 141, 142, 142, 142, 143, 143, 143, 144, 144, 144, 145,\n 145, 145, 146, 146, 146, 147, 147, 147, 148, 148, 148, 149, 149, 149, 150,"
+ },
+ {
+ "comment": "This code appears to be a sequence of numbers, and it is not clear what the purpose or functionality of this specific section of code is without further context.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":586-598",
+ "content": " 150, 150, 151, 151, 151, 152, 152, 152, 153, 153, 153, 154, 154, 154, 155,\n 155, 155, 156, 156, 156, 157, 157, 157, 158, 158, 158, 159, 159, 159, 160,\n 160, 160, 161, 161, 161, 162, 162, 162, 163, 163, 163, 164, 164, 164, 165,\n 165, 165, 166, 166, 166, 167, 167, 167, 168, 168, 168, 169, 169, 169, 170,\n 170, 170, 171, 171, 171, 172, 172, 172, 173, 173, 173, 174, 174, 174, 175,\n 175, 175, 176, 176, 176, 177, 177, 177, 178, 178, 178, 179, 179, 179, 180,\n 180, 180, 181, 181, 181, 182, 182, 182, 183, 183, 183, 184, 184, 184, 185,\n 185, 185, 186, 186, 186, 187, 187, 187, 188, 188, 188, 189, 189, 189, 190,\n 190, 190, 191, 191, 191, 192, 192, 192, 193, 193, 193, 194, 194, 194, 195,\n 195, 195, 196, 196, 196, 197, 197, 197, 198, 198, 198, 199, 199, 199, 200,\n 200, 200, 201, 201, 201, 202, 202, 202, 203, 203, 203, 204, 204, 204, 205,\n 205, 205, 206, 206, 206, 207, 207, 207, 208, 208, 208, 209, 209, 209, 210,\n 210, 210, 211, 211, 211, 212, 212, 212, 213, 213, 213, 214, 214, 214, 215,"
+ },
+ {
+ "comment": "The code snippet initializes a Manager object and calls its train() method. The list of numbers represents the dimensions of an image, potentially used for resizing or preprocessing during training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/Ma-Net/train_stage2.py\":599-611",
+ "content": " 215, 215, 216, 216, 216, 217, 217, 217, 218, 218, 218, 219, 219, 219, 220,\n 220, 220, 221, 221, 221, 222, 222, 222, 223, 223, 223, 224, 224, 224, 225,\n 225, 225, 226, 226, 226, 227, 227, 227, 228, 228, 228, 229, 229, 229, 230,\n 230, 230, 231, 231, 231, 232, 232, 232, 233, 233, 233, 234, 234, 234, 235,\n 235, 235, 236, 236, 236, 237, 237, 237, 238, 238, 238, 239, 239, 239, 240,\n 240, 240, 241, 241, 241, 242, 242, 242, 243, 243, 243, 244, 244, 244, 245,\n 245, 245, 246, 246, 246, 247, 247, 247, 248, 248, 248, 249, 249, 249, 250,\n 250, 250, 251, 251, 251, 252, 252, 252, 253, 253, 253, 254, 254, 254, 255,\n 255, 255\n]\nmanager = Manager()\nmanager.train()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/91c7470d-ffc8-4732-bca4-350b25078914.json b/docs/doc/91c7470d-ffc8-4732-bca4-350b25078914.json
new file mode 100644
index 000000000..c7ead9d4c
--- /dev/null
+++ b/docs/doc/91c7470d-ffc8-4732-bca4-350b25078914.json
@@ -0,0 +1,35 @@
+{
+ "summary": "DistributedShortSampler streamlines distributed data loading, dynamic batch sizes, and GPU support for PaddleVideo's multigrid. It efficiently calculates average batch size and offers sample dropping options.",
+ "details": [
+ {
+ "comment": "The code defines a DistributedShortSampler class which is a sampler for restricting data loading to a subset of the dataset in distributed training. It allows each process to load exclusive subsets by passing the DistributedBatchSampler as a DataLoader sampler and supports dynamic batch size changes following short cycle schedules. The class takes in a dataset, batch_sizes list, and optionally num_replicas (process number in distributed training).",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/short_sampler.py\":0-27",
+ "content": "from __future__ import print_function\nfrom __future__ import division\nimport numpy as np\nimport math\nimport paddle\n__all__ = [\"DistributedShortSampler\"]\nclass DistributedShortSampler(paddle.io.BatchSampler):\n \"\"\"Sampler that restricts data loading to a subset of the dataset.\n In such case, each process can pass a DistributedBatchSampler instance\n as a DataLoader sampler, and load a subset of the original dataset that\n is exclusive to it.\n .. note::\n Batch size is dynamic changed following short cycle schedule.\n Args:\n dataset(paddle.io.Dataset): this could be a `paddle.io.Dataset` implement\n or other python object which implemented\n `__len__` for BatchSampler to get sample\n number of data source.\n batch_sizes(list): batch size list of one cycle.\n num_replicas(int, optional): porcess number in distributed training.\n If :attr:`num_replicas` is None, :attr:`num_replicas` will be\n retrieved from :code:`paddle.fluid.dygraph.parallel.ParallenEnv`."
+ },
+ {
+ "comment": "The `__init__` method initializes an instance of the class with a dataset, batch sizes, number of replicas (optional), rank (optional), whether to shuffle indices (optional), and whether to drop last incomplete batch (optional). The batch_sizes should be positive integers. The method performs assertions on the inputs to ensure validity.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/short_sampler.py\":28-50",
+ "content": " Default None.\n rank(int, optional): the rank of the current process among :attr:`num_replicas`\n processes. If :attr:`rank` is None, :attr:`rank` is retrieved from\n :code:`paddle.fluid.dygraph.parallel.ParallenEnv`. Default None.\n shuffle(bool): whther to shuffle indices order before genrating\n batch indices. Default False.\n drop_last(bool): whether drop the last incomplete batch dataset size\n is not divisible by the batch size. Default False\n \"\"\"\n def __init__(self,\n dataset,\n batch_sizes,\n num_replicas=None,\n rank=None,\n shuffle=False,\n drop_last=False):\n self.dataset = dataset\n assert any(isinstance(batch_size, int) and batch_size > 0 for batch_size in batch_sizes), \\\n \"batch_size should be a positive integer\"\n self.batch_sizes = batch_sizes\n self.len_batch_sizes = len(self.batch_sizes)\n assert isinstance(shuffle, bool), \\"
+ },
+ {
+ "comment": "The code initializes a MultigridSampler object, which manages the sampling of data across multiple ranks in distributed training. It checks for valid input values (boolean for shuffle and drop_last) and ensures positive integer for num_replicas. It determines the number of ranks and local rank based on provided values or environment. The total number of samples is calculated based on the dataset size and number of ranks, and an array of indices is created.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/short_sampler.py\":51-78",
+ "content": " \"shuffle should be a boolean value\"\n self.shuffle = shuffle\n assert isinstance(drop_last, bool), \\\n \"drop_last should be a boolean number\"\n if num_replicas is not None:\n assert isinstance(num_replicas, int) and num_replicas > 0, \\\n \"num_replicas should be a positive integer\"\n self.nranks = num_replicas\n else:\n self.nranks = paddle.distributed.ParallelEnv().nranks\n if rank is not None:\n assert isinstance(rank, int) and rank >= 0, \\\n \"rank should be a non-negative integer\"\n self.local_rank = rank\n else:\n self.local_rank = paddle.distributed.ParallelEnv().local_rank\n self.drop_last = drop_last\n self.epoch = 0\n self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.nranks))\n self.total_size = self.num_samples * self.nranks\n def __iter__(self):\n num_samples = len(self.dataset)\n indices = np.arange(num_samples).tolist()"
+ },
+ {
+ "comment": "This code ensures that the number of samples selected is equal to the total size, and then subsamples them by batch sizes. It handles the last batch with potentially fewer samples due to modulo operations and shuffles the indices if desired.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/short_sampler.py\":79-101",
+ "content": " indices += indices[:(self.total_size -\n len(indices))] #completion last iter\n assert len(indices) == self.total_size\n if self.shuffle:\n np.random.RandomState(self.epoch).shuffle(indices)\n self.epoch += 1\n # subsample\n def _get_indices_by_batch_size(indices):\n total_batch_size = sum(self.batch_sizes)\n subsampled_indices = []\n last_batch_size = self.total_size % (\n total_batch_size * self.nranks) #number samples of last batch\n assert last_batch_size % self.nranks == 0\n last_local_batch_size = last_batch_size // self.nranks\n for i in range(self.local_rank * total_batch_size,\n len(indices) - last_batch_size,\n total_batch_size * self.nranks):\n subsampled_indices.extend(indices[i:i + total_batch_size])\n indices = indices[len(indices) - last_batch_size:]\n subsampled_indices.extend("
+ },
+ {
+ "comment": "This code is responsible for creating a sampler that supports dynamic batch sizes. It first sub-samples the input indices based on the local rank and local batch size. Then, it handles cases with multiple GPUs (ranks > 1), dividing the indices into batches of uniform size. Finally, it yields these batches until all samples have been used, or if the drop_last flag is set to False, it yields remaining samples even if they don't form a full batch. The average batch size is also calculated and stored in the class variable avg_batch_size.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/short_sampler.py\":102-129",
+ "content": " indices[self.local_rank *\n last_local_batch_size:(self.local_rank + 1) *\n last_local_batch_size])\n return subsampled_indices\n if self.nranks > 1:\n indices = _get_indices_by_batch_size(indices)\n assert len(indices) == self.num_samples #index length in each card\n _sample_iter = iter(indices)\n batch_indices = []\n counter = 0\n batch_size = self.batch_sizes[0]\n for idx in _sample_iter:\n batch_indices.append(\n (idx, counter %\n self.len_batch_sizes)) #to be used in dataloader get_item\n if len(batch_indices) == batch_size:\n yield batch_indices\n counter += 1\n batch_size = self.batch_sizes[counter % self.len_batch_sizes]\n batch_indices = []\n if not self.drop_last and len(batch_indices) > 0:\n yield batch_indices\n def __len__(self):\n avg_batch_size = sum(self.batch_sizes) / float(self.len_batch_sizes)"
+ },
+ {
+ "comment": "This code defines a class for a sampler that can be used with PaddleVideo's multigrid. It calculates the number of samples to return based on batch size and either rounds down or up depending on whether drop_last is set. The set_epoch method sets the epoch number and, when shuffle is True, uses it as seeds for random numbers. This can result in the same ordering being yielded at all epochs if the same number is set each time.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/short_sampler.py\":130-145",
+ "content": " if self.drop_last:\n return int(np.floor(self.num_samples / avg_batch_size))\n else:\n return int(np.ceil(self.num_samples / avg_batch_size))\n def set_epoch(self, epoch):\n \"\"\"\n Sets the epoch number. When :attr:`shuffle=True`, this number is used\n as seeds of random numbers. By default, users may not set this, all\n replicas (workers) use a different random ordering for each epoch.\n If set same number at each epoch, this sampler will yield the same\n ordering at all epoches.\n Arguments:\n epoch (int): Epoch number.\n \"\"\"\n self.epoch = epoch"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9207f54d-f34e-430b-b1ba-3c78b87b0be4.json b/docs/doc/9207f54d-f34e-430b-b1ba-3c78b87b0be4.json
new file mode 100644
index 000000000..5f57d69ec
--- /dev/null
+++ b/docs/doc/9207f54d-f34e-430b-b1ba-3c78b87b0be4.json
@@ -0,0 +1,20 @@
+{
+ "summary": "This code defines a PaddlePaddle base class for semi-Video Object Segmentation, with methods for training, validating, testing, and inference, determined by the mode argument.",
+ "details": [
+ {
+ "comment": "This code is a base class for semi-Video Object Segmentation in PaddlePaddle. It requires subclasses to overwrite training, validation, and testing forward methods. The class also includes backbone modules for feature extraction and head modules for processing features, with specified loss functions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/base.py\":1-29",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom abc import abstractmethod\nfrom ... import builder\nimport paddle.nn as nn\nclass BaseSegment(nn.Layer):\n \"\"\"Base class for semi-Video Object Segmentation.\n All subclass should overwrite:\n - Methods:``train_step``, supporting to forward when training.\n - Methods:``valid_step``, supporting to forward when validating.\n - Methods:``test_step``, supporting to forward when testing.\n Args:\n backbone (dict): Backbone modules to extract feature.\n head (dict): Head to process feature.\n loss(dict): Loss function."
+ },
+ {
+ "comment": "The code initializes a model by building the backbone, head, and loss components. In the forward function, it defines how the model processes input data for training or inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/base.py\":30-58",
+ "content": " \"\"\"\n def __init__(self, backbone=None, head=None, loss=None):\n super().__init__()\n if backbone != None:\n self.backbone = builder.build_backbone(backbone)\n if hasattr(self.backbone, 'init_weights'):\n self.backbone.init_weights()\n else:\n self.backbone = None\n if head != None:\n self.head_name = head.name\n if head.name == 'IntVOS':\n head.update({'feature_extracter': self.backbone})\n self.head = builder.build_head(head)\n else:\n self.head = builder.build_head(head)\n if hasattr(self.head, 'init_weights'):\n self.head.init_weights()\n else:\n self.head = None\n if loss != None:\n self.loss = builder.build_loss(loss)\n else:\n self.loss = None\n def forward(self, data_batch, mode='infer', **kwargs):\n \"\"\"\n 1. Define how the model is going to run, from input to output.\n 2. Console of train, valid, test or infer step"
+ },
+ {
+ "comment": "This code defines a class with different step methods for training, validating, testing, and inference. The mode argument determines which method to execute based on the current task. Each step method is marked as an abstractmethod requiring subclass implementation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/base.py\":59-94",
+ "content": " 3. Set mode='infer' is used for saving inference model, refer to tools/export_model.py\n \"\"\"\n if mode == 'train':\n return self.train_step(data_batch, **kwargs)\n elif mode == 'valid':\n return self.val_step(data_batch, **kwargs)\n elif mode == 'test':\n return self.test_step(data_batch, **kwargs)\n elif mode == 'infer':\n return self.infer_step(data_batch, **kwargs)\n else:\n raise NotImplementedError\n @abstractmethod\n def train_step(self, data_batch, **kwargs):\n \"\"\"Training step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def val_step(self, data_batch, **kwargs):\n \"\"\"Validating step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def test_step(self, data_batch, **kwargs):\n \"\"\"Test step.\n \"\"\"\n raise NotImplementedError\n @abstractmethod\n def infer_step(self, data_batch, **kwargs):\n \"\"\"Infer step.\n \"\"\"\n raise NotImplementedError"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/929ed878-ca6a-465e-bfb5-f439a1a19297.json b/docs/doc/929ed878-ca6a-465e-bfb5-f439a1a19297.json
new file mode 100644
index 000000000..48fc19509
--- /dev/null
+++ b/docs/doc/929ed878-ca6a-465e-bfb5-f439a1a19297.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The script uses PyQt5 to create a video processing GUI with functions for initializing variables, opening file dialogs, handling combo box indexing, and pen color changes. It also includes a `open_frame` function that updates the progress slider and stops the video at the last frame.",
+ "details": [
+ {
+ "comment": "This code is the initial part of a Python script for building a GUI (Graphical User Interface) application using PyQt5 library. It defines a class called BuildGUI that inherits from QMainWindow and Ui_MainWindow, which likely contains the layout and design elements of the GUI. The __init__ method sets up some initial variables such as the selected video path and save path for results. The infer method is meant to start an inference process.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/build_gui.py\":0-35",
+ "content": "# Author: Acer Zhang\n# Datetime:2022/1/11 \n# Copyright belongs to the author.\n# Please indicate the source for reprinting.\nimport json\nimport os\nimport numpy as np\nfrom PIL import Image\nfrom PyQt5 import QtCore, QtWidgets\nfrom PyQt5.QtGui import *\nfrom PyQt5.QtWidgets import *\nfrom PyQt5.QtCore import *\nimport cv2\nfrom EIVideo.api import json2frame, png2json, load_video\nfrom EIVideo.main import main\n# ToDo To AP-kai: \u8fd9\u662f\u5b9a\u4e49\u524d\u7aef\u4e34\u65f6\u4fdd\u5b58\u7528\u4e8e\u63a8\u7406\u7684json\u7684\u5730\u70b9\u4e4b\u7c7b\u7684\uff0c\u56e0\u4e3a\u662f\u56fa\u5b9a\u7684\uff0c\u6240\u4ee5\u58f0\u660e\u4e3a\u5168\u5c40\u5e38\u91cf\u662f\u6700\u597d\u7684\nfrom EIVideo import TEMP_JSON_SAVE_PATH, TEMP_IMG_SAVE_PATH, TEMP_JSON_FINAL_PATH\nfrom QEIVideo.gui.ui_main_window import Ui_MainWindow\nclass BuildGUI(QMainWindow, Ui_MainWindow):\n def __init__(self):\n super(BuildGUI, self).__init__()\n # ToDo To AP-kai: \u8fd9\u91cc\u5b9a\u4e49\u5f53\u524d\u9009\u62e9\u7684\u89c6\u9891\u8def\u5f84\u7684\u5360\u4f4d\u7b26\uff0c\u76f8\u5f53\u4e8e\u5168\u5c40\u53d8\u91cf\n self.select_video_path = None\n # ToDo To AP-kai: \u672a\u6765\u4e3a\u7528\u6237\u63d0\u4f9b\u4e2a\u4fdd\u5b58\u8def\u5f84\u7684\u5165\u53e3\u54c8\uff0c\u8fd9\u91cc\u5148\u968f\u610f\u5b9a\u4e49\u4e86\u4e2a\u8def\u5f84\n self.save_path = \"./result\"\n os.makedirs(self.save_path, exist_ok=True)\n self.setupUi(self)\n def infer(self):\n self.label.setText(\"Start infer\")"
+ },
+ {
+ "comment": "The code snippet sets the progress bar value, saves an image, prints frame numbers, calls a main function to perform inference, loads JSON frames, opens a frame, and updates a label when play button is clicked.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/build_gui.py\":36-58",
+ "content": " self.progressBar.setProperty(\"value\", 0)\n image = self.paintBoard.get_content_as_q_image()\n image.save(TEMP_IMG_SAVE_PATH)\n print(self.slider_frame_num)\n self.progressBar.setProperty(\"value\", 25)\n # ToDo To AP-kai:\u76f8\u540c\u7684\u6587\u4ef6\u8def\u5f84\uff0c\u76f4\u63a5\u5b9a\u4e49\u4e00\u4e2a\u5e38\u91cf\u5c31\u597d\n png2json(TEMP_IMG_SAVE_PATH, self.slider_frame_num, TEMP_JSON_SAVE_PATH)\n self.progressBar.setProperty(\"value\", 50)\n # ToDo To AP-kai:\u6253\u5370\u7684\u4fe1\u606f\uff0c\u9700\u8981\u6ce8\u610f\u9996\u5b57\u6bcd\u5927\u5199\n # ToDo To AP-kai: \u6b64\u5904\u4f20\u5165\u4fdd\u5b58\u8def\u5f84\u4ee5\u53ca\u5f53\u524d\u9009\u62e9\u7684\u89c6\u9891\u8def\u5f84\uff0c\u6700\u540e\u4f1a\u5728manet_stage1.py\u91cc\u901a\u8fc7cfg\u6765\u4f20\u5165\n out = main(video_path=self.select_video_path, save_path=self.save_path)\n print('Infer ok')\n self.progressBar.setProperty(\"value\", 75)\n self.all_frames = json2frame(TEMP_JSON_FINAL_PATH)\n print(\"Success get submit_masks\")\n self.open_frame()\n self.progressBar.setProperty(\"value\", 100)\n self.label.setText(\"Infer succeed\")\n def btn_func(self, btn):\n if btn == self.playbtn:\n self.label.setText(\"Play video\")\n if self.progress_slider.value() == self.cap.get(7) - 1:"
+ },
+ {
+ "comment": "The code above contains three elif conditions for button press events. If the self.pushButton_1 is pressed, it initializes variables and starts a timer to update the video frame. If self.pushButton_2 is pressed, it stops the video and sets the label text to \"Stop video\". If self.pushButton_4 is pressed, it opens a file dialog for choosing a video file, and if a file is chosen, it prints the selected video file path. The current code is checking if there is a non-empty selected video file path after the file dialog is closed.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/build_gui.py\":59-77",
+ "content": " self.slider_frame_num = 0\n self.progress_slider.setValue(self.slider_frame_num)\n self.time_label.setText('{}/{}'.format(self.slider_frame_num, self.cap.get(7)))\n self.timer_camera = QTimer() # \u5b9a\u4e49\u5b9a\u65f6\u5668\n self.timer_camera.start(1000 / self.cap.get(cv2.CAP_PROP_FPS))\n self.slider_frame_num = self.progress_slider.value()\n self.timer_camera.timeout.connect(self.open_frame)\n elif btn == self.pushButton_2:\n self.label.setText(\"Stop video\")\n self.slot_stop()\n elif btn == self.pushButton_4:\n self.label.setText(\"Choose video\")\n self.select_video_path, _ = QFileDialog.getOpenFileName(self.frame, \"Open\", \"\", \"*.mp4;;All Files(*)\")\n print(\"-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-\")\n print(\"Select video file path:\\t\" + self.select_video_path)\n # ToDo To AP-kai:\u4e0b\u65ad\u70b9\u6765\u770b\u4e00\u4e0b\uff0c\u5982\u679c\u4e0d\u9009\u62e9\u7684\u65f6\u5019\u8fd4\u56de\u503c\u662f\u4ec0\u4e48\u6837\u7684\uff0c\u7136\u540e\u518d\u505a\u5224\u65ad\uff0c\u76ee\u524d\u8fd9\u4e2aif\u6ca1\u6709\u751f\u6548\n if self.select_video_path != \"\":"
+ },
+ {
+ "comment": "The code snippet creates a GUI for video processing. It sets up a VideoCapture object, stores all frames, and initializes a progress slider with the total number of frames. The Eraser button toggles between EraserMode on/off in the paintBoard. The fill_color_list function populates a color combo box with predefined colors, including black at a specific index.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/build_gui.py\":78-106",
+ "content": " self.cap = cv2.VideoCapture(self.select_video_path)\n # \u5b58\u6240\u6709frame\n self.save_temp_frame()\n print(\"save temp frame done\")\n self.progress_slider.setRange(0, self.cap.get(cv2.CAP_PROP_FRAME_COUNT))\n self.slider_frame_num = 0\n self.open_frame()\n # ToDo To AP-kai: \u672a\u6765\u8fd9\u4e2a\u5730\u65b9\u589e\u52a0\u63d0\u793a\u6846\uff0c\u544a\u8bc9\u4ed6\u6ca1\u6709\u9009\u62e9\u6587\u4ef6\n def on_cbtn_eraser_clicked(self):\n self.label.setText(\"Eraser On\")\n if self.cbtn_Eraser.isChecked():\n self.paintBoard.EraserMode = True # \u8fdb\u5165\u6a61\u76ae\u64e6\u6a21\u5f0f\n else:\n self.paintBoard.EraserMode = False # \u9000\u51fa\u6a61\u76ae\u64e6\u6a21\u5f0f\n def fill_color_list(self, combo_box):\n index_black = 0\n index = 0\n for color in self.colorList:\n if color == \"black\":\n index_black = index\n index += 1\n pix = QPixmap(70, 20)\n pix.fill(QColor(color))\n combo_box.addItem(QIcon(pix), None)\n combo_box.setIconSize(QSize(70, 20))\n combo_box.setSizeAdjustPolicy(QComboBox.AdjustToContents)"
+ },
+ {
+ "comment": "This code defines several functions for a GUI application. It sets the current index of a combo box, handles pen color changes by updating the paintBoard's color, updates the video position based on progress slider input, saves a temporary frame from a video file, and stops the timer if it is running.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/build_gui.py\":108-134",
+ "content": " combo_box.setCurrentIndex(index_black)\n def on_pen_color_change(self):\n self.label.setText(\"Change pen color\")\n color_index = self.comboBox_penColor.currentIndex()\n color_str = self.colorList[color_index]\n self.paintBoard.change_pen_color(color_str)\n # \u62d6\u62fd\u8fdb\u5ea6\u6761\n def update_video_position_func(self):\n self.label.setText(\"Change slider position\")\n self.slider_frame_num = self.progress_slider.value()\n self.slot_stop()\n self.open_frame()\n self.progress_slider.setValue(self.slider_frame_num)\n self.time_label.setText('{}/{}'.format(self.slider_frame_num, self.cap.get(7)))\n def save_temp_frame(self):\n _, self.all_frames = load_video(self.select_video_path, 480)\n def slot_stop(self):\n if self.cap != []:\n self.timer_camera.stop() # \u505c\u6b62\u8ba1\u65f6\u5668\n else:\n # ToDo To AP-kai: QMessageBox.warning\u6ca1\u6709\u8fd4\u56de\u503c\uff0c\u8fd9\u91cc\u6211\u628aWarming = QMessageBox.warning\u7684Warming\u5220\u53bb\u4e86\n QMessageBox.warning(self, \"Warming\", \"Push the left upper corner button to Quit.\","
+ },
+ {
+ "comment": "This code defines a function `open_frame` which updates the progress slider, displays the current frame using QImage and QPixmap, increments the slider value, sets the time label, and stops the video if at the last frame.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/QEIVideo/build_gui.py\":135-150",
+ "content": " QMessageBox.Yes)\n def open_frame(self):\n self.progress_slider.setValue(self.slider_frame_num)\n self.slider_frame_num = self.progress_slider.value()\n self.frame = self.all_frames[self.slider_frame_num]\n frame = self.frame\n height, width, bytes_per_component = frame.shape\n bytes_per_line = bytes_per_component * width\n q_image = QImage(frame.data, width, height, bytes_per_line,\n QImage.Format_RGB888).scaled(self.picturelabel.width(), self.picturelabel.height())\n self.picturelabel.setPixmap(QPixmap.fromImage(q_image))\n self.slider_frame_num = self.slider_frame_num + 1\n self.time_label.setText('{}/{}'.format(self.slider_frame_num, self.cap.get(7)))\n if self.progress_slider.value() == self.cap.get(7) - 1:\n self.slot_stop()"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/92ab2251-9095-46cf-9122-551bb4a0a6c6.json b/docs/doc/92ab2251-9095-46cf-9122-551bb4a0a6c6.json
new file mode 100644
index 000000000..1bbcdb6b2
--- /dev/null
+++ b/docs/doc/92ab2251-9095-46cf-9122-551bb4a0a6c6.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The code extracts audio features for Table Tennis prediction, using spectrogram and Mel scale transformation, and reads WAV files with VGG-16 model for MFCC and STFT feature extraction.",
+ "details": [
+ {
+ "comment": "This code is for audio feature extraction in TableTennis application. It defines functions `frame`, `periodic_hann` and `stft_magnitude`. The `frame` function resizes the data array into frames with specified window length and hop length. The `periodic_hann` function generates a periodic Hann window for the STFT operation. Finally, `stft_magnitude` calculates the magnitude of the Short-Time Fourier Transform (STFT) of an audio signal.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":0-38",
+ "content": "\"\"\"\naudio feature extract\n\"\"\"\n# coding: utf-8\nimport os\nimport numpy as np\nimport pickle\nimport mfcc.vgg_params as vgg_params\nimport sys\ndef frame(data, window_length, hop_length):\n \"\"\"\n frame\n \"\"\"\n num_samples = data.shape[0]\n #print(\"window_length , hop_length\", window_length, hop_length)\n #print(\"num_sample = \", num_samples)\n num_frames = 1 + int(np.floor((num_samples - window_length) / hop_length))\n #print(\" num_frames = \", num_frames)\n shape = (num_frames, window_length) + data.shape[1:]\n #print(\" shape = \", shape)\n strides = (data.strides[0] * hop_length, ) + data.strides\n #print(\"data.strides = \", data.strides)\n #print(\"strides = \", strides)\n return np.lib.stride_tricks.as_strided(data, shape=shape, strides=strides)\ndef periodic_hann(window_length):\n \"\"\"\n periodic_hann\n \"\"\"\n return 0.5 - (0.5 *\n np.cos(2 * np.pi / window_length * np.arange(window_length)))\ndef stft_magnitude(signal, fft_length, hop_length=None, window_length=None):\n \"\"\"\n stft_magnitude"
+ },
+ {
+ "comment": "The code defines functions for feature extraction and conversion of audio signals. The \"hertz_to_mel\" function converts frequencies from Hertz to Mel scale, which is used in psychoacoustics. The \"spectrogram_to_mel_matrix\" function creates a mel-frequency cepstral coefficients (MFCC) matrix for audio spectrograms. It checks for lower and upper frequency edge validity and calculates Mel frequencies based on the provided parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":39-69",
+ "content": " \"\"\"\n frames = frame(signal, window_length, hop_length)\n window = periodic_hann(window_length)\n windowed_frames = frames * window\n return np.abs(np.fft.rfft(windowed_frames, int(fft_length)))\n_MEL_BREAK_FREQUENCY_HERTZ = 700.0\n_MEL_HIGH_FREQUENCY_Q = 1127.0\ndef hertz_to_mel(frequencies_hertz):\n \"\"\"\n hertz_to_mel\n \"\"\"\n return _MEL_HIGH_FREQUENCY_Q * np.log(1.0 + (frequencies_hertz /\n _MEL_BREAK_FREQUENCY_HERTZ))\ndef spectrogram_to_mel_matrix(num_mel_bins=20,\n num_spectrogram_bins=129,\n audio_sample_rate=8000,\n lower_edge_hertz=125.0,\n upper_edge_hertz=3800.0):\n \"\"\"\n spectrogram_to_mel_matrix\n \"\"\"\n nyquist_hertz = audio_sample_rate / 2.\n if lower_edge_hertz >= upper_edge_hertz:\n raise ValueError(\"lower_edge_hertz %.1f >= upper_edge_hertz %.1f\" %\n (lower_edge_hertz, upper_edge_hertz))"
+ },
+ {
+ "comment": "This code is performing Mel frequency cepstral coefficients (MFCC) feature extraction on audio data. It creates spectrogram bins in Hz, converts them to the mel scale, defines mel band edges, and computes the corresponding mel weights matrix. The function returns this matrix after setting the first row to zero. This process is commonly used for speech processing and analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":70-90",
+ "content": " spectrogram_bins_hertz = np.linspace(0.0, nyquist_hertz,\n num_spectrogram_bins)\n spectrogram_bins_mel = hertz_to_mel(spectrogram_bins_hertz)\n band_edges_mel = np.linspace(hertz_to_mel(lower_edge_hertz),\n hertz_to_mel(upper_edge_hertz),\n num_mel_bins + 2)\n mel_weights_matrix = np.empty((num_spectrogram_bins, num_mel_bins))\n for i in range(num_mel_bins):\n lower_edge_mel, center_mel, upper_edge_mel = band_edges_mel[i:i + 3]\n lower_slope = ((spectrogram_bins_mel - lower_edge_mel) /\n (center_mel - lower_edge_mel))\n upper_slope = ((upper_edge_mel - spectrogram_bins_mel) /\n (upper_edge_mel - center_mel))\n mel_weights_matrix[:,\n i] = np.maximum(0.0,\n np.minimum(lower_slope, upper_slope))\n mel_weights_matrix[0, :] = 0.0\n return mel_weights_matrix\ndef log_mel_spectrogram(data,"
+ },
+ {
+ "comment": "This code defines a function called `log_mel_spectrogram` which takes audio data, sample rate, and optional keyword arguments. It calculates window length in samples, hop length in samples, FFT length, and then uses the Short-Time Fourier Transform (STFT) to generate a spectrogram from the input audio data. The resulting spectrogram is stored in the `spectrogram` variable and its shape is printed for debugging or reference purposes.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":91-111",
+ "content": " audio_sample_rate=8000,\n log_offset=0.0,\n window_length_secs=0.025,\n hop_length_secs=0.010,\n **kwargs):\n \"\"\"\n log_mel_spectrogram\n \"\"\"\n window_length_samples = int(round(audio_sample_rate * window_length_secs))\n #print(\"audio_sample_rate = \", audio_sample_rate)\n #print(\"window_length_secs = \", window_length_secs)\n #print(\"window_length_sample \", window_length_samples)\n hop_length_samples = int(round(audio_sample_rate * hop_length_secs))\n #print(\"hop_length_samples \", hop_length_samples)\n fft_length = 2**int(np.ceil(np.log(window_length_samples) / np.log(2.0)))\n #print(\" fft_lengt = \", fft_length)\n spectrogram = stft_magnitude(data,\n fft_length=fft_length,\n hop_length=hop_length_samples,\n window_length=window_length_samples)\n #print(\" spectrogram.shape = \", spectrogram.shape)"
+ },
+ {
+ "comment": "Function `spectrogram_to_mel_matrix` converts the spectrogram to Mel scale. Code calculates Mel spectrogram by taking dot product of spectrogram with `spectrogram_to_mel_matrix`. The result is then log transformed to avoid numerical underflow and returned.\nThe function `wav_to_example` takes wav file data, validates sample type, pads zeros to achieve desired window length, scales the wav data to range -1 to 1 by dividing by 32768.0. It is used for audio feature extraction in TableTennis application of PaddleVideo.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":112-136",
+ "content": " mel_spectrogram = np.dot(\n spectrogram,\n spectrogram_to_mel_matrix(num_spectrogram_bins=spectrogram.shape[1],\n audio_sample_rate=audio_sample_rate,\n **kwargs))\n return np.log(mel_spectrogram + log_offset)\ndef wav_to_example(wav_data, sample_rate):\n \"\"\"\n wav_to_example\n \"\"\"\n #sample_rate, wav_data = wavfile.read(wav_file)\n assert wav_data.dtype == np.int16, 'Bad sample type: %r' % wav_data.dtype\n #wav_data = wav_data[:16000*30]\n #print(\" wav_data \", wav_data.shape)\n #print(\" wav_data \", wav_data.shape)\n pad_zero_num = int(sample_rate * (vgg_params.STFT_WINDOW_LENGTH_SECONDS -\n vgg_params.STFT_HOP_LENGTH_SECONDS))\n wav_data_extend = np.hstack((wav_data, np.zeros(pad_zero_num)))\n wav_data = wav_data_extend\n #print(\" wav_data \", wav_data.shape)\n wav_data = wav_data / 32768.0 # Convert to [-1.0, +1.0]\n #print(\" wav_data after convert to -1 1\", wav_data)"
+ },
+ {
+ "comment": "This code extracts audio features for Table Tennis prediction. It first reshapes and resamples the input wav_data if necessary, then calculates log mel spectrogram from wav_data using given parameters. Finally, it frames these features into examples with a specific window length.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":137-157",
+ "content": " #if wav_data.shape[0] > max_second * sample_rate:\n # wav_data = wav_data[:max_second * sample_rate, :]\n if len(wav_data.shape) > 1:\n wav_data = np.mean(wav_data, axis=1)\n #print(\" wav_data after mean\", wav_data.shape, len(wav_data.shape), wav_data)\n # Resample to the rate assumed by vgg.\n #if sample_rate != vgg_params.SAMPLE_RATE:\n # wav_data = resampy.resample(wav_data, sample_rate, vgg_params.SAMPLE_RATE)\n log_mel = log_mel_spectrogram(\n wav_data,\n audio_sample_rate=vgg_params.SAMPLE_RATE,\n log_offset=vgg_params.LOG_OFFSET,\n window_length_secs=vgg_params.STFT_WINDOW_LENGTH_SECONDS,\n hop_length_secs=vgg_params.STFT_HOP_LENGTH_SECONDS,\n num_mel_bins=vgg_params.NUM_MEL_BINS,\n lower_edge_hertz=vgg_params.MEL_MIN_HZ,\n upper_edge_hertz=vgg_params.MEL_MAX_HZ)\n # Frame features into examples.\n features_sample_rate = 1.0 / vgg_params.STFT_HOP_LENGTH_SECONDS\n example_window_length = int(\n round(vgg_params.EXAMPLE_WINDOW_SECONDS * features_sample_rate))"
+ },
+ {
+ "comment": "This code extracts audio features from a WAV file using the VGG-16 model, specifically focusing on MFCC (Mel Frequency Cepstral Coefficients) and STFT (Short-Time Fourier Transform). The code also defines a function to convert PCM data into examples and another to extract MFCC features. Lastly, it demonstrates how to use the code by reading a WAV file and printing its shape.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/TableTennis/predict/action_detect/mfcc/feature_extractor.py\":159-182",
+ "content": " example_hop_length = int(\n round(vgg_params.EXAMPLE_HOP_SECONDS * features_sample_rate))\n log_mel_examples = frame(log_mel,\n window_length=example_window_length,\n hop_length=example_hop_length)\n return log_mel_examples\ndef extract_pcm(pcm_file, sample_rate):\n with open(pcm_file, \"rb\") as f:\n pcm_data = f.read()\n audio_data = np.fromstring(pcm_data, dtype=np.int16)\n examples = wav_to_example(audio_data, sample_rate)\n return examples\nif __name__ == \"__main__\":\n wav_file = sys.argv[1]\n print(\"wav_file = \", wav_file)\n with open(wav_file, \"rb\") as f:\n pcm_data = f.read()\n audio_data = np.fromstring(pcm_data, dtype=np.int16)\n examples_batch = wav_to_example(audio_data, 16000)\n print(\"examples_batch.shape\", examples_batch.shape)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/92c95f7f-34ab-4b71-809d-39d9cb2e56df.json b/docs/doc/92c95f7f-34ab-4b71-809d-39d9cb2e56df.json
new file mode 100644
index 000000000..f5931ebbc
--- /dev/null
+++ b/docs/doc/92c95f7f-34ab-4b71-809d-39d9cb2e56df.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code snippet contains copyright and license information, imports necessary modules from the PaddlePaddle framework, and defines two classes 'BaseSegment' and 'Manet'. It also specifies that these are the components included in the current module.",
+ "details": [
+ {
+ "comment": "This code snippet contains copyright and license information, imports necessary modules from the PaddlePaddle framework, and defines two classes 'BaseSegment' and 'Manet'. It also specifies that these are the components included in the current module.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/framework/segment/__init__.py\":0-18",
+ "content": "# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nfrom .base import BaseSegment\nfrom .manet_stage1 import Manet\n__all__ = [\n 'BaseSegment',\n 'Manet',\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/92e86a04-a8ed-4bb9-b909-a920495c6ac7.json b/docs/doc/92e86a04-a8ed-4bb9-b909-a920495c6ac7.json
new file mode 100644
index 000000000..c733e658b
--- /dev/null
+++ b/docs/doc/92e86a04-a8ed-4bb9-b909-a920495c6ac7.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code defines the Compose class for video transformation pipelines, composing multiple pipeline elements and handling temporary list-type parameters while including a workaround for old format config files.",
+ "details": [
+ {
+ "comment": "This code defines the Compose class, which composes multiple pipelines (decode func, sample func, and transforms) together. It registers the class in the PIPELINES registry. The code also handles temporary list-type configuration parameters for flexibility.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/compose.py\":0-32",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom collections.abc import Sequence\nfrom ..registry import PIPELINES\nimport traceback\nfrom ...utils import build\nfrom ...utils import get_logger\n@PIPELINES.register()\nclass Compose(object):\n \"\"\"\n Composes several pipelines(include decode func, sample func, and transforms) together.\n Note: To deal with ```list``` type cfg temporaray, like:\n transform:\n - Crop: # A list\n attribute: 10"
+ },
+ {
+ "comment": "This code is creating a Compose class which takes a list of transforms and composes them sequentially. It checks if the input is in the correct format, builds each transform using the build function from PIPELINES, and stores them in a list. The code also includes a workaround for handling old format config files that may have inconsistent key-value pairs in their lists.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/compose.py\":33-60",
+ "content": " - Resize: # A list\n attribute: 20\n every key of list will pass as the key name to build a module.\n XXX: will be improved in the future.\n Args:\n pipelines (list): List of transforms to compose.\n Returns:\n A compose object which is callable, __call__ for this Compose\n object will call each given :attr:`transforms` sequencely.\n \"\"\"\n def __init__(self, pipelines):\n #assert isinstance(pipelines, Sequence)\n self.pipelines = []\n for p in pipelines.values():\n if isinstance(p, dict):\n p = build(p, PIPELINES)\n self.pipelines.append(p)\n elif isinstance(p, list):\n for t in p:\n #XXX: to deal with old format cfg, ugly code here!\n temp_dict = dict(name=list(t.keys())[0])\n for all_sub_t in t.values():\n if all_sub_t is not None:\n temp_dict.update(all_sub_t) \n t = build(temp_dict, PIPELINES)"
+ },
+ {
+ "comment": "This code is defining a class for video transformation pipelines. It appends callable functions or dictionaries to the pipeline list and has a __call__ method that applies each pipeline operation to data in sequence, handling exceptions and logging failures if they occur.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/loader/pipelines/compose.py\":61-78",
+ "content": " self.pipelines.append(t)\n elif callable(p):\n self.pipelines.append(p)\n else:\n raise TypeError('pipelines must be callable or a dict,'\n 'but got {type(p)}')\n def __call__(self, data):\n \"\"\"call\"\"\"\n for p in self.pipelines:\n try:\n data = p(data)\n except Exception as e:\n stack_info = traceback.format_exc()\n logger = get_logger(\"paddlevideo\")\n logger.info(\"fail to perform transform [{}] with error: \"\n \"{} and stack:\\n{}\".format(p, e, str(stack_info)))\n raise e\n return data"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/930c71be-937d-41c7-b08e-fa16358f3a52.json b/docs/doc/930c71be-937d-41c7-b08e-fa16358f3a52.json
new file mode 100644
index 000000000..113a1e8d7
--- /dev/null
+++ b/docs/doc/930c71be-937d-41c7-b08e-fa16358f3a52.json
@@ -0,0 +1,70 @@
+{
+ "summary": "This code imports libraries, defines a ResNet-TSN model with basic and bottleneck blocks in PaddlePaddle, initializes weights for training, and outputs results.",
+ "details": [
+ {
+ "comment": "This code is for importing necessary libraries, defining a ResNet-TSN backbone model in PaddlePaddle, and registering it to the BACKBONES registry. It also includes license information and mentions function-level future imports for compatibility and division settings. The code initializes parameters, defines Conv2D, BatchNorm, MaxPool2D, AvgPool2D layers, and sets up weight initialization functions and loading checkpoints utilities.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":0-28",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nimport paddle\nfrom paddle import ParamAttr\nimport paddle.nn as nn\nimport paddle.nn.functional as F\nfrom paddle.regularizer import L2Decay\nfrom paddle.nn import Conv2D, BatchNorm\nfrom paddle.nn import MaxPool2D, AvgPool2D\nfrom ..registry import BACKBONES\nfrom ..weight_init import weight_init_\nfrom ...utils import load_ckpt"
+ },
+ {
+ "comment": "Defines a ConvBNLayer class with an average pooling layer and convolutional layer. The class takes input parameters for channels, kernel size, stride, groups, and more. It initializes the layers and sets is_tweaks_mode flag.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":30-57",
+ "content": "__all__ = [\"ResNetTSN_MRI\"]\nclass ConvBNLayer(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n kernel_size,\n stride=1,\n groups=1,\n is_tweaks_mode=False,\n act=None,\n lr_mult=1.0,\n name=None):\n super(ConvBNLayer, self).__init__()\n self.is_tweaks_mode = is_tweaks_mode\n self._pool2d_avg = AvgPool2D(kernel_size=2,\n stride=2,\n padding=0,\n ceil_mode=True)\n self._conv = Conv2D(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=(kernel_size - 1) // 2,\n groups=groups,\n weight_attr=ParamAttr(name=name + \"_weights\",\n learning_rate=lr_mult),"
+ },
+ {
+ "comment": "This code defines a class for Resnet_TSN, which is a type of backbone model. It includes an initialization function that initializes the BatchNorm layer and a forward function that applies pooling (if in tweaks mode), convolution, and batch normalization to inputs. Additionally, there is a BottleneckBlock class defined for creating bottleneck blocks within the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":58-88",
+ "content": " bias_attr=False)\n if name == \"conv1\":\n bn_name = \"bn_\" + name\n else:\n bn_name = \"bn\" + name[3:]\n self._batch_norm = BatchNorm(\n out_channels,\n act=act,\n param_attr=ParamAttr(name=bn_name + '_scale',\n learning_rate=lr_mult,\n regularizer=L2Decay(0.0)),\n bias_attr=ParamAttr(bn_name + '_offset',\n learning_rate=lr_mult,\n regularizer=L2Decay(0.0)),\n moving_mean_name=bn_name + '_mean',\n moving_variance_name=bn_name + '_variance')\n def forward(self, inputs):\n if self.is_tweaks_mode:\n inputs = self._pool2d_avg(inputs)\n y = self._conv(inputs)\n y = self._batch_norm(y)\n return y\nclass BottleneckBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,\n shortcut=True,"
+ },
+ {
+ "comment": "The code defines a BottleneckBlock class, which is a layer in the ResNet model. It consists of three ConvBNLayer layers with different properties such as kernel size, stride, and activation functions. The class initializes these layers and takes input and output channel counts, learning rate multiplier, and name as parameters.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":89-110",
+ "content": " if_first=False,\n lr_mult=1.0,\n name=None):\n super(BottleneckBlock, self).__init__()\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=1,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2b\")\n self.conv2 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n act=None,"
+ },
+ {
+ "comment": "The code defines a ResNet TSN backbone with two branches, where the first branch contains convolutional layers and the second branch has a shortcut connection. The forward function performs addition between the shortcut connection and the output of the convolutional layers. The BasicBlock class is a subclass for implementing basic building blocks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":111-143",
+ "content": " lr_mult=lr_mult,\n name=name + \"_branch2c\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels * 4,\n kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n lr_mult=lr_mult,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n conv2 = self.conv2(conv1)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv2)\n y = F.relu(y)\n return y\nclass BasicBlock(nn.Layer):\n def __init__(self,\n in_channels,\n out_channels,\n stride,"
+ },
+ {
+ "comment": "This code defines a BasicBlock class in PaddleVideo for ResNet TSN MRI model. It has an input, output channels, and stride. The class initializes convolution layers (conv0 and conv1) with specified parameters. If shortcut is not set, it also includes a ConvBNLayer as the 'short' attribute.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":144-166",
+ "content": " shortcut=True,\n if_first=False,\n lr_mult=1.0,\n name=None):\n super(BasicBlock, self).__init__()\n self.stride = stride\n self.conv0 = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n act='relu',\n lr_mult=lr_mult,\n name=name + \"_branch2a\")\n self.conv1 = ConvBNLayer(in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n act=None,\n lr_mult=lr_mult,\n name=name + \"_branch2b\")\n if not shortcut:\n self.short = ConvBNLayer(in_channels=in_channels,\n out_channels=out_channels,"
+ },
+ {
+ "comment": "This code defines a ResNetTSN_MRI backbone with specified depth, pretrained model option, and learning rate multipliers for each layer. The forward function performs convolutions and shortcut connections, applying ReLU activation at the end. This backbone is registered in BACKBONES for use in the PaddleVideo library.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":167-201",
+ "content": " kernel_size=1,\n stride=1,\n is_tweaks_mode=False if if_first else True,\n lr_mult=lr_mult,\n name=name + \"_branch1\")\n self.shortcut = shortcut\n def forward(self, inputs):\n y = self.conv0(inputs)\n conv1 = self.conv1(y)\n if self.shortcut:\n short = inputs\n else:\n short = self.short(inputs)\n y = paddle.add(x=short, y=conv1)\n y = F.relu(y)\n return y\n@BACKBONES.register()\nclass ResNetTSN_MRI(nn.Layer):\n \"\"\"ResNetTweaksTSN backbone.\n Args:\n depth (int): Depth of resnet model.\n pretrained (str): pretrained model. Default: None.\n \"\"\"\n def __init__(self,\n layers=50,\n pretrained=None,\n lr_mult_list=[1.0, 1.0, 1.0, 1.0, 1.0],\n in_channels=1):\n super(ResNetTSN_MRI, self).__init__()"
+ },
+ {
+ "comment": "This code initializes a ResNet TSN backbone model with specified layers, in_channels, and pretrained weight option. It supports specific layer options (18, 34, 50, 101, 152, 200) and checks if the input layer is within supported range. The code also ensures lr_mult_list is a list or tuple and has a length of 5. Depending on the layers, it assigns depth values for each block in the model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":203-231",
+ "content": " self.pretrained = pretrained\n self.layers = layers\n supported_layers = [18, 34, 50, 101, 152, 200]\n assert layers in supported_layers, \\\n \"supported layers are {} but input layer is {}\".format(\n supported_layers, layers)\n self.lr_mult_list = lr_mult_list\n self.in_channels = in_channels\n assert isinstance(\n self.lr_mult_list,\n (list, tuple\n )), \"lr_mult_list should be in (list, tuple) but got {}\".format(\n type(self.lr_mult_list))\n assert len(\n self.lr_mult_list\n ) == 5, \"lr_mult_list length should should be 5 but got {}\".format(\n len(self.lr_mult_list))\n if layers == 18:\n depth = [2, 2, 2, 2]\n elif layers == 34 or layers == 50:\n depth = [3, 4, 6, 3]\n elif layers == 101:\n depth = [3, 4, 23, 3]\n elif layers == 152:\n depth = [3, 8, 36, 3]\n elif layers == 200:\n depth = [3, 12, 48, 3]"
+ },
+ {
+ "comment": "This code defines a ResNet model for Temporal Segment Networks (TSN) with multiple branch inputs. It initializes the layers of the network, including convolutional and batch normalization operations. The number of channels and filters used in each layer depend on the total number of layers specified. Different learning rate multipliers are assigned to each layer for efficient training.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":232-252",
+ "content": " num_channels = [64, 256, 512, 1024\n ] if layers >= 50 else [64, 64, 128, 256]\n num_filters = [64, 128, 256, 512]\n self.conv1_1 = ConvBNLayer(in_channels=self.in_channels,\n out_channels=32,\n kernel_size=3,\n stride=2,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_1\")\n self.conv1_2 = ConvBNLayer(in_channels=32,\n out_channels=32,\n kernel_size=3,\n stride=1,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_2\")\n self.conv1_3 = ConvBNLayer(in_channels=32,\n out_channels=64,\n kernel_size=3,"
+ },
+ {
+ "comment": "Initializing layers of ResNet-TSN with specified depth, creating bottleneck blocks for each layer. If layers are 101, 152 or 200 and block is 2, specific naming convention applied. BottleneckBlock is added as sublayer in a sequential manner.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":253-274",
+ "content": " stride=1,\n act='relu',\n lr_mult=self.lr_mult_list[0],\n name=\"conv1_3\")\n self.pool2d_max = MaxPool2D(kernel_size=3, stride=2, padding=1)\n self.block_list = []\n if layers >= 50:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n if layers in [101, 152, 200] and block == 2:\n if i == 0:\n conv_name = \"res\" + str(block + 2) + \"a\"\n else:\n conv_name = \"res\" + str(block + 2) + \"b\" + str(i)\n else:\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n bottleneck_block = self.add_sublayer(\n 'bb_%d_%d' % (block, i),\n BottleneckBlock(\n in_channels=num_channels[block]"
+ },
+ {
+ "comment": "The code creates a ResNet TSN model with bottleneck and basic blocks. It initializes the block_list by adding each block, sets shortcut to True for the first block of each stage, and appends each block to block_list. The number of filters, out_channels, stride, and other parameters are determined based on the stage and block indexes. The name of each block is also specified according to its position in the network.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":275-293",
+ "content": " if i == 0 else num_filters[block] * 4,\n out_channels=num_filters[block],\n stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n if_first=block == i == 0,\n lr_mult=self.lr_mult_list[block + 1],\n name=conv_name))\n self.block_list.append(bottleneck_block)\n shortcut = True\n else:\n for block in range(len(depth)):\n shortcut = False\n for i in range(depth[block]):\n conv_name = \"res\" + str(block + 2) + chr(97 + i)\n basic_block = self.add_sublayer(\n 'bb_%d_%d' % (block, i),\n BasicBlock(in_channels=num_channels[block]\n if i == 0 else num_filters[block],\n out_channels=num_filters[block],"
+ },
+ {
+ "comment": "This code initializes the weights of a ResNet TSN backbone model. It creates blocks with specified parameters and appends them to the block list. The `init_weights` function initializes the parameters based on whether pretrained loading path is indicated or not, following specific initialization functions for Conv2D and BatchNorm2d layers.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":294-310",
+ "content": " stride=2 if i == 0 and block != 0 else 1,\n shortcut=shortcut,\n if_first=block == i == 0,\n name=conv_name,\n lr_mult=self.lr_mult_list[block + 1]))\n self.block_list.append(basic_block)\n shortcut = True\n def init_weights(self):\n \"\"\"Initiate the parameters.\n Note:\n 1. when indicate pretrained loading path, will load it to initiate backbone.\n 2. when not indicating pretrained loading path, will follow specific initialization initiate backbone. Always, Conv2D layer will be\n initiated by KaimingNormal function, and BatchNorm2d will be initiated by Constant function.\n Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/api/paddle/nn/initializer/kaiming/KaimingNormal_en.html\n \"\"\"\n # XXX: check bias!!! check pretrained!!!"
+ },
+ {
+ "comment": "This code is checking if the pretrained model path is provided and initializing weights for Conv2D and BatchNorm2D layers if not. The forward function performs convolutions, max pooling, and processes through blocks to output a result.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/backbones/resnet_tsn_MRI.py\":312-330",
+ "content": " if isinstance(self.pretrained, str) and self.pretrained.strip() != \"\":\n load_ckpt(self, self.pretrained)\n elif self.pretrained is None or self.pretrained.strip() == \"\":\n for layer in self.sublayers():\n if isinstance(layer, nn.Conv2D):\n # XXX: no bias\n weight_init_(layer, 'KaimingNormal')\n elif isinstance(layer, nn.BatchNorm2D):\n weight_init_(layer, 'Constant', value=1)\n def forward(self, inputs):\n y = self.conv1_1(inputs)\n y = self.conv1_2(y)\n y = self.conv1_3(y)\n y = self.pool2d_max(y)\n for block in self.block_list:\n y = block(y)\n return y"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/93ea5b89-273b-4528-9e44-c40bd93ff284.json b/docs/doc/93ea5b89-273b-4528-9e44-c40bd93ff284.json
new file mode 100644
index 000000000..f2fe94e0e
--- /dev/null
+++ b/docs/doc/93ea5b89-273b-4528-9e44-c40bd93ff284.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code is an import script for PaddleVideo, registering various modules like backbones, heads, recognizers, localizers, and losses in relevant registries. It defines key function names and includes popular models like 'DeepLab' and 'IntVOS'.",
+ "details": [
+ {
+ "comment": "This code appears to be an import script for PaddleVideo, importing various modules such as backbones, heads, recognizers, localizers, and losses from different parts of the codebase. It also registers these items in relevant registries (e.g., BACKBONES, HEADS) and defines __all__ to include those registered items.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/__init__.py\":0-22",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .backbones import DeepLab\nfrom .builder import (build_backbone, build_head, build_localizer, build_loss,\n build_recognizer)\nfrom .heads import IntVOS\nfrom .registry import (BACKBONES, DETECTORS, HEADS, LOCALIZERS, LOSSES,\n PARTITIONERS, RECOGNIZERS, ROI_EXTRACTORS)\nfrom .weight_init import kaiming_normal_, trunc_normal_, weight_init_\n__all__ = [\n 'BACKBONES', 'HEADS', 'RECOGNIZERS', 'LOCALIZERS', 'PARTITIONERS',"
+ },
+ {
+ "comment": "This code defines several variables and function names used in the PaddleVideo library, including loss functions ('build_loss'), backbone building ('build_backbone'), detectors ('DETECTORS'), and initialization methods ('weight_init_'). It also includes references to popular models such as 'DeepLab' and 'IntVOS'.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/modeling/__init__.py\":23-26",
+ "content": " 'LOSSES', 'build_recognizer', 'build_localizer', 'build_head',\n 'build_backbone', 'build_loss', 'DETECTORS', 'kaiming_normal_', 'trunc_normal_',\n 'weight_init_', 'DeepLab', 'IntVOS'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/94337949-c250-4aa2-8a25-eba76cf708c9.json b/docs/doc/94337949-c250-4aa2-8a25-eba76cf708c9.json
new file mode 100644
index 000000000..6aa0fcc91
--- /dev/null
+++ b/docs/doc/94337949-c250-4aa2-8a25-eba76cf708c9.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code adds a YOWOMetric class to the PaddleVideo framework for measuring YOWO metrics in two stages: saving test results and calculating metrics from saved results files. The code also handles batch processing, logging progress, and evaluates mAP metrics.",
+ "details": [
+ {
+ "comment": "This code defines a YOWOMetric class within the PaddleVideo framework. The class measures metrics for YOWO in two stages: first, it saves test results using a trained model, and then calculates metrics from the saved results file.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/yowo_metric.py\":0-29",
+ "content": "# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport os.path as osp\nimport os\nfrom paddlevideo.utils import get_logger\nfrom .registry import METRIC\nfrom .base import BaseMetric\nfrom .ucf24_utils import get_mAP\nlogger = get_logger(\"paddlevideo\")\n@METRIC.register\nclass YOWOMetric(BaseMetric):\n \"\"\"\n Metrics for YOWO. Two Stages in this metric:\n (1) Get test results using trained model, results will be saved in YOWOMetric.result_path;\n (2) Calculate metrics using results file from stage (1)."
+ },
+ {
+ "comment": "The code initializes an instance of a BMN metrics class with specified parameters. It checks if the result path exists and creates it if not, then updates the metric by writing detection results to corresponding files in the result path for each batch.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/yowo_metric.py\":30-61",
+ "content": " \"\"\"\n def __init__(self,\n data_size,\n batch_size,\n gt_folder,\n result_path,\n threshold=0.5,\n save_path=None,\n log_interval=1):\n \"\"\"\n Init for BMN metrics.\n Params:\n gtfolder:groundtruth folder path for ucf24\n \"\"\"\n super().__init__(data_size, batch_size, log_interval)\n self.result_path = result_path\n self.gt_folder = gt_folder\n self.threshold = threshold\n self.save_path = save_path\n if not osp.isdir(self.result_path):\n os.makedirs(self.result_path)\n def update(self, batch_id, data, outputs):\n frame_idx = outputs['frame_idx']\n boxes = outputs[\"boxes\"]\n for j in range(len(frame_idx)):\n detection_path = osp.join(self.result_path, frame_idx[j])\n with open(detection_path, 'w+') as f_detect:\n for box in boxes[j]:\n x1 = round(float(box[0] - box[2] / 2.0) * 320.0)"
+ },
+ {
+ "comment": "This code snippet is part of the PaddleVideo library. It calculates and writes yolo v5 box information into a file, handling batch processing and logging progress with an interval. The accumulate function collects mAP metrics for evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/metrics/yowo_metric.py\":62-81",
+ "content": " y1 = round(float(box[1] - box[3] / 2.0) * 240.0)\n x2 = round(float(box[0] + box[2] / 2.0) * 320.0)\n y2 = round(float(box[1] + box[3] / 2.0) * 240.0)\n det_conf = float(box[4])\n for j in range((len(box) - 5) // 2):\n cls_conf = float(box[5 + 2 * j].item())\n prob = det_conf * cls_conf\n f_detect.write(\n str(int(box[6]) + 1) + ' ' + str(prob) + ' ' + str(x1) + ' ' + str(y1) + ' ' + str(\n x2) + ' ' + str(y2) + '\\n')\n if batch_id % self.log_interval == 0:\n logger.info(\"[TEST] Processing batch {}/{} ...\".format(\n batch_id,\n self.data_size // (self.batch_size * self.world_size)))\n def accumulate(self):\n metric_list = get_mAP(self.gt_folder, self.result_path, self.threshold, self.save_path)\n for info in metric_list:\n logger.info(info)"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/945ca542-cb79-4595-ab34-f63406fa5b18.json b/docs/doc/945ca542-cb79-4595-ab34-f63406fa5b18.json
new file mode 100644
index 000000000..d9a1b152a
--- /dev/null
+++ b/docs/doc/945ca542-cb79-4595-ab34-f63406fa5b18.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code sets up the logging configuration based on a provided JSON file. If the file is found, it modifies the filename paths in the configuration according to the save_dir and then uses logging.config.dictConfig() to configure the logging system. If the file is not found, it uses basicConfig() with default level to set up logging. The function returns the filename for the \"info_file_handler\".",
+ "details": [
+ {
+ "comment": "This code sets up the logging configuration based on a provided JSON file. If the file is found, it modifies the filename paths in the configuration according to the save_dir and then uses logging.config.dictConfig() to configure the logging system. If the file is not found, it uses basicConfig() with default level to set up logging. The function returns the filename for the \"info_file_handler\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/logger.py\":0-24",
+ "content": "import os\nimport logging\nimport logging.config\nfrom pathlib import Path\nfrom utils import read_json\ndef setup_logging(save_dir, log_config='logger/logger_config.json',\n default_level=logging.INFO):\n \"\"\"Setup logging configuration.\"\"\"\n print(os.getcwd())\n log_config = Path(log_config)\n print(f\"log config: {log_config} exists: {log_config.exists()}\")\n if log_config.is_file():\n config = read_json(log_config)\n # modify logging paths based on run config\n for _, handler in config['handlers'].items():\n if 'filename' in handler:\n handler['filename'] = str(save_dir / handler['filename'])\n logging.config.dictConfig(config)\n else:\n print(f\"Warning: logging configuration file is not found in {log_config}.\")\n logging.basicConfig(level=default_level)\n return config[\"handlers\"][\"info_file_handler\"][\"filename\"]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/94a1a1ca-0f94-45cf-ad4f-6f8a8d7d67d2.json b/docs/doc/94a1a1ca-0f94-45cf-ad4f-6f8a8d7d67d2.json
new file mode 100644
index 000000000..3e782a306
--- /dev/null
+++ b/docs/doc/94a1a1ca-0f94-45cf-ad4f-6f8a8d7d67d2.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code imports diverse loss functions from different modules, such as CrossEntropyLoss, BMNLoss, and TransNetV2Loss for PaddleVideo's video recognition, segmentation tasks, providing a comprehensive list of usable losses. The PaddleVideo model uses BaseWeightedLoss, ASRFLoss, DistillationCELoss, and DistillationDMLLoss for audio-visual speech recognition, distillation-based learning, and region-specific loss computation.",
+ "details": [
+ {
+ "comment": "This code imports different types of loss functions from various modules, such as CrossEntropyLoss, BMNLoss, and TransNetV2Loss. These losses are used in PaddleVideo for various applications like video recognition, segmentation, and more. The code provides a comprehensive list of loss functions that can be used depending on the specific task.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/__init__.py\":0-25",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .base import BaseWeightedLoss\nfrom .bmn_loss import BMNLoss\nfrom .cross_entropy_loss import CrossEntropyLoss\nfrom .depth_loss import ADDSLoss\nfrom .transnetv2_loss import TransNetV2Loss\nfrom .actbert_loss import ActBertLoss\nfrom .asrf_loss import ASRFLoss\nfrom .distillation_loss import DistillationCELoss, DistillationDMLLoss\nfrom .yowo_loss import RegionLoss\n__all__ = [\n 'CrossEntropyLoss', 'BMNLoss', 'TransNetV2Loss', 'ActBertLoss', 'ADDSLoss',"
+ },
+ {
+ "comment": "The code defines a list of loss functions used in the PaddleVideo model. These losses include BaseWeightedLoss, ASRFLoss, DistillationCELoss, and DistillationDMLLoss for various tasks like audio-visual speech recognition, distillation-based learning, and region-specific loss computation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/losses/__init__.py\":26-28",
+ "content": " 'BaseWeightedLoss', 'ASRFLoss', 'DistillationCELoss', 'DistillationDMLLoss',\n 'RegionLoss'\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9540c56a-4dbe-4ffb-abf9-efb046cf75de.json b/docs/doc/9540c56a-4dbe-4ffb-abf9-efb046cf75de.json
new file mode 100644
index 000000000..9a57ebcc8
--- /dev/null
+++ b/docs/doc/9540c56a-4dbe-4ffb-abf9-efb046cf75de.json
@@ -0,0 +1,20 @@
+{
+ "summary": "The code defines a PaddlePaddle class \"ppTimeSformerHead\" as a head for the TimeSformer model, extending BaseHead and initializing fully connected layers with truncated normal distribution.",
+ "details": [
+ {
+ "comment": "This code defines a class called \"ppTimeSformerHead\" which is a head for the TimeSformer model in PaddlePaddle framework. It extends the BaseHead class, and has attributes such as num_classes, in_channels. The class also registers itself in the HEADS registry of the PaddleVideo module. The code uses Linear and ParamAttr from paddle.nn and weight_init from .base, and imports trunc_normal_ and L2Decay from other modules.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptimesformer_head.py\":0-29",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom paddle.nn import Linear\nfrom ..registry import HEADS\nfrom ..weight_init import trunc_normal_, weight_init_\nfrom .base import BaseHead\nfrom paddle import ParamAttr\nfrom paddle.regularizer import L2Decay\n@HEADS.register()\nclass ppTimeSformerHead(BaseHead):\n \"\"\"TimeSformerHead Head.\n Args:\n num_classes (int): The number of classes to be classified.\n in_channels (int): The number of channles in input feature."
+ },
+ {
+ "comment": "The code defines a class named \"PPTimesformerHead\" with an __init__ method that takes parameters such as num_classes, in_channels, loss_cfg (with default value), std (with default 0.02), and optional kwargs. It initializes superclass attributes, sets self.std, and initializes the FC layer parameters using weight_init_. The TruncatedNormal initialization method is used with specific attribute names.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptimesformer_head.py\":30-57",
+ "content": " loss_cfg (dict): Config for building config. Default: dict(name='CrossEntropyLoss').\n std(float): Std(Scale) value in normal initilizar. Default: 0.01.\n kwargs (dict, optional): Any keyword argument to initialize.\n \"\"\"\n def __init__(self,\n num_classes,\n in_channels,\n loss_cfg=dict(name='CrossEntropyLoss'),\n std=0.02,\n **kwargs):\n super().__init__(num_classes, in_channels, loss_cfg, **kwargs)\n self.std = std\n self.fc = Linear(self.in_channels,\n self.num_classes,\n bias_attr=ParamAttr(regularizer=L2Decay(0.0)))\n def init_weights(self):\n \"\"\"Initiate the FC layer parameters\"\"\"\n weight_init_(self.fc,\n 'TruncatedNormal',\n 'fc_0.w_0',\n 'fc_0.b_0',\n mean=0.0,\n std=self.std)\n # NOTE: Temporarily use trunc_normal_ instead of TruncatedNormal"
+ },
+ {
+ "comment": "The code defines a head for the PPTimesformer model. It initializes the fully connected layer (fc) with truncated normal distribution and defines the forward pass, which involves passing input through fc to generate scores for classification tasks.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/modeling/heads/pptimesformer_head.py\":58-73",
+ "content": " trunc_normal_(self.fc.weight, std=self.std)\n def forward(self, x):\n \"\"\"Define how the head is going to run.\n Args:\n x (paddle.Tensor): The input data.\n Returns:\n score: (paddle.Tensor) The classification scores for input samples.\n \"\"\"\n # XXX: check dropout location!\n # x.shape = [N, embed_dim]\n score = self.fc(x)\n # [N, num_class]\n # x = F.softmax(x) # NOTE remove\n return score"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9540f655-8d4a-4d20-b072-692b7b466706.json b/docs/doc/9540f655-8d4a-4d20-b072-692b7b466706.json
new file mode 100644
index 000000000..217345615
--- /dev/null
+++ b/docs/doc/9540f655-8d4a-4d20-b072-692b7b466706.json
@@ -0,0 +1,15 @@
+{
+ "summary": "This code defines a Softmax class with an Inplace_Run method that applies softmax function in-place to iterator ranges of float vectors. The code also includes a virtual function for postprocessing operations in the PaddleVideo library.",
+ "details": [
+ {
+ "comment": "This code defines a class Softmax that contains a method Inplace_Run. The method takes an iterator range of a vector of floats and applies softmax function in-place to the values within this range.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/include/postprocess_op.h\":0-38",
+ "content": "// Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n#pragma once\n#include \"opencv2/core.hpp\"\n#include \"opencv2/imgcodecs.hpp\"\n#include \"opencv2/imgproc.hpp\"\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \"include/utility.h\"\nnamespace PaddleVideo\n{\n class Softmax\n {\n public:\n virtual void Inplace_Run(const std::vector::iterator &_begin, const std::vector::iterator &_end);"
+ },
+ {
+ "comment": "This code defines a virtual function that takes in two iterators to a vector of floats and returns a vector of floats as output. It is part of the PaddleVideo library's postprocessing operation namespace.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/include/postprocess_op.h\":39-42",
+ "content": " virtual std::vector Run(const std::vector::iterator &_begin, const std::vector::iterator &_end);\n };\n} // namespace PaddleVideo"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/95a099dd-1734-422a-b059-32270567b137.json b/docs/doc/95a099dd-1734-422a-b059-32270567b137.json
new file mode 100644
index 000000000..db14ba1c7
--- /dev/null
+++ b/docs/doc/95a099dd-1734-422a-b059-32270567b137.json
@@ -0,0 +1,35 @@
+{
+ "summary": "The 2s-AGCN model, an enhanced ST-GCN version for motion recognition, utilizes dual-flow adaptive convolutional networks and focuses on second-order bone data. Code offers test scripts, accuracy results, and download links for models trained on different datasets, with PaddleVideo exporting an action recognition model using AGCN2s.",
+ "details": [
+ {
+ "comment": "This code provides an introduction to the 2s-AGCN model, an improved version of ST-GCN published in CVPR2019. It uses a dual-flow adaptive convolutional network and focuses on the second-order information of bone data for motion recognition.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/agcn2s.md\":0-19",
+ "content": "[\u7b80\u4f53\u4e2d\u6587](../../../zh-CN/model_zoo/recognition/2sAGCN.md) | English\n# CTR-GCN\n---\n## Contents\n- [Introduction](#Introduction)\n- [Data](#Data)\n- [Train](#Train)\n- [Test](#Test)\n- [Inference](#Inference)\n- [Reference](#Reference)\n## Introduction\n\n[2s-AGCN](https://openaccess.thecvf.com/content_CVPR_2019/papers/Shi_Two-Stream_Adaptive_Graph_Convolutional_Networks_for_Skeleton-Based_Action_Recognition_CVPR_2019_paper.pdf) is an improved article on ST-GCN published in CVPR2019. It proposes a dual-flow adaptive convolutional network, which improves the shortcomings of the original ST-GCN. In the existing GCN based approach, the topology of the graph is set manually and fixed to all layers and input samples. In addition, the second-order information of bone data (bone length and orientation) is naturally more beneficial and discriminating for motion recognition, which was rarely studied in the methods at that time. Therefore, this paper puts forward a node and bones of tw"
+ },
+ {
+ "comment": "This code provides information on the AGCN2S model, a skeleton-based gesture recognition network. It uses data from NTU-RGBD, with details of its preparation found in another file. The code also outlines how to train the CTR-GCN model on various configurations such as cross-subject and cross-view training scenarios, using bone or joint data. This serves as a guide for running the model's training scripts.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/agcn2s.md\":19-39",
+ "content": "o kinds of information fusion based on skeleton shuangliu network, and join in figure convolution adjacency matrix adaptive matrix, a sharp rise in the bones of gesture recognition accuracy, also has laid the foundation for subsequent work (the subsequent basic skeleton gesture recognition are based on the flow of network framework).\n## Data\nData download and processing are consistent with CTR-GCN. For details, please refer to [NTU-RGBD Data Preparation](../../dataset/ntu-rgbd.md)\n## Train\n### Train on NTU-RGBD\nTrain CTR-GCN on NTU-RGBD scripts using single gpu\uff1a\n```bash\n# train cross subject with bone data\npython main.py --validate -c configs/recognition/agcn2s/agcn2s_ntucs_bone.yaml --seed 1\n# train cross subject with joint data\npython main.py --validate -c configs/recognition/agcn2s/agcn2s_ntucs_joint.yaml --seed 1\n# train cross view with bone data\npython main.py --validate -c configs/recognition/agcn2s/agcn2s_ntucv_bone.yaml --seed 1\n# train cross view with joint data\npython main.py --validate -c configs/recognition/agcn2s/agcn2s_ntucv_joint.yaml --seed 1"
+ },
+ {
+ "comment": "The code provides test scripts for the 2s-AGCN model on the NTU-RGB+D dataset, both with cross-subject and cross-view splits. The accuracy results for joint and bone data are given, along with a download link to the training log.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/agcn2s.md\":40-70",
+ "content": "```\nconfig file `agcn2s_ntucs_joint.yaml` corresponding to the config of 2s-AGCN on NTU-RGB+D dataset with cross-subject splits.\n## Test\n### Test on NTU-RGB+D\nTest scripts\uff1a\n```bash\n# test cross subject with bone data\npython main.py --test -c configs/recognition/2sagcn/2sagcn_ntucs_bone.yaml -w data/2SAGCN_ntucs_bone.pdparams\n# test cross subject with joint data\npython main.py --test -c configs/recognition/2sagcn/2sagcn_ntucs_joint.yaml -w data/2SAGCN_ntucs_joint.pdparams\n# test cross view with bone data\npython main.py --test -c configs/recognition/2sagcn/2sagcn_ntucv_bone.yaml -w data/2SAGCN_ntucv_bone.pdparams\n# test cross view with joint data\npython main.py --test -c configs/recognition/2sagcn/2sagcn_ntucv_joint.yaml -w data/2SAGCN_ntucv_joint.pdparams\n```\n* Specify the config file with `-c`, specify the weight path with `-w`.\nAccuracy on NTU-RGB+D dataset:\n| | CS | CV |\n| :------------: | :---: | :----: |\n| Js-AGCN(joint) | 85.8% | 94.13% |\n| Bs-AGCN(bone) | 86.7% | 93.9% |\nTrain log\uff1a[download](https://github.com/ELKYang/2s-AGCN-paddle/tree/main/work_dir/ntu)"
+ },
+ {
+ "comment": "Code snippet contains download links for different checkpoints of the AGCN-2s model trained on various datasets:\n1. ntu_cs_agcn_joint\n2. ntu_cs_agcn_bone\n3. ntu_cv_agcn_joint\n4. ntu_cv_agcn_bone",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/agcn2s.md\":72-78",
+ "content": "VisualDL log\uff1a[download](https://github.com/ELKYang/2s-AGCN-paddle/tree/main/runs)\ncheckpoints\uff1a\n| CS-Js | CS-Bs | CV-Js | CV-Bs |\n| :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: |\n| [ntu_cs_agcn_joint](https://github.com/ELKYang/2s-AGCN-paddle/blob/main/weights/ntu_cs_agcn_joint-48-30674.pdparams) | [ntu_cs_agcn_bone](https://github.com/ELKYang/2s-AGCN-paddle/blob/main/weights/ntu_cs_agcn_bone-44-28170.pdparams) | [ntu_cv_agcn_joint](https://github.com/ELKYang/2s-AGCN-paddle/blob/main/weights/ntu_cv_agcn_joint-38-22932.pdparams) | [ntu_cv_agcn_bone](https://github.com/ELKYang/2s-AGCN-paddle/blob/main/weights/ntu_cv_agcn_bone-49-29400.pdparams) |"
+ },
+ {
+ "comment": "This code is exporting and inferring a model for action recognition using PaddleVideo's AGCN2s. It uses the `export_model.py` script to generate an inference model archive, which includes the model architecture file (AGCN2s_ntucs_joint.pdmodel) and parameters file (AGCN2s_ntucs_joint.pdiparams). The `predict.py` script is then used to perform inference on input data with the specified configuration and model files, using GPU if available and disabling TensorRT.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/agcn2s.md\":80-102",
+ "content": "## Inference\n### export inference model\n```bash\npython3.7 tools/export_model.py -c configs/recognition/agcn2s/2sagcn_ntucs_joint.yaml \\\n -p data/AGCN2s_ntucs_joint.pdparams \\\n -o inference/AGCN2s_ntucs_joint\n```\nTo get model architecture file `AGCN2s_ntucs_joint.pdmodel` and parameters file `AGCN2s_ntucs_joint.pdiparams`.\n- Args usage please refer to [Model Inference](https://github.com/PaddlePaddle/PaddleVideo/blob/release/2.0/docs/zh-CN/start.md#2-%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86).\n### infer\n```bash\npython3.7 tools/predict.py --input_file data/example_NTU-RGB-D_sketeton.npy \\\n --config configs/recognition/agcn2s/2sagcn_ntucs_joint.yaml \\\n --model_file inference/AGCN2s_ntucs_joint/AGCN2s_ntucs_joint.pdmodel \\\n --params_file inference/AGCN2s_ntucs_joint/AGCN2s_ntucs_joint.pdiparams \\\n --use_gpu=True \\\n --use_tensorrt=False"
+ },
+ {
+ "comment": "This code block shows the prediction engine result for the AGCN2S model. It displays an image of the prediction results and references the original paper on Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/model_zoo/recognition/agcn2s.md\":103-111",
+ "content": "```\n### infer result\n\n## Reference\n- [Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition](https://openaccess.thecvf.com/content_CVPR_2019/papers/Shi_Two-Stream_Adaptive_Graph_Convolutional_Networks_for_Skeleton-Based_Action_Recognition_CVPR_2019_paper.pdf), Lei Shi and Yifan Zhang and Jian Cheng and Hanqing Lu"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/95a0b7ca-1040-4328-868c-0d127a7bbaca.json b/docs/doc/95a0b7ca-1040-4328-868c-0d127a7bbaca.json
new file mode 100644
index 000000000..5493d4185
--- /dev/null
+++ b/docs/doc/95a0b7ca-1040-4328-868c-0d127a7bbaca.json
@@ -0,0 +1,35 @@
+{
+ "summary": "This code defines a PyTorch class for batch normalization, initializing a BatchNorm3D layer and including methods to compute mean and standard deviation. It also supports aggregating statistics from multiple splits and performs forward pass for training or evaluation.",
+ "details": [
+ {
+ "comment": "This code defines a function `get_norm` that returns the normalization layer based on the provided bn_norm_type and bn_num_splits. If bn_norm_type is 'batchnorm', it returns paddle.nn.BatchNorm3D, otherwise if it's 'sub_batchnorm', it returns a partially applied SubBatchNorm3D function with num_splits parameter set to bn_num_splits. If the norm type isn't supported, it raises a NotImplementedError. It also defines `aggregate_sub_bn_stats` function that recursively finds all SubBN modules in the given model and aggregates sub-BN stats by calling aggregate_stats() on each found SubBatchNorm3D module. It returns the count of SubBN modules found.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/batchnorm_helper.py\":0-35",
+ "content": "from functools import partial\nimport paddle\ndef get_norm(bn_norm_type, bn_num_splits):\n \"\"\"\n Args:\n cfg (CfgNode): model building configs, details are in the comments of\n the config file.\n Returns:\n nn.Layer: the normalization layer.\n \"\"\"\n if bn_norm_type == \"batchnorm\":\n return paddle.nn.BatchNorm3D\n elif bn_norm_type == \"sub_batchnorm\":\n return partial(SubBatchNorm3D, num_splits=bn_num_splits)\n else:\n raise NotImplementedError(\n \"Norm type {} is not supported\".format(bn_norm_type))\ndef aggregate_sub_bn_stats(model):\n \"\"\"\n Recursively find all SubBN modules and aggregate sub-BN stats.\n Args:\n model (nn.Layer): model to be aggregate sub-BN stats\n Returns:\n count (int): number of SubBN module found.\n \"\"\"\n count = 0\n for child in model.children():\n if isinstance(child, SubBatchNorm3D):\n child.aggregate_stats()\n count += 1\n else:\n count += aggregate_sub_bn_stats(child)"
+ },
+ {
+ "comment": "The code defines a SubBatchNorm3D class that implements Batch Normalization with the option to split the batch dimension into N splits. It computes stats for each subset of examples independently during training and aggregates them during evaluation. The class takes num_splits as an argument and other parameters such as num_features, weight_attr, and bias_attr are set in its constructor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/batchnorm_helper.py\":36-63",
+ "content": " return count\nclass SubBatchNorm3D(paddle.nn.Layer):\n \"\"\"\n Implement based on paddle2.0.\n The standard BN layer computes stats across all examples in a GPU. In some\n cases it is desirable to compute stats across only a subset of examples\n SubBatchNorm3D splits the batch dimension into N splits, and run BN on\n each of them separately (so that the stats are computed on each subset of\n examples (1/N of batch) independently. During evaluation, it aggregates\n the stats from all splits into one BN.\n \"\"\"\n def __init__(self, num_splits, **args):\n \"\"\"\n Args:\n num_splits (int): number of splits.\n args (list): list of args\n \"\"\"\n super(SubBatchNorm3D, self).__init__()\n self.num_splits = num_splits\n self.num_features = args[\"num_features\"]\n self.weight_attr = args[\"weight_attr\"]\n self.bias_attr = args[\"bias_attr\"]\n # Keep only one set of weight and bias (outside).\n if self.weight_attr == False:\n self.weight = self.create_parameter("
+ },
+ {
+ "comment": "This code initializes the weight and bias parameters of a BatchNorm layer in PaddlePaddle. If learning rate is 0, it sets weight to have no gradient update, and if `bias_attr` is False, it sets the bias to True and stops its gradients from being updated.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/batchnorm_helper.py\":64-84",
+ "content": " attr=None,\n shape=[self.num_features],\n default_initializer=paddle.nn.initializer.Constant(1.0))\n self.weight.stop_gradient = True\n else:\n self.weight = self.create_parameter(\n attr=self.weight_attr,\n shape=[self.num_features],\n default_initializer=paddle.nn.initializer.Constant(1.0))\n self.weight.stop_gradient = self.weight_attr is not None \\\n and self.weight_attr.learning_rate == 0.\n if self.bias_attr == False:\n self.bias = self.create_parameter(attr=None,\n shape=[self.num_features],\n is_bias=True)\n self.bias.stop_gradient = True\n else:\n self.bias = self.create_parameter(attr=self.bias_attr,\n shape=[self.num_features],\n is_bias=True)"
+ },
+ {
+ "comment": "Class is initializing a BatchNorm3D layer and storing two instances of it (self.bn and self.split_bn). The first instance has its weights and bias set as fixed (inner), while the second instance handles splitting the features for a specified number of splits. The function _get_aggregated_mean_std calculates the aggregated mean and standard deviation by summing each set's means and stds, then dividing them by the total count to get the average values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/batchnorm_helper.py\":85-107",
+ "content": " self.bias.stop_gradient = self.bias_attr is not None \\\n and self.bias_attr.learning_rate == 0.\n # set weights and bias fixed (inner).\n args[\"weight_attr\"] = False\n args[\"bias_attr\"] = False\n self.bn = paddle.nn.BatchNorm3D(**args)\n # update number of features used in split_bn\n args[\"num_features\"] = self.num_features * self.num_splits\n self.split_bn = paddle.nn.BatchNorm3D(**args)\n def _get_aggregated_mean_std(self, means, stds, n):\n \"\"\"\n Calculate the aggregated mean and stds.\n Use the method of update mean and std when merge multi-part data.\n Args:\n means (tensor): mean values.\n stds (tensor): standard deviations.\n n (int): number of sets of means and stds.\n \"\"\"\n mean = paddle.sum(paddle.reshape(means, (n, -1)), axis=0) / n\n std = (paddle.sum(paddle.reshape(stds, (n, -1)), axis=0) / n +\n paddle.sum(paddle.reshape("
+ },
+ {
+ "comment": "This code is defining a class that implements batch normalization in PyTorch. The class has methods to compute the mean and standard deviation, aggregate statistics from multiple splits of batch normalization, and perform forward pass for training or evaluation.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/batchnorm_helper.py\":108-134",
+ "content": " paddle.pow((paddle.reshape(means, (n, -1)) - mean), 2),\n (n, -1)),\n axis=0) / n)\n return mean, std\n def aggregate_stats(self):\n \"\"\"\n Synchronize running_mean, and running_var to self.bn.\n Call this before eval, then call model.eval();\n When eval, forward function will call self.bn instead of self.split_bn,\n During this time the running_mean, and running_var of self.bn has been obtained from\n self.split_bn.\n \"\"\"\n if self.split_bn.training:\n bn_mean_tensor, bn_variance_tensor = self._get_aggregated_mean_std(\n self.split_bn._mean,\n self.split_bn._variance,\n self.num_splits,\n )\n self.bn._mean.set_value(bn_mean_tensor)\n self.bn._variance.set_value(bn_variance_tensor)\n def forward(self, x):\n if self.training:\n n, c, t, h, w = x.shape\n x = paddle.reshape(\n x, (n // self.num_splits, c * self.num_splits, t, h, w))"
+ },
+ {
+ "comment": "The code applies batch normalization to the input tensor and multiplies it by a weight matrix. Then, it adds a bias vector and returns the normalized tensor.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/utils/multigrid/batchnorm_helper.py\":135-141",
+ "content": " x = self.split_bn(x)\n x = paddle.reshape(x, (n, c, t, h, w))\n else:\n x = self.bn(x)\n x = paddle.multiply(x, paddle.reshape(self.weight, (-1, 1, 1, 1)))\n x = paddle.add(x, paddle.reshape(self.bias, (-1, 1, 1, 1)))\n return x"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/96d6083f-7eee-4521-879b-900fb8576159.json b/docs/doc/96d6083f-7eee-4521-879b-900fb8576159.json
new file mode 100644
index 000000000..01cf49d48
--- /dev/null
+++ b/docs/doc/96d6083f-7eee-4521-879b-900fb8576159.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code imports all functions and classes from the logger and log_parser modules in the T2VLAD application of PaddleVideo.",
+ "details": [
+ {
+ "comment": "This code imports all functions and classes from the logger and log_parser modules in the T2VLAD application of PaddleVideo.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/__init__.py\":0-1",
+ "content": "from .logger import *\nfrom .log_parser import *"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/97ee9329-376e-4b29-bc36-82c6df1c716a.json b/docs/doc/97ee9329-376e-4b29-bc36-82c6df1c716a.json
new file mode 100644
index 000000000..a01bc1ff7
--- /dev/null
+++ b/docs/doc/97ee9329-376e-4b29-bc36-82c6df1c716a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "The code outlines six different tasks in action recognition and detection using various algorithms, including TSN (Two-Stream Convolutional Networks), TSM (Temporal Shift Module), SlowFast Networks, LSTM (Long Short-Term Memory), and BNM (Boundary-aware Multi-scale Network). The tasks include single-class action recognition, multi-class action recognition, action localization, spatio-temporal action detection, 3000-class tagging application, and highlights detection application.",
+ "details": [
+ {
+ "comment": "The code outlines six different tasks in action recognition and detection using various algorithms, including TSN (Two-Stream Convolutional Networks), TSM (Temporal Shift Module), SlowFast Networks, LSTM (Long Short-Term Memory), and BNM (Boundary-aware Multi-scale Network). The tasks include single-class action recognition, multi-class action recognition, action localization, spatio-temporal action detection, 3000-class tagging application, and highlights detection application.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/english_documents/tutorials/demos\":0-7",
+ "content": "some useful demo todo. \n1\u3001single-class action recognition\uff0c tsn/tsm/slowfast \n2\u3001multi-class action recognition\uff0clstm \n3\u3001action localization\uff0cbmn \n4\u3001spatio temporal action detection\uff0ctodo \n5\u30013000-class tagging application(videotag)\uff1atsn+lstm \n6\u3001Highlights detection application\uff1abmn+tsn+lstm "
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9899d8bb-c2af-4b2f-977e-15fab28c8797.json b/docs/doc/9899d8bb-c2af-4b2f-977e-15fab28c8797.json
new file mode 100644
index 000000000..7b38d7e40
--- /dev/null
+++ b/docs/doc/9899d8bb-c2af-4b2f-977e-15fab28c8797.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The ModelAudio class extracts audio features using wav_to_example and slices the data into parts, calculating features for each part. The predict method appends these features to a list and returns the audio feature list after dividing by sample rate.",
+ "details": [
+ {
+ "comment": "The code defines a ModelAudio class which takes in audio-related configurations and performs audio feature extraction using the feature_extractor module's wav_to_example function. The class also predicts audio by converting PCM data to numpy array and handles audio file reading exceptions.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/mfcc/model_config.py\":0-41",
+ "content": "\"\"\"\naudio model config\n\"\"\"\nimport numpy as np\nimport mfcc.feature_extractor as feature_extractor\nclass ModelAudio(object):\n \"\"\"\n modelAudio\n \"\"\"\n def __init__(self, configs, use_gpu=1):\n self.use_gpu = use_gpu\n self.audio_fps = configs.COMMON.fps\n self.audio_feat_scale = configs.TSN.audio_scale\n self.sample_rate = 16000\n def predict_slice(self, wav_data, sample_rate):\n \"\"\"\n audio predict\n \"\"\"\n examples_batch = feature_extractor.wav_to_example(\n wav_data, sample_rate)[0]\n return examples_batch\n def predict_audio(self, audio_file):\n \"\"\"\n predict_audio\n \"\"\"\n audio_feature_list = []\n # read pcm\n sample_rate = self.sample_rate\n try:\n with open(audio_file, \"rb\") as f:\n pcm_data = f.read()\n audio_data = np.fromstring(pcm_data, dtype=np.int16)\n audio_status = \"audio load success\"\n except Exception as e:\n audio_data = []\n audio_status = \"audio load failed\""
+ },
+ {
+ "comment": "The code slices the audio data into parts of size 'step' and calculates features for each part using a predict method, then appends the features to a list. The length of the entire audio data is divided by the sample rate to determine how many steps can fit in it. This function returns the audio feature list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/BasketballAction/predict/action_detect/mfcc/model_config.py\":42-50",
+ "content": " step = 1\n len_video = int(len(audio_data) / sample_rate)\n print(len_video)\n for i in range(0, len_video, step):\n audio_data_part = audio_data[i * sample_rate:(i + step) *\n sample_rate]\n feature_audio = self.predict_slice(audio_data_part, sample_rate)\n audio_feature_list.append(feature_audio)\n return audio_feature_list"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/98ccb6a1-3f49-4753-a871-0e0f550fa97f.json b/docs/doc/98ccb6a1-3f49-4753-a871-0e0f550fa97f.json
new file mode 100644
index 000000000..d6f7ca89c
--- /dev/null
+++ b/docs/doc/98ccb6a1-3f49-4753-a871-0e0f550fa97f.json
@@ -0,0 +1,30 @@
+{
+ "summary": "The `log_summary` function gathers performance stats, identifies seeds, searches metrics, and calculates scores for epochs. If evaluation mode is \"fixed_num_epochs,\" it logs the fixed training length, then calculates mean and standard deviation for each metric in aggregated scores using numpy functions.",
+ "details": [
+ {
+ "comment": "The function `log_summary` extracts performance statistics from log files and takes arguments such as a logger reference, log file path, evaluation mode (test run, fixed number of epochs or geometric mean), and optional fixed number of epochs. The log is read, and the performance statistics are extracted based on the given evaluation mode.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/log_parser.py\":0-23",
+ "content": "import re\nimport scipy.stats\nimport logging\nimport numpy as np\nfrom collections import defaultdict\ndef log_summary(logger, log_path, eval_mode=\"test_run\", fixed_num_epochs=None):\n \"\"\"Extract performace statistics from experiment log files.\n Args:\n logger (logger): reference to primary logging instance\n log_path (Path): the path to the log file\n eval_mode (str): the method use to collect the statistics. Can be one of:\n `test_run`, `fixed_num_epochs` or `geometric_mean`\n NOTE: The `eval_mode` argument differs by dataset: for datasets which provide a\n validation set, we use validation set performance to complete a single test run. For\n datasets where no validation set is available, we aim to match prior work by either\n fixing the number of training epochs, or selecting directly from validation set\n performance (Details can be found in the supplementary material of the paper.)\n \"\"\"\n with open(str(log_path), \"r\") as f:\n log = f.read().splitlines()"
+ },
+ {
+ "comment": "This code is parsing a log file, identifying the random seed used for each part of the log. It searches for specific metrics and extracts information related to \"R1\", \"R5\", \"R10\", \"R50\", \"MedR\", and \"MeanR\" in two modes: \"t2v\" and \"v2t\". It also differentiates between evaluation modes like \"test_run\" and \"val\".",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/log_parser.py\":25-55",
+ "content": " # keep track of the random seed used for the part of the logfile being processed\n current_seed = None\n # Regex tag for finding the seed\n seed_tag = \"Setting experiment random seed to\"\n if eval_mode == \"test_run\":\n subset = \"test\"\n else:\n subset = \"val\"\n for mode in \"t2v\", \"v2t\":\n logger.info(\"\")\n logger.info(\"----------------------------------------------------\")\n logger.info(f\"[{mode}] loaded log file with {len(log)} lines....\")\n logger.info(\"----------------------------------------------------\")\n # Search for the following metrics\n scores = {\n \"R1\": defaultdict(list),\n \"R5\": defaultdict(list),\n \"R10\": defaultdict(list),\n \"R50\": defaultdict(list),\n \"MedR\": defaultdict(list),\n \"MeanR\": defaultdict(list),\n }\n for row in log:\n if seed_tag in row:\n # Search for the log file entry describing the current random seed\n match = re.search(seed_tag + \" (\\d+)$\", row) # NOQA"
+ },
+ {
+ "comment": "This code is parsing log data, extracting relevant metrics and scores for a specific seed. It asserts that the log matches the expected format and then populates a dictionary of scores for each seed. If the log contains a specific tag, it extracts the corresponding value and adds it to the appropriate score list. Finally, it defines an empty dictionary for aggregation and raises a NotImplementedError if evaluating in geometric mean mode as it needs to be fixed for new log format.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/log_parser.py\":56-77",
+ "content": " assert len(match.groups()) == 1, \"expected a single regex match\"\n current_seed = match.groups()[0]\n if f\"{subset}_{mode}_metrics\" in row:\n tokens = row.split(\" \")\n for key in scores:\n tag = f\"{subset}_{mode}_metrics_{key}:\"\n if tag in tokens:\n pos = tokens.index(tag) + 1\n val = tokens[pos]\n val = float(val)\n assert current_seed is not None, \"failed to determine the seed\"\n scores[key][current_seed].append(val)\n agg_scores = {\"R1\": [], \"R5\": [], \"R10\": [], \"R50\": [], \"MedR\": [], \"MeanR\": []}\n # compute the best performance for a single epoch (i.e. sharing the same model\n # to compute all stats)\n geometric_stats = defaultdict(list)\n best_epochs = {}\n if eval_mode == \"geometric_mean\":\n raise NotImplementedError(\"Need to fix this for new log format\")"
+ },
+ {
+ "comment": "Code calculates scores for different seeds and metrics, then selects the best epochs based on geometric means. It then determines the final score statistic for each metric depending on the eval_mode, and appends it to agg_scores.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/log_parser.py\":78-98",
+ "content": " consider = [\"R1\", \"R5\", \"R10\"]\n seeds = list(scores[\"R1\"].keys())\n for seed in seeds:\n for metric, subdict in scores.items():\n if metric in consider:\n geometric_stats[seed].append(subdict[seed])\n gms_raw = np.array(geometric_stats[seed])\n geo_means = scipy.stats.mstats.gmean(gms_raw, axis=0)\n best_epochs[seed] = np.argmax(geo_means)\n for metric, subdict in scores.items():\n for seed, values in subdict.items():\n if eval_mode == \"test_run\":\n stat = values[0]\n elif eval_mode == \"fixed_num_epochs\":\n stat = values[fixed_num_epochs - 1]\n elif \"LSMDC\" in log_path and eval_mode == \"geometric_mean\":\n stat = values[best_epochs[seed]]\n else:\n raise ValueError(f\"unrecognised eval_mode: {eval_mode}\")\n agg_scores[metric].append(stat)"
+ },
+ {
+ "comment": "This code snippet checks if the evaluation mode is set to \"fixed_num_epochs\". If so, it logs a message indicating the fixed training length. Then, for each metric in the aggregated scores, it calculates the mean and standard deviation using numpy's `np.mean()` and `np.std()`, respectively, and logs the values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/T2VLAD/logger/log_parser.py\":100-103",
+ "content": " if eval_mode == \"fixed_num_epochs\":\n logger.info(f\"Reporting stats with fixed training length: {fixed_num_epochs}\")\n for metric, values in agg_scores.items():\n logger.info(f\"{metric}: {np.mean(values):.1f}, {np.std(values, ddof=1):.1f}\")"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/990248c3-d36d-4951-ba5c-f6108aeb20bc.json b/docs/doc/990248c3-d36d-4951-ba5c-f6108aeb20bc.json
new file mode 100644
index 000000000..88fa479cf
--- /dev/null
+++ b/docs/doc/990248c3-d36d-4951-ba5c-f6108aeb20bc.json
@@ -0,0 +1,15 @@
+{
+ "summary": "The code imports modules, registers them in the registry, and exports a list of model-building modules with functions to build these models and specific classes like ResNet and TSNHead.",
+ "details": [
+ {
+ "comment": "This code is importing various modules from different sub-directories and registers them in the registry. It also includes a license notice and a function for weight initialization.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/__init__.py\":0-23",
+ "content": "\"\"\"\n# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nfrom .backbones import ResNet\nfrom .builder import (build_backbone, build_head, build_recognizer,\n build_localizer, build_loss)\nfrom .heads import BaseHead, TSNHead, TSMRecHead\nfrom .losses import SmoothL1Loss, L1Loss\nfrom .framework.recognizers import BaseRecognizer, recognizer2d\nfrom .registry import BACKBONES, HEADS, LOSSES, RECOGNIZERS, LOCALIZERS\nfrom .weight_init import weight_init_"
+ },
+ {
+ "comment": "This code exports a list of modules for model building, including backbones, heads, recognizers, localizers, and losses. It also includes functions to build these models and specific model classes like ResNet and TSNHead.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/VideoQualityAssessment/paddlevideo/modeling/__init__.py\":25-44",
+ "content": "__all__ = [\n 'BACKBONES',\n 'HEADS',\n 'RECOGNIZERS',\n 'LOCALIZERS',\n 'LOSSES',\n 'build_recognizer',\n 'build_localizer',\n 'build_head',\n 'build_backbone',\n 'build_loss',\n 'ResNet',\n 'TSNHead',\n 'BaseHead',\n 'TSMRecHead',\n 'BaseRecognizer',\n 'Recognizer2d',\n 'SmoothL1Loss',\n 'L1Loss',\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9a349970-8717-43c2-b78c-b04b5474c921.json b/docs/doc/9a349970-8717-43c2-b78c-b04b5474c921.json
new file mode 100644
index 000000000..c7b9bfdc4
--- /dev/null
+++ b/docs/doc/9a349970-8717-43c2-b78c-b04b5474c921.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code snippet is importing the \"test_model\" function from the \"test.py\" module in the same directory and adding it to the __all__ list. The text at the beginning of the file contains license information and copyright notice.",
+ "details": [
+ {
+ "comment": "This code snippet is importing the \"test_model\" function from the \"test.py\" module in the same directory and adding it to the __all__ list. The text at the beginning of the file contains license information and copyright notice.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/applications/EIVideo/EIVideo/paddlevideo/tasks/__init__.py\":0-18",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .test import test_model\n__all__ = [\n 'test_model',\n]"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9a7eb176-dea7-494e-87cd-4eff0b41bece.json b/docs/doc/9a7eb176-dea7-494e-87cd-4eff0b41bece.json
new file mode 100644
index 000000000..0868bb748
--- /dev/null
+++ b/docs/doc/9a7eb176-dea7-494e-87cd-4eff0b41bece.json
@@ -0,0 +1,40 @@
+{
+ "summary": "The code imports libraries, defines functions for parsing arguments and log files, checks for names, reads a log file, stores results in \"parser_results\", loads ground truth from multiple files, and compares log results with ground truth for testing.",
+ "details": [
+ {
+ "comment": "This code imports necessary libraries and defines functions for parsing command-line arguments, running shell commands, and retrieving results from log files. It uses ArgumentParser to handle command line arguments, subprocess to execute shell commands, and os to check file existence.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":0-39",
+ "content": "import numpy as np\nimport os\nimport subprocess\nimport json\nimport argparse\nimport glob\ndef init_args():\n parser = argparse.ArgumentParser()\n # params for testing assert allclose\n parser.add_argument(\"--atol\", type=float, default=1e-3)\n parser.add_argument(\"--rtol\", type=float, default=1e-3)\n parser.add_argument(\"--gt_file\", type=str, default=\"\")\n parser.add_argument(\"--log_file\", type=str, default=\"\")\n parser.add_argument(\"--precision\", type=str, default=\"fp32\")\n return parser\ndef parse_args():\n parser = init_args()\n return parser.parse_args()\ndef run_shell_command(cmd):\n p = subprocess.Popen(cmd,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n shell=True)\n out, err = p.communicate()\n if p.returncode == 0:\n return out.decode('utf-8')\n else:\n return None\ndef parser_results_from_log_by_name(log_path, names_list):\n if not os.path.exists(log_path):\n raise ValueError(\"The log file {} does not exists!\".format(log_path))"
+ },
+ {
+ "comment": "This code checks if there are any names in the \"names_list\" and reads a log file at the specified \"log_path\". If the file contains \"python_infer\", it parses the python inference results, while for other log files, it parses C++ inference results. It stores the results in the \"parser_results\" dictionary with names as keys and corresponding values as either integers or floats.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":41-63",
+ "content": " if names_list is None or len(names_list) < 1:\n return []\n parser_results = {}\n lines = open(log_path, 'r').read().splitlines()\n if 'python_infer' in log_path: # parse python inference\n for line in lines:\n split_items = line.replace('\\t', ' ')\n split_items = split_items.split(' ')\n split_items = [item for item in split_items if len(item) > 0]\n for name in names_list:\n if name in line:\n if '.' in split_items[-1]:\n parser_results[name] = float(split_items[-1])\n else:\n parser_results[name] = int(split_items[-1])\n else: # parse cpp inference\n for line in lines:\n split_items = line.replace('\\t', ' ')\n split_items = split_items.split(' ')\n split_items = [item for item in split_items if len(item) > 0]\n if all([(name + ':') in split_items for name in names_list]):\n # print(split_items)"
+ },
+ {
+ "comment": "This code defines a function `load_gt_from_file` that reads and parses the contents of a log file. It first checks if the file exists, then opens it in read mode. For each line containing 'top-1 class' or 'top-1 score', it extracts the class and score values, storing them as key-value pairs in `parser_gt`. If the file is not found, it raises a ValueError with an error message. The code also handles dictionaries with string keys, allowing for easy integration into larger programs.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":64-88",
+ "content": " parser_results['class'] = int(split_items[2])\n parser_results['score'] = float(split_items[-1])\n return parser_results\ndef load_gt_from_file(gt_file):\n if not os.path.exists(gt_file):\n raise ValueError(\"The log file {} does not exists!\".format(gt_file))\n with open(gt_file, 'r') as f:\n data = f.readlines()\n f.close()\n parser_gt = {}\n for line in data:\n if 'top-1 class' in line:\n split_items = line.replace('\\t', ' ')\n split_items = split_items.split(' ')\n split_items = [item for item in split_items if len(item) > 0]\n parser_gt['top-1 class'] = int(split_items[-1])\n elif 'top-1 score' in line:\n split_items = line.replace('\\t', ' ')\n split_items = split_items.split(' ')\n split_items = [item for item in split_items if len(item) > 0]\n parser_gt['top-1 score'] = float(split_items[-1])\n elif \"score\" in line and 'segment' in line:\n location_dict = eval(line)"
+ },
+ {
+ "comment": "The code defines three functions:\n1. `load_gt_from_file` loads ground truth data from a file, handling both the cases when each line contains location details or class and score information.\n2. `collect_predict_from_logs` collects predict results from logs based on given key list.\n3. `load_gt_from_txts` loads ground truth collections from multiple files (fp32, fp16, int8), organizing them under corresponding keys in a dictionary.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":89-117",
+ "content": " parser_gt[f\"score_{len(parser_gt)}\"] = location_dict['score']\n parser_gt[f\"segment_{len(parser_gt)}\"] = location_dict['segment']\n elif \"class:\" in line and \"score:\" in line:\n split_items = line.replace('\\t', ' ')\n split_items = split_items.split(' ')\n split_items = [item for item in split_items if len(item) > 0]\n parser_gt['class'] = int(split_items[2])\n parser_gt['score'] = float(split_items[-1])\n return parser_gt\ndef load_gt_from_txts(gt_file):\n gt_list = glob.glob(gt_file)\n gt_collection = {}\n for gt_f in gt_list:\n gt_dict = load_gt_from_file(gt_f)\n basename = os.path.basename(gt_f)\n if \"fp32\" in basename:\n gt_collection[\"fp32\"] = [gt_dict, gt_f]\n elif \"fp16\" in basename:\n gt_collection[\"fp16\"] = [gt_dict, gt_f]\n elif \"int8\" in basename:\n gt_collection[\"int8\"] = [gt_dict, gt_f]\n else:\n continue\n return gt_collection\ndef collect_predict_from_logs(log_path, key_list):"
+ },
+ {
+ "comment": "The code reads logs from specified file paths and compares the results with ground truth data for testing purposes. It uses numpy's assert_allclose function to validate the accuracy of the predicted results against the ground truth. The usage example provides command line options to compare Python and C++ inferencing results.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":118-145",
+ "content": " log_list = glob.glob(log_path)\n pred_collection = {}\n for log_f in log_list:\n pred_dict = parser_results_from_log_by_name(log_f, key_list)\n key = os.path.basename(log_f)\n pred_collection[key] = pred_dict\n return pred_collection\ndef testing_assert_allclose(dict_x, dict_y, atol=1e-7, rtol=1e-7):\n for k in dict_x:\n np.testing.assert_allclose(np.array(dict_x[k]),\n np.array(dict_y[k]),\n atol=atol,\n rtol=rtol)\nif __name__ == \"__main__\":\n # Usage example:\n # test python infer:\n ## python3.7 test_tipc/compare_results.py --gt_file=./test_tipc/results/PP-TSM/*.txt --log_file=./test_tipc/output/PP-TSM/python_infer_*.log\n # test cpp infer:\n ## python3.7 test_tipc/compare_results.py --gt_file=./test_tipc/results/PP-TSM_CPP/*.txt --log_file=./test_tipc/output/PP-TSM_CPP/cpp_infer_*.log\n args = parse_args()\n gt_collection = load_gt_from_txts(args.gt_file)"
+ },
+ {
+ "comment": "Iterates through the log files, compares \"fp32\", \"fp16\" and \"int8\" results with ground truth, uses testing_assert_allclose to check for consistency and prints success/failure messages.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":146-169",
+ "content": " key_list = gt_collection[\"fp32\"][0].keys()\n pred_collection = collect_predict_from_logs(args.log_file, key_list)\n for filename in pred_collection.keys():\n if \"fp32\" in filename:\n gt_dict, gt_filename = gt_collection[\"fp32\"]\n elif \"fp16\" in filename:\n gt_dict, gt_filename = gt_collection[\"fp16\"]\n elif \"int8\" in filename:\n gt_dict, gt_filename = gt_collection[\"int8\"]\n else:\n continue\n pred_dict = pred_collection[filename]\n try:\n testing_assert_allclose(gt_dict,\n pred_dict,\n atol=args.atol,\n rtol=args.rtol)\n print(\n \"Assert allclose passed! The results of {} and {} are consistent!\"\n .format(filename, gt_filename))\n except Exception as E:\n print(E)\n raise ValueError(\n \"The results of {} and the results of {} are inconsistent!\"."
+ },
+ {
+ "comment": "This line of code formats the filename and ground truth filename for comparison purposes in the context of image or video analysis.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/test_tipc/compare_results.py\":170-170",
+ "content": " format(filename, gt_filename))"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9aea14bf-d2ff-4d11-99d2-f8c6e9e4ece3.json b/docs/doc/9aea14bf-d2ff-4d11-99d2-f8c6e9e4ece3.json
new file mode 100644
index 000000000..f39237054
--- /dev/null
+++ b/docs/doc/9aea14bf-d2ff-4d11-99d2-f8c6e9e4ece3.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is a Python module with the license and copyright information. It imports a class named PaddleVideo from the tools package, and defines its availability as part of the __all__ list.",
+ "details": [
+ {
+ "comment": "This code is a Python module with the license and copyright information. It imports a class named PaddleVideo from the tools package, and defines its availability as part of the __all__ list.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/__init__.py\":0-15",
+ "content": "# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n__all__ = ['PaddleVideo']\nfrom .tools import PaddleVideo"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9aee4f8d-0e16-441a-8b8e-242b67a6e02a.json b/docs/doc/9aee4f8d-0e16-441a-8b8e-242b67a6e02a.json
new file mode 100644
index 000000000..c957fdc8b
--- /dev/null
+++ b/docs/doc/9aee4f8d-0e16-441a-8b8e-242b67a6e02a.json
@@ -0,0 +1,10 @@
+{
+ "summary": "This code is the initialization file for PaddleVideo's tasks module. It imports functions from various task-specific modules and adds them to `__all__` for export. The code also includes a license notice, copyright information, and a disclaimer.",
+ "details": [
+ {
+ "comment": "This code is the initialization file for PaddleVideo's tasks module. It imports functions from various task-specific modules and adds them to `__all__` for export. The code also includes a license notice, copyright information, and a disclaimer.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/paddlevideo/tasks/__init__.py\":0-19",
+ "content": "# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom .train import train_model\nfrom .test import test_model\nfrom .train_dali import train_dali\nfrom .train_multigrid import train_model_multigrid\n__all__ = ['train_model', 'test_model', 'train_dali', 'train_model_multigrid']"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/docs/doc/9b615ae9-82b6-4320-aff8-c1adac9d33d2.json b/docs/doc/9b615ae9-82b6-4320-aff8-c1adac9d33d2.json
new file mode 100644
index 000000000..f76b07166
--- /dev/null
+++ b/docs/doc/9b615ae9-82b6-4320-aff8-c1adac9d33d2.json
@@ -0,0 +1,65 @@
+{
+ "summary": "The code processes video frames, performs inference using an AI model and measures processing times. It preprocesses data in batches and utilizes TensorRT with GPU optimizations and MKLDNN support for efficiency.",
+ "details": [
+ {
+ "comment": "The code initializes variables and performs batch size operations. It copies the batch of frames and resizes it to accommodate for multiple segments per frame batch. The times vector will store execution time values.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/video_rec.cpp\":0-25",
+ "content": "// Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n#include \nnamespace PaddleVideo\n{\n void VideoRecognizer::Run(const std::vector &frames_batch_path, const std::vector > &frames_batch, std::vector *times)\n {\n // Copy parameters to the function\n int real_batch_num = frames_batch.size();\n std::vector srcframes(real_batch_num * this->num_seg, cv::Mat());\n for (int i = 0; i < real_batch_num; ++i)"
+ },
+ {
+ "comment": "This code preprocesses video frames for model inference. It copies frames from source to destination, resizes them using a scale operation, and performs center cropping. The number of views is set to 1 if the model name is \"ppTSM\". The preprocessing steps include scaling and centering cropping to ensure the frames are properly formatted for inference.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/video_rec.cpp\":26-54",
+ "content": " {\n for (int j = 0; j < this->num_seg; ++j)\n {\n frames_batch[i][j].copyTo(srcframes[i * this->num_seg + j]);\n }\n }\n auto preprocess_start = std::chrono::steady_clock::now();\n /* Preprocess */\n std::vector resize_frames;\n std::vector crop_frames;\n std::vector input;\n int num_views = 1;\n if (this->inference_model_name == \"ppTSM\")\n {\n num_views = 1;\n // 1. Scale\n resize_frames = std::vector(real_batch_num * this->num_seg, cv::Mat());\n for (int i = 0; i < real_batch_num; ++i)\n {\n for (int j = 0; j < this->num_seg; ++j)\n {\n this->scale_op_.Run(srcframes[i * this->num_seg + j], resize_frames[i * this->num_seg + j], this->use_tensorrt_, 256);\n }\n }\n // 2. CenterCrop\n crop_frames = std::vector(real_batch_num * num_views * this->num_seg, cv::Mat());"
+ },
+ {
+ "comment": "This code performs image preprocessing and conversion on video frames before feeding them into a neural network. It first resizes, centers, and crops the video frames using `centercrop_op_.Run()`. Then it normalizes the frames in an in-place operation using `normalize_op_.Run()`, with the mean and scale values provided. Finally, it converts the normalized frames into a single array using the dimensions from the first frame, and stores them in the 'input' vector.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/video_rec.cpp\":55-79",
+ "content": " for (int i = 0; i < real_batch_num; ++i)\n {\n for (int j = 0; j < this->num_seg; ++j)\n {\n this->centercrop_op_.Run(resize_frames[i * this->num_seg + j], crop_frames[i * this->num_seg + j], this->use_tensorrt_, 224);\n }\n }\n // 3. Normalization(inplace operation)\n for (int i = 0; i < real_batch_num; ++i)\n {\n for (int j = 0; j < this->num_seg; ++j)\n {\n for (int k = 0; k < num_views; ++k)\n {\n this->normalize_op_.Run(&crop_frames[i * num_views * this->num_seg + j * num_views + k], this->mean_, this->scale_, this->is_scale_);\n }\n }\n }\n // 4. Image2Array\n int rh = crop_frames[0].rows;\n int rw = crop_frames[0].cols;\n int rc = crop_frames[0].channels();\n input = std::vector(real_batch_num * num_views * this->num_seg * crop_frames[0].rows * crop_frames[0].cols * rc, 0.0f);"
+ },
+ {
+ "comment": "The code is iterating over real_batch_num number of batches and num_seg segments within each batch. For each segment, it's applying a set of operations (permute, scale, TenCrop) to a series of frames. These operations are used for data preprocessing before inputting into an inference model.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/video_rec.cpp\":80-104",
+ "content": " for (int i = 0; i < real_batch_num; ++i)\n {\n for (int j = 0; j < this->num_seg; ++j)\n {\n for (int k = 0; k < num_views; ++k)\n {\n this->permute_op_.Run(&crop_frames[i * num_views * this->num_seg + j * num_views + k], input.data() + (i * num_views * this->num_seg + j * num_views + k) * (rh * rw * rc));\n }\n }\n }\n }\n else if(this->inference_model_name == \"ppTSN\")\n {\n num_views = 10;\n // 1. Scale\n resize_frames = std::vector(real_batch_num * this->num_seg, cv::Mat());\n for (int i = 0; i < real_batch_num; ++i)\n {\n for (int j = 0; j < this->num_seg; ++j)\n {\n this->scale_op_.Run(srcframes[i * this->num_seg + j], resize_frames[i * this->num_seg + j], this->use_tensorrt_, 256);\n }\n }\n // 2. TenCrop"
+ },
+ {
+ "comment": "This code performs image preprocessing for video frames. It initializes a vector of crop_frames, iterates through real_batch_num and num_seg to run resizing and cropping operations on each frame using tencrop_op_. Next, it applies normalization inplace operation on each frame using normalize_op_. Finally, it converts the processed frames into an array by extracting rows and columns size from the first crop_frame.",
+ "location": "\"/media/root/Prima/works/PaddleVideo/docs/src/deploy/cpp_infer/src/video_rec.cpp\":105-128",
+ "content": " crop_frames = std::vector ![](\"images/model.png\")
![](\"images/show.gif\")
![](\"images.png\")
![](\"../../images/skeleton_example.png\")
![](\"../../../images/BMN.png\")
![](\"../../../images/yowo.jpg\")
\n![\"Horse](\"../../../images/horse_riding.gif\")
\n![](\"../../../images/ctrgcn.jpg\")
![](\"../../../images/tokenshift_structure.png\")
\n![](\"../../../images/tsm_architecture.png\")