Dirty changes to add support for Rocm, sam, mediapipe, mask-rcnn, skeleton-tracker, polygon-tracker etc. - #1
Open
Ankk98 wants to merge 50 commits into
Open
Dirty changes to add support for Rocm, sam, mediapipe, mask-rcnn, skeleton-tracker, polygon-tracker etc.#1Ankk98 wants to merge 50 commits into
Ankk98 wants to merge 50 commits into
Conversation
Release v2.45.0
Release v2.46.0
Release v2.46.1
Release v2.47.0
Release v2.48.0
Release v2.48.1
Release v2.49.0
Release v2.50.0
…ntation - Introduced `deploy_egocentric_models.sh` for streamlined deployment of hand-pose, semantic segmentation, and instance segmentation models on AMD/ROCm devices. - Created `index.md` to document serverless model implementations and deployment strategies. - Added detailed plans for AMD GPU acceleration and CPU deployment for MMPose and YOLO11 models. - Implemented new Nuclio function configurations for MediaPipe Pose and YOLO11 Pose, enhancing model accessibility and performance. - Updated existing MMPose function to support Python 3.9 and improved dependency management. - Enhanced documentation for model capabilities and deployment processes, ensuring clarity for users.
- Added a complete MediaPipe Pose Detection Service, including pose and hand tracking capabilities optimized for egocentric vision. - Introduced RESTful API endpoints for health checks and pose detection. - Created integration scripts for seamless CVAT functionality, including automatic function registration and health checks. - Developed Docker support with a comprehensive `docker-compose.yml` for easy deployment. - Included setup scripts for virtual environment and Docker configurations, along with detailed README documentation for user guidance. - Configured CVAT integration with a `cvat-config.json` for simplified setup and usage. - Ensured comprehensive logging and error handling throughout the service for improved reliability.
- Add .gitignore for serverless directory - Update index.md with testing framework documentation - Fix MediaPipe ROCm configuration for Python 3.10 - Add health monitoring to docker-compose - Add essential testing scripts (real images only) - Remove synthetic testing and complex analysis tools Testing framework includes: - Real image downloading from Unsplash API - Primary testing tool for all models (SAM, Detectron2, MediaPipe) - Automated test orchestration and result analysis - Production-ready, focused on real-world validation All working models verified: SAM (100%), Detectron2 (100%), MediaPipe (100%)
- Added MediaPipe as a built-in function in the LambdaGateway, enabling real-time pose and hand tracking for egocentric vision. - Implemented a new method to handle MediaPipe service invocation directly, improving integration with CVAT's auto-annotation system. - Updated documentation to reflect MediaPipe's integration, highlighting its features and API endpoints. - Enhanced error handling for MediaPipe service calls to ensure reliability and logging of failures.
- Introduced SAM Auto for automatic object segmentation, enhancing the existing SAM capabilities. - Updated deployment script to include SAM Auto option and configuration. - Added new YAML configuration for SAM Auto deployment. - Created main handler and model handler for SAM Auto, enabling seamless integration with the existing framework. - Developed comprehensive testing script for SAM Auto, including automated evaluation and performance metrics. - Updated documentation to reflect the new SAM Auto model and its testing procedures.
…pport - Added SAM Auto as a built-in detector function in the LambdaGateway, enabling automatic object segmentation. - Implemented methods for invoking SAM Auto segmentation and handling its configuration. - Updated deployment scripts to support custom configuration files for model deployment. - Improved service management scripts to handle multiple process IDs and ensure proper service status checks. - Introduced new JSON configuration files for SAM Auto, detailing its specifications and deployment settings. - Enhanced the MediaPipe Pose Service with port availability checks to prevent startup conflicts.
- Updated multiple Nuclio function YAML files to include GPU resource limits, setting `amd.com/gpu: 1` for enhanced performance on AMD ROCm devices. - This change applies to the following models: SAM, Detectron2, MediaPipe Pose, Open3D, and YOLO11 Pose, ensuring consistent resource allocation across deployments.
- Added support for setting up, starting, and stopping a standalone MediaPipe service within the deployment script. - Updated command-line options to include `--mediapipe-service` for service deployment and `--stop` for stopping all services. - Enhanced documentation to reflect new MediaPipe service management features and usage examples. - Improved logging for service status checks and management commands.
…T auto-annotation - Add MediaPipe standalone service with direct CVAT backend integration - Implement 83-keypoint skeleton detection (33 body + 42 hand + 8 palm joints) - Create complete skeleton label configuration for CVAT import - Fix skeleton rendering issues with proper data attributes - Update documentation to reflect working integration - Remove deprecated Nuclio and ROCm implementations to avoid confusion The MediaPipe function now appears in CVAT's auto-annotation dropdown and successfully detects and annotates human poses with detailed hand tracking for egocentric video analysis.
This commit fixes critical issues preventing MediaPipe pose annotations from rendering correctly in CVAT canvas and ensures proper backend integration. Backend Changes (cvat/apps/lambda_manager/views.py): - Add complete MediaPipe function spec with all 57 sublabels (33 body + 24 hand keypoints) - Implement _invoke_mediapipe() method for direct FastAPI service invocation - Add MediaPipe-specific handler in invoke() method - Fix undefined invoke_mode variable by reading from settings - Support both Docker (host.docker.internal) and host (localhost) environments Frontend Changes (cvat-canvas/src/typescript/): - Fix skeleton rendering in canvasView.ts by adding fallback matching via data-node-id - Handle SVG template ID mismatch (sequential IDs 0-56 vs database IDs 217-273) - Add defensive checks to prevent crashes when template elements are not found - Update drawHandler.ts with robust data-label-id matching The fix ensures that when SVG templates use sequential IDs but database sublabels have different IDs, the frontend can still match elements by sublabel name using the data-node-id attribute. Documentation: - Update serverless/index.md to reflect MediaPipe production-ready status - Update keypoint count from 83 to 57 (corrected: 33 body + 24 hand) - Mark MediaPipe as fully tested and skeleton rendering as fixed Testing: - Verified skeleton annotations render correctly in CVAT job view - All 57 keypoints display properly on canvas - Auto-annotation workflow end-to-end tested with egocentric videos Fixes: MediaPipe skeleton annotations not rendering in CVAT canvas
This commit fixes MediaPipe auto-annotation for egocentric videos by addressing low detection rates, incorrect keypoint locations, and hand detection failures. The changes optimize MediaPipe for first-person perspective videos where hands are the primary focus. - Implemented hand prioritization to focus on hands closest to frame center (critical for egocentric videos) - Fixed coordinate transformation issues (RGB format, coordinate clamping) - Added hand-only skeleton configuration for CVAT projects - Relaxed filtering logic to allow hand-only detections (3+ hand keypoints vs 2+ total keypoints) Technical changes: - Modified MediaPipe service (app.py) with egocentric optimizations: * Lower confidence thresholds for pose and hand detection * Hand prioritization function based on center proximity * RGB image format conversion (MediaPipe expects RGB, not BGR) * Coordinate clamping to valid [0,1] range * Improved filtering logic for hand-only scenarios - Added CVAT skeleton config for hand-only tracking (42 keypoints) This fixes the issue where only ~10% of frames were being annotated and annotations appeared in incorrect locations. Detection rate has improved significantly and annotations now correctly align with hand positions in egocentric videos.
Implement clean Mask R-CNN R50 for instance segmentation with proper CVAT mask format support. Masks now render correctly in CVAT canvas. Key changes: - Add mask_rcnn_r50_rocm/ with clean implementation - Fix mask format: use flattened pixels (matches OpenVINO to_cvat_mask) - CVAT backend converts pixels to RLE automatically - Restore RetinaNet to original object detection (bounding boxes) - Add configuration files for COCO and egocentric masks - Update deployment script with --mask-rcnn option - Update documentation with production-ready status - Remove deprecated mask_rcnn_r50/ directory
…essary files Update and add comprehensive test scripts to validate Mask R-CNN instance segmentation with correct CVAT mask format (flattened pixels). Key updates: - Update test_detectron2_masks.py to validate flattened pixels format - Add _decode_flattened_pixels() to test_semantic_and_pose_integration.py - Update test_frontend_mask_rendering.py with format flow documentation - Fix test_detectron2_egocentric.py to handle Mask R-CNN response format - Add check_cvat_annotations.py for CVAT annotation inspection - Add comprehensive testing documentations
…notation analysis This commit introduces a new debug script (`debug_annotations.py`) that analyzes MediaPipe annotations against CVAT annotations for egocentric videos. The script includes functionality for downloading frames, comparing annotations, and visualizing results. Additionally, a usage guide (`DEBUG_SCRIPT_USAGE.md`) is added to provide instructions on how to utilize the script effectively. Key changes: - Implemented `debug_annotations.py` for detailed annotation comparison and analysis. - Added `COMBINED_CONFIG_COMPATIBILITY.md` to document compatibility between MediaPipe hands and Detectron2 masks. - Created `test_coordinates.py` to validate MediaPipe coordinate transformations. - Removed outdated shell script for semantic pose testing. This enhances the debugging capabilities for MediaPipe integration with CVAT, ensuring better detection accuracy and coordinate validation.
- Updated svg to also show edges between the points - Added a python script to validate the configs for labels
- Updated docker-compose configuration to manage MediaPipe service as a Docker container. - Refactored deployment script to streamline MediaPipe function deployment and health checks. - Added Nuclio function for MediaPipe Pose + Hands detection, optimizing for egocentric video analysis. - Pre-downloaded MediaPipe models during Docker build to avoid runtime issues. - Improved logging and error handling in the proxy function for better debugging and reliability. This update ensures a more robust integration of MediaPipe with CVAT, enhancing pose and hand detection capabilities for egocentric videos.
…ation injection - Updated `deploy_egocentric_models.sh` to prepare `function.yaml` by injecting skeleton spec from a JSON file. - Introduced `prepare_function_yaml.py` to handle the extraction and injection of the skeleton spec into the YAML configuration. - Added `mediapipe-skeletons-raw-editor.json` to serve as the source for skeleton configurations, ensuring compatibility with CVAT. - Improved logging and error handling during the preparation process for better debugging. This update streamlines the deployment process for MediaPipe functions, ensuring that the skeleton configurations are correctly integrated for enhanced pose and hand detection capabilities in egocentric videos.
…gration - Updated skeleton handling in `app.py` to create distinct `person-skeleton` and `hands-skeleton` structures for improved clarity in pose and hand detection. - Modified `cvat-integration.py` to reflect the new `person-skeleton` label, ensuring compatibility with CVAT. - Added comprehensive documentation in `EGOCENTRIC_ANNOTATION_GUIDE.md` to guide users on the new skeleton labeling system and its usage in egocentric video annotations. - Introduced new JSON files for `mediapipe-hands-skeleton` and `mediapipe-person-skeleton` to standardize skeleton configurations. - Enhanced validation scripts to check for the correct skeleton configurations. This update streamlines the annotation process for egocentric videos, allowing for better differentiation between body and hand keypoints in the MediaPipe service.
…deos Add support for a new \"hands-shoulders-skeleton\" label that includes shoulders, elbows, wrists, and all hand keypoints (46 total keypoints). This label is specifically optimized for egocentric videos where only the upper body and hands are typically visible. Changes: - Add generate_hands_shoulders_skeleton() function to generate skeleton config with 6 upper body keypoints (shoulders, elbows, wrists) + 40 hand keypoints - Fix wrist collision by ensuring wrist appears only once (from upper body, not duplicated in hand positions) - Update prepare_function_yaml.py to extract all three skeleton labels (hands-shoulders, person, hands) from the config - Update MediaPipe service (app.py) to generate hands-shoulders-skeleton annotations alongside person-skeleton and hands-skeleton - Update EGOCENTRIC_ANNOTATION_GUIDE.md to document the new label as the recommended option for egocentric videos - Regenerate skeleton configuration files including the new label The hands-shoulders-skeleton label provides a focused annotation option for egocentric videos, avoiding unnecessary lower body keypoints (hips, knees, ankles) that are rarely visible in first-person perspective. All three skeleton labels are now available in CVAT: - hands-shoulders-skeleton (46 keypoints) - Recommended for egocentric - person-skeleton (57 keypoints) - Full body + hands - hands-skeleton (42 keypoints) - Hands only
This update introduces server-side skeleton tracking capabilities for video sequences using MediaPipe. Key changes include: - Added `enable_skeleton_tracking` and `frame_number` fields to the `FunctionCallRequestSerializer` for tracking configuration. - Implemented `SkeletonTrackBuilder` to manage skeleton track creation and submission to CVAT. - Updated `LambdaJob` to utilize the new skeleton tracking feature when enabled. - Enhanced MediaPipe service to support video mode detection with timestamp handling. - Updated deployment scripts to include a new tracker function for MediaPipe. These enhancements improve the annotation process for egocentric videos, allowing for more accurate tracking of skeletons across frames.
…validation This update improves the skeleton tracking feature by adding validation for tracking requests and refining the skeleton conversion process. Key changes include: - Introduced `validate_skeleton_tracking_request` function to ensure proper conditions for enabling skeleton tracking. - Updated `SkeletonTrackBuilder` to handle skeletons more effectively, including keeping all skeletons during track building. - Enhanced the `DetectionResultConverter` to support the new skeleton tracking logic. - Added `enable_skeleton_tracking` option in the frontend to control tracking behavior based on model capabilities and mapping. These enhancements ensure more robust skeleton tracking for video tasks, improving the overall annotation process.
…verter This update introduces a new static method `_mask_to_polygon` in the `DetectionResultConverter` class, which converts mask points to polygon points. Key changes include: - Implementation of mask validation and conversion logic, ensuring proper handling of mask formats. - Integration of the new method into the `convert` function to support mask-to-polygon conversion when specified. - Enhanced error handling and logging for conversion failures. These enhancements improve the annotation process by allowing for more accurate shape representation from mask annotations.
This update enhances the SAM plugin by introducing support for multiple model IDs, allowing for more flexible model management. Key changes include: - Added `modelIDs` array to store multiple model identifiers. - Updated model ID checks to utilize the new `modelIDs` array for initialization and processing. - Improved click handling by adding checks for the presence of bounding boxes and points before processing. These enhancements improve the plugin's functionality and robustness in handling different models and user interactions.
…acking functionality This update introduces several key enhancements to the annotation process: - Added functionality to convert mask shapes to polygon shapes, ensuring compatibility with tracking features. - Improved the tracking process by allowing the selection of supported shape types from trackers, preventing duplicate shapes during tracking. - Enhanced event handling to prevent shape creation when tracking is active, reducing unnecessary clutter on the canvas. - Updated UI components to support the new conversion feature, including a button for converting masks to polygons in the object sidebar. These changes improve the overall user experience and accuracy of annotations in the CVAT application.
… Mask R-CNN This update adds a new module for polygon tracking, enabling the association of mask annotations across video frames. Key changes include: - Implementation of the `PolygonTrackBuilder` class for building and submitting polygon tracks. - Integration of polygon tracking into the LambdaJob processing flow, allowing for server-side tracking of polygon annotations. - Addition of a new `enable_polygon_tracking` field in serializers and UI components to control tracking behavior. - Deployment scripts updated to include a new Mask R-CNN tracker function for polygon tracking. These enhancements improve the annotation process by allowing for more accurate tracking of objects across frames, enhancing user experience and annotation quality in CVAT.
…onality This commit introduces a new test script for validating the skeleton tracking feature using MediaPipe Pose and Hands. Key additions include: - Comprehensive testing of skeleton tracking capabilities, including automatic annotation, track creation, and validation of detection associations across frames. - Implementation of a command-line interface for specifying parameters such as CVAT URL, username, password, task ID, and detection function. - Detailed logging of the testing process, including authentication, task fetching, and annotation results. These enhancements facilitate the verification of skeleton tracking functionality, ensuring robust performance in video annotation tasks.
… UI updates This commit introduces several improvements to the tracking capabilities in the LambdaJob and the UI components: - Added logging statements to track the usage of skeleton and polygon tracking modes, providing better debugging information. - Updated the LambdaJob to fall back to standard detection when skeleton or polygon tracking is disabled, with corresponding log messages. - Introduced a new UI switch for enabling skeleton tracking, enhancing user control over tracking options. - Adjusted the logic to ensure that the skeleton tracking state is correctly reflected in the request sent to the backend. These changes improve the overall user experience and provide better insights into the tracking process during video annotation tasks.
This update enhances the DetectorRunner component by introducing a conditional visibility for the tracking toggle. The old generic "Enable tracking" toggle is hidden when specific tracking modes (skeleton or polygon) are available, reducing user confusion. The logic for sending tracking parameters to the backend is also updated to ensure that the `enable_tracking` flag is only included when the toggle is visible. These changes improve the user experience by streamlining the tracking options presented to users.
…side detection - Add outside=True markers when skeleton tracks end before last frame - Fix Track.getPosition() to detect when current frame is after track's last frame - Improve SkeletonTrack filtering logic in Collection.get() - Fixes issue where 8 skeleton tracks were shown on frame 225 instead of 1 Backend changes: - skeleton_tracker.py: Add termination markers for main tracks and elements Frontend changes: - annotations-objects.ts: Detect track end in getPosition() methods - annotations-collection.ts: Improve skeleton track filtering logic Test improvements: - test_skeleton_tracking.py: Better label pagination and error handling
…TrackBuilder - Updated methods to use patching for job and task data submissions instead of direct data insertion. - Enhanced logging for successful submissions of polygon and skeleton tracks to improve traceability. These changes streamline the data handling process in the tracking modules, ensuring better integration with the backend.
- Updated the LambdaFunction class to include supported_shape_types for all function types that have them, improving response consistency. - Incremented version number in function-rocm.yaml and refined the description for clarity. - Added new parameters for pedestrian dimensions in function-rocm.yaml to enhance detection capabilities. - Improved installation command in function-rocm.yaml to include scikit-learn for better functionality. - Introduced confidence computation and heading estimation methods in model_handler.py to enhance detection accuracy. These changes collectively improve the functionality and clarity of the Lambda manager and detector configurations.
…ormance improvements - Updated function-rocm.yaml to include annotations for display name, version, type, and supported shape types, improving clarity and usability in the CVAT UI. - Enhanced main.py with detailed docstrings for initialization and request handling, clarifying the function's purpose and expected input/output. - Expanded model_handler.py with comprehensive documentation on the SIT Point Cloud Detector, including algorithm overview, parameters, and performance characteristics. - Improved detection logic with refined clustering parameters and confidence scoring, optimizing pedestrian detection in 3D point clouds. These changes collectively enhance the functionality, documentation, and performance of the SIT Point Cloud Detector, facilitating better integration and usability in social navigation scenarios.
…intCloudDetector - Adjusted the default confidence threshold from 0.2 to 0.10 for improved detection sensitivity. - Enhanced documentation for cuboid annotation format, clarifying the required structure and padding for CVAT compatibility. - Refined confidence computation by incorporating size and shape factors, leading to more robust pedestrian detection in LiDAR data. These changes collectively improve the accuracy and clarity of the PointCloudDetector's functionality, facilitating better performance in social navigation scenarios.
… updates - Introduced a comprehensive deployment summary for FCAF3D, detailing setup, configuration, and performance specifications. - Updated deployment scripts to support individual model actions (deploy/stop) and improved error handling for missing configurations. - Added new Dockerfile and README for FCAF3D, ensuring clear instructions for building and deploying the model in a ROCm environment. - Enhanced the main.py and function-rocm.yaml with improved logging, configuration management, and CVAT integration for better usability and performance. These changes collectively enhance the functionality, documentation, and deployment process for the FCAF3D model, facilitating better integration with CVAT for 3D point cloud annotation tasks.
…nd configuration management - Added a symlink for the Nuclio Python executable in the Dockerfile to ensure proper environment setup. - Updated function-rocm.yaml to set the working directory for better path resolution and modified the PYTHONPATH for improved module accessibility. - Enhanced main.py with detailed logging for initialization, model loading, and inference processes, providing better insights into the system's state and performance. These changes collectively improve the deployment and operational efficiency of the FCAF3D model, facilitating better integration and usability in 3D point cloud annotation tasks.
Convert FCAF3D from direct Nuclio function to proxy pattern: - Add FCAF3D service container with FastAPI web service - Create lightweight Nuclio proxy function that forwards requests - Update deployment script to manage both service and function lifecycle - Fix model loading paths and configuration for MMDetection3D - Add proper error handling and health checks - Clean up obsolete config files This follows the same pattern as MediaPipe implementation for better reliability and resource management. The proxy handles CVAT requests and forwards them to the FCAF3D service running in a dedicated container. Usage: FCAF3D_USE_ROCM=1 ./deploy_rocm_pointcloud_models.sh fcaf3d deploy
…ging - Modified Lambda function to convert PCD format to BIN for compatibility with MMDetection3D. - Added detailed logging for point cloud data processing, including format detection and conversion steps. - Updated Dockerfile to include Open3D for PCD handling and improved health check configurations. - Introduced a new configuration file for ScanNet dataset settings, enhancing model training and evaluation capabilities. These changes collectively improve the robustness and usability of the FCAF3D model in 3D point cloud annotation tasks.
…logging (Not working without CUDA, not even on CPU) - Added checks for MinkowskiEngine availability during service initialization, ensuring proper error handling if not found. - Updated Dockerfile to install MinkowskiEngine with a CPU-only flag to avoid GPU compilation issues. - Introduced a new modified main.py file for enhanced functionality and logging during inference. - Added a MinkowskiEngine-compatible detector implementation to support CPU-only operations. These changes collectively improve the robustness and usability of the FCAF3D model for 3D object detection tasks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To-dos: