Repository navigation
Releases
Releases · OSLabTools/nicetoolbox
v0.3.2
- New installer scripts with an interactive detector selection. The installer now uses uv instead of conda and venv for all detector environments. The installation is 4x faster and 22% smaller than in the previous release.
- Faster runs and smaller outputs. On the example sequence, the detectors need 31% less time and generated 7x smaller output than in the previous release.
- Configurable frame extraction with optional hardware-accelerated decoding.
- MediaPipe Face Landmarker (
mediapipe_face_landmarks) for 3D facial landmarks and facial blendshape coefficients. - New person detection algorithms from MMDetection.
- New subject identity system for video and audio. In
dataset_properties.toml, each camera and audio track declares a subject rule (vsplit,hsplit,assign,bbox,audio_range,path,none). All detectors use the subject names of the sequence. See the subject rules wiki. - pyannote (
pyannote) - speaker diarization is now a separate detector that works with any audio transcription model. - New optimized and more stable implementation for MMPose and SAM 3D Body.
- Reworked documentation with new tutorials on configuring detectors and including your own dataset.
- Many other small fixes and changes in different detectors and visualizers.
Breaking changes:
- All detector environments now use uv instead of venv and conda. Please reinstall NICE Toolbox.
dataset_properties.tomluses subject rules instead ofsubjects_descr,sees_subjectsandhears_subjects. Please update it based on the provided example.machine_specific_paths.tomlrequires the newffmpeg_hwaccelfield. Addffmpeg_hwaccel = 'cuda', or'none'to decode videos on the CPU.- The
[io]section ofdetectors_run_file.tomlhas new keys for the new output folder structure. Please update it based on the provided example. - The 2D pose estimation algorithms need
video_subject_detectorand a person detector such asrtmdet_m_personin thealgorithmslist. whisperxno longer runs the speaker diarization itself. Addaudio_subject_detector(andpyannotefor theaudio_rangerule) to thealgorithmslist, or usewhisperx_no_diar.- The
sam_3d_body_dinov3andsam_3d_body_vithinstances replacesam_3d_body. They have new parameters.
v0.3.1
- [License Change] To support the open-source community and make NICE Toolbox more accessible, our team decided to change the license of the project from
CC BY-NC-SA 4.0toAGPL-3.0. You can learn more about the new license terms here. - UniGaze (
unigaze) - a new state-of-the-art gaze direction estimation model. - CrisperWhisper (
crisper_whisper) - audio transcription model for exact word capture (including verbatims like"[um], [uh], [laughter]"). - InsightFace (
insight_face) - face analysis model, providing face bounding boxes and facial landmarks. - Py-Feat (
py_feat) facial expression library is updated to the latest version, v2.1.1. The new release adds support for the face_multitask_v2 model, capable of estimating FACS action units, emotions, valence/arousal, gaze, and other modalities in a single model pass. - Eye closure detector (
eye_closure_ear) using the eye aspect ratio, which estimates an eyelid closure score from 2D facial landmarks. - Closed eyes and blinks detector (
eye_closure_threshold) with configurable timing thresholds. - Audio evaluation metrics: transcription word error rate against reference transcripts, with configurable text normalization and plots.
- Extension of the ELAN connector for audio transcription and the napari-deeplabcut connector for body joint labeling.
- New unified detector input/output system with typed array schemas and metadata, applied to both video and audio detectors.
- Gaze fusion can now use the detected eye state to interpolate over or discard gaze direction when eyes are closed.
- Rework of the installation - new
makecommands and a Docker build script that give more control over which model virtual environments are installed. dataset_properties.tomlwas redesigned to support customization at the individual sequence level, named wildcards and templates.- Rerun visualization of SAM 3D Body meshes.
- Many other small fixes and changes made across different detectors and visualizers.
Breaking changes:
- Most of the third-party detector venv and conda environments were updated, please reinstall them.
- The NICE Toolbox Docker image now distributes only the base NICE Toolbox dependencies and environment. Check the updated documentation on how to install specific detectors inside Docker.
dataset_properties.tomlwas redesigned, please update it based on the provided example.- The
cur_session_IDplaceholder is deprecated. Thecur_sequence_IDplaceholder was renamed tocur_sequence_id(lowercase). - The
videossection indetectors_run_file.tomlwas renamed tosequences.sequence_idnow supports wildcards (e.g.session_*_take_*). - Most of the detectors in
detectors_config.tomlwere updated with new asset links and the new input dependency system.
v0.3.0
- SAM 3D Body (
sam_3d_body) - 3D whole-body pose estimation. Supports single- and multi-view setups. - WhisperX (
whisperx) - audio transcription and speaker diarization. - MotionBERT (
motionbert) - 3D body pose lifting from precomputed 2D body joint detections. - New MMPose algorithms:
vitpose_huge,rtmpose_l_aic,rtmpose_l_wholebody, andrtmpose_m_mpii. - New ELAN connector: export outputs to ELAN annotation format for manual labeling workflows.
- New napari-deeplabcut connector: export body joint detections to DeepLabCut format.
- Updated evaluation pipeline: new ground-truth-based metrics, improved configuration schema, and flexible group-by and aggregation options.
- New asset download manager: model weights are now downloaded automatically during setup or first run.
- New project config: a central config file per project that holds paths to your dataset and detector configs, decoupling project settings from the NICE Toolbox installation folder.
- Algorithms Instances support, allows to create multiple configurations of the same algorithms with different parameters.
- Sequences time ranges
video_startandvideo_stopnow accept timestamps (e.g."00:01:30") in addition to frame numbers.
Breaking changes:
detectors_run_file.tomlhas changed.component_algorithm_mappingand per sequencecomponentslists are deprecated. Usealgorithmslist for all desired algorithms instances.evaluation_config.tomlwas redesigned, please update it based on the provided example.- Separate evaluation summaries are currently deprecated and now a part of metrics.
EvaluationWrapperfor exporting evaluation results to pandas is deprecated.frameworksindetectors_config.tomlare deprecated. There are more general usetemplatesnow. Please update your config.
v0.2.2
0.2.2
- Refactoring of data preprocessing and inference for all detectors.
- Major optimization and bug-fixing of py-feat inference.
- Refactoring, optimization, and bug-fixing of multiview-ethgaze.
- Refactoring of config placeholders resolution, making it faster and more stable.
- New config validation system. It will detect missing required fields or wrong field types across all configs.
- Fixes for subject tracking consistency in multiple detectors.
- In
detectors_run_file.tomlyou can setvideo_length = -1to process all frames inside a video.
Breaking changes:
- The frame index leading zeroes format was extended from
05dto09dto support longer videos. This results in new filenames. - CSV exported files are now saved inside individual video folders, not inside the root output folder. This can be customized in config.
- All runtime placeholders now start with
cur_<placeholder_name>. For example, the<session_ID>placeholder was renamed to<cur_session_ID>. - Cyclic placeholder dependencies are deprecated. For example,
git_hash = "<git_hash>"will now raise an error. - Placeholder shadowing is deprecated. Use unique placeholder names at each level of the config file.
- NICE Toolbox now uses submodule forks of mmpose and SPIGA. Library versions remain the same, so there should be no changes in results.
- Multiview-ETH-XGaze now supports multiview only inside NICE Toolbox. All logic for multi-camera fusion was moved to NICE.
eth_xgazenow exports raw3dand3d_filteredfor individual cameras andxgaze_gaze_fusedandxgaze_gaze_fused_filteredfused from all cameras.eth_xgazenow exportslandmarks_2dwith confidence scores.detectors_run_file.tomlconfig now requireslog_levelanderror_levelfields to be set.
Full Changelog: v0.2.1...v0.2.2
v0.2.1
Includes an evaluation module, Docker support, additional detector output, and many other improvements.
Full Changelog: v0.2.0...v0.2.1
v0.2.0
Includes code refactoring, easier installation, and new detectors for emotion individuals and head orientation.