feat: per-item HM IoU and quality_percentile in evaluation#23
Merged
ziv-lazarov-nagish merged 1 commit intonagishfrom Apr 18, 2026
Merged
feat: per-item HM IoU and quality_percentile in evaluation#23ziv-lazarov-nagish merged 1 commit intonagishfrom
ziv-lazarov-nagish merged 1 commit intonagishfrom
Conversation
AmitMY
requested changes
Apr 16, 2026
Contributor
AmitMY
left a comment
AmitMY
left a comment
There was a problem hiding this comment.
i'm confused by quality_percentile
| help="drop predicted segments shorter than this many frames (0=off)") | ||
| parser.add_argument("--merge_gap", type=int, default=0, | ||
| help="merge predicted segments separated by ≤ this many frames (0=off)") | ||
| parser.add_argument("--quality_percentile", type=float, default=1.0, |
Contributor
Author
There was a problem hiding this comment.
used in build_datasets when passing eval_args (acts like the parser variable).
Contributor
There was a problem hiding this comment.
can you link it? i can't see it used - it is added in this PR, but not used in this PR?
Contributor
Author
Contributor
Author
There was a problem hiding this comment.
on second thought, i can read the quality_percentile that was used in training from the splits_manifest.json file (it's being written when manifest is created) in the checkpoint's directory and remove the argument, but that prevents us from using a different quality_percentile in evaluation. what do you think?
AmitMY
requested changes
Apr 16, 2026
| help="drop predicted segments shorter than this many frames (0=off)") | ||
| parser.add_argument("--merge_gap", type=int, default=0, | ||
| help="merge predicted segments separated by ≤ this many frames (0=off)") | ||
| parser.add_argument("--quality_percentile", type=float, default=1.0, |
Contributor
There was a problem hiding this comment.
can you link it? i can't see it used - it is added in this PR, but not used in this PR?
- compute HM IoU per batch item (matching training validation_step) instead of HM of averaged IoUs - add --quality_percentile arg for platform dataset filtering
ziv-lazarov-nagish
force-pushed
the
feat/per-video-eval
branch
from
fcec4f8 to
8e2a358
Compare
AmitMY
approved these changes
Apr 17, 2026
| help="drop predicted segments shorter than this many frames (0=off)") | ||
| parser.add_argument("--merge_gap", type=int, default=0, | ||
| help="merge predicted segments separated by ≤ this many frames (0=off)") | ||
| parser.add_argument("--quality_percentile", type=float, default=1.0, |
ziv-lazarov-nagish
added a commit
that referenced
this pull request
Apr 18, 2026
…_model evaluate_model now returns hm_IoU computed per item (nagish PR #23). Wrapping it with _add_hm_iou would overwrite that with the less accurate average-of-averages metric.
ziv-lazarov-nagish
added a commit
that referenced
this pull request
Apr 19, 2026
…_model evaluate_model now returns hm_IoU computed per item (nagish PR #23). Wrapping it with _add_hm_iou would overwrite that with the less accurate average-of-averages metric.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
validation_stepcalculation. Previously HM was computed as harmonic mean of the already-averaged sign/sentence IoUs.--quality_percentileCLI arg to filter platform dataset videos by quality score during evaluation.Files changed
sign_language_segmentation/evaluate.py— per-item HM IoU, quality_percentile argTest plan
ruff check .— all checks passedpytest— 61 passedpython -m sign_language_segmentation.evaluate --checkpoint ... --datasets dgs --split test --device cuda— HM IoU prints correctlypython -m sign_language_segmentation.evaluate --checkpoint ... --datasets platform --split test --device cuda --quality_percentile 0.8— quality filtering works