Skip to content

Release 0.7.5

Latest

Choose a tag to compare

@dylanuys dylanuys released this 09 Jul 00:51
be92ff8

36 new holdout datasets across image and video modalities, expanding coverage of synthetic generation methods and real-world content.

Released dataset name Obfuscated holdout name Modality Media type
sa-1b-shards real-image-holdout-975f791a image real
sa-1b-shards real-image-holdout-9d17e346 image real
sa-1b-shards real-image-holdout-c8922129 image real
sa-1b-shards real-image-holdout-82555af6 image real
sa-1b-shards real-image-holdout-35352d75 image real
communityforensics-real real-image-holdout-81694902 image real
glint360k real-image-holdout-30d9329b image real
communityforensics-fake synthetic-image-holdout-25aca886 image synthetic
communityforensics-fake synthetic-image-holdout-a5c5ef0f image synthetic
imagepulsev2-addremove synthetic-image-holdout-dec37878 image synthetic
imagepulsev2-change synthetic-image-holdout-888684fc image synthetic
imagepulsev2-angle synthetic-image-holdout-a97c0d16 image synthetic
imagepulsev2-crop synthetic-image-holdout-284cb6bc image synthetic
imagepulsev2-inpaint synthetic-image-holdout-c9fbb212 image synthetic
imagepulsev2-humanface synthetic-image-holdout-016cd50e image synthetic
imagepulsev2-style synthetic-image-holdout-3139ff06 image synthetic
justweirdimages synthetic-image-holdout-027002fc image synthetic
flux-sd15-ipadapter synthetic-image-holdout-a7ea3f58 image synthetic
ip-adapter-faceid synthetic-image-holdout-3fe582cc image synthetic
ip-adapter-faceid synthetic-image-holdout-d7ed0468 image synthetic
sd35-ip-adapter-faceid synthetic-image-holdout-45d8c442 image synthetic
sd35-ip-adapter-faceid synthetic-image-holdout-7f11881b image synthetic
llava-video-178k real-video-holdout-43262857 video real
omnivideo-100k real-video-holdout-ce760bb4 video real
omnivideo-test real-video-holdout-47243aa8 video real
ophora-160k real-video-holdout-a799bc65 video real
qa-v1 real-video-holdout-3c0c27c8 video real
qa-test-h264 real-video-holdout-de456938 video real
deeptumorvqa real-video-holdout-2f424a18 video real
tiktok-10m real-video-holdout-c00a826f video real
heico-focus-vqa real-video-holdout-f847a136 video real
tonystark03-deepfake-real real-video-holdout-0cce6c4d video real
gen-videos-fastwan2 synthetic-video-holdout-c3a396e1 video synthetic
gen-videos-hunyuanvideo synthetic-video-holdout-d793de35 video synthetic
gen-videos-kling synthetic-video-holdout-46ee35f7 video synthetic
gen-videos-veo3 synthetic-video-holdout-dca04654 video synthetic
gen-videos-wan2 synthetic-video-holdout-eb2d2439 video synthetic
gen-videos-sora2 synthetic-video-holdout-9a13e6c6 video synthetic
cameraclone-dataset synthetic-video-holdout-c097c59a video synthetic
scene-decoupled-video-dataset synthetic-video-holdout-39db0365 video synthetic
synthetic-vstat-block-counting synthetic-video-holdout-f22b057c video synthetic
synthetic-vstat-shuffle-puzzle synthetic-video-holdout-c5935b8b video synthetic
tonystark03-deepfake-fake synthetic-video-holdout-b0fe3d10 video synthetic

Adds augmentation robustness tracking to benchmark runs.

  • Blends aug_sn34_score into sn34_score using aug_weight in finalize_run
  • Carries aug MCC, Brier, and CE forward for persistence alongside the final aug score
  • Fixes .npy double-extension bug when writing to tmp files
  • Fixes aug cache to log write errors and unpack video tuples in caller
  • Fixes build_sample_id to include member_path for correct sample identification
  • Includes aug fields (aug_sn34_score, aug_binary_mcc, aug_binary_brier, aug_binary_ce) in save_results_to_json output