Skip to content

FediAID artifact update: reproducibility improvements and five-seed MMR sensitivity

Pre-release
Pre-release

Choose a tag to compare

@4AlexMin 4AlexMin released this 05 Oct 13:50
· 1 commit to master since this release

This release addresses artifact-review feedback and extends the optional MMR sensitivity analysis.

  • Reproduction environment: pinned the validated core runtime and tokenizer dependencies and the DeBERTa encoder revision; added SentencePiece and the protobuf version constraint.
  • Stricter verification: both evaluation workflows now reject missing references, incomplete coverage, duplicate results, non-finite metrics, and invalid tolerances.
  • Clearer evaluation instructions: documented a complete raw-text reproduction route, aligned raw and embedded dataset names for reference comparison, and corrected command examples.
  • Per-seed references: added expected F1/AUC values for all 60 dataset/seed combinations in the main evaluation.
  • Five-seed sensitivity analysis: extended the optional sweep from seed 0 to seeds 0–4. Each platform is averaged over five seeds before computing the mean and population standard deviation across platforms.
  • Documentation and licensing: documented the released checkpoint’s training data, split construction, settings, and training command; added an MIT source-code license.

The released checkpoint, memory bank, main expected metrics, and external evaluation configuration are unchanged.