Skip to content

v2.0 — Phase C v2 production deploy + Phase E external validation

Choose a tag to compare

View all tags
@Cureeeeeeee Cureeeeeeee released this 24 May 19:33
v2.0
1d864ab

v2.0 Release

Educational prototype for AI-assisted multiclass skin lesion classification on the HAM10000 dataset. Non-commercial use only (CC BY-NC 4.0 datasets). Not a medical device.

Headline Results

Metric HAM10000 test (in-distribution) ISIC 2019 (external, HAM-disjoint)
v2 ResNet50 melanoma recall 73.40% 37.09%
v2 ResNet50 macro F1 70.08% 34.72%
v2 ResNet50 test ECE (calibrated) 0.0248 0.1814
v1 4-model ensemble macro F1 74.10% 41.24%

Key honest finding: the in-distribution +18.6 pp melanoma recall improvement from Phase C does not generalize to external (ISIC 2019) data — mel recall drops to ~37%. A three-line preprocessing audit confirmed this is genuine distribution shift, not a pipeline artefact. The strong in-distribution number does not generalize; this is an educational prototype, not a medical device.

What's In This Release

  • 6 model checkpoints: 4 v1 baseline models (ResNet50, DenseNet121, EfficientNet-B0, MobileNetV3-small), v2 ResNet50 winner (focal loss + balanced sampler), and v1 ResNet50 backup for rollback.
  • FastAPI backend with calibrated single-model /predict, 4-model ensemble /predict-ensemble, and Grad-CAM /predict-cam.
  • Flutter mobile UI prototype (clinical-style 5-screen layout).
  • Reproducibility: requirements-lock.txt, scripts/run_demo.sh / run_demo.ps1, scripts/download_checkpoints.py with SHA256 verification.

Installation

CPU-only inference, ~250 MB checkpoint download, API ready at http://127.0.0.1:8126.

bash scripts/run_demo.sh                                          # macOS / Linux
powershell -ExecutionPolicy Bypass -File scripts\run_demo.ps1     # Windows

See README Quick Start for full prerequisites and troubleshooting.

Phase Highlights

  • Phase C — focal loss + balanced sampler lifts melanoma recall from 54.79% to 73.40% on HAM10000 (single-model ResNet50).
  • Phase E — honest external validation on a HAM10000-disjoint subset of ISIC 2019 (4,353 images). Documents distribution shift, calibration OOD failure, and bias audit limitations.
  • Phase F — dependency lock, CPU-only deployment scripts, release packaging, Docker design (build deferred to future work).

Full Documentation

  • docs/release_v2.0.md — comprehensive release notes
  • docs/phase_e_external_validation.md — external validation findings + preprocessing audit
  • docs/licenses.md — full dataset citations and license terms

Citation

MIT for code; CC BY-NC 4.0 propagates to weights via HAM10000. See LICENSE.

HAM10000 citation:

Tschandl P., Rosendahl C. & Kittler H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data 5, 180161 (2018). doi:10.1038/sdata.2018.161

Assets 14
Loading