A research testbed for general-purpose, local-first multimodal medical AI.
HAIMed is a research and engineering project exploring how compact, general-purpose medical AI models can operate locally and on-device.
The project investigates whether one model can understand different forms of medical information—including clinical text, health records, laboratory documents, radiographs, and CT images—while working within limited compute, memory, storage, and connectivity.
HAIMed follows a local-first direction: sensitive medical information should be processed as close as possible to where it is generated. Cloud infrastructure may still support training and large-scale experiments, but using the resulting model should not always require sending medical data to a remote service.
This direction is particularly relevant to privacy-sensitive workflows, offline environments, edge deployments, and healthcare settings where reliable cloud connectivity cannot be assumed.
HAIMed is an experimental research project. It is not a certified medical device and must not be used independently for diagnosis, treatment, patient monitoring, or clinical decision-making.
Most capable AI models depend on large cloud infrastructure. In medicine, that dependency creates additional challenges around privacy, latency, cost, connectivity, and control of sensitive information.
HAIMed explores a different question:
How much useful, general-purpose medical capability can be developed and preserved in a model small enough to operate locally?
The project studies models that can:
- understand medical questions and clinical text;
- interpret structured and longitudinal health records;
- extract information from laboratory and clinical documents;
- interpret selected radiograph and CT tasks;
- reason across medical text and images;
- combine information from multiple medical modalities;
- operate without continuous internet access;
- run on workstations, edge systems, and eventually supported devices;
- communicate uncertainty and abstain from unsupported tasks.
The goal is not simply to make a medical chatbot. HAIMed studies the broader model, data, evaluation, optimization, and deployment system required for credible local medical AI.
HAIMed is organized around five primary questions:
- Can a compact model develop useful capabilities across multiple medical modalities?
- Which training methods preserve the most medical capability under limited compute?
- How much performance is lost when a model is compressed for local deployment?
- Which medical tasks can be performed locally without sending sensitive data to a remote inference service?
- How should local medical models be evaluated for accuracy, uncertainty, safety, privacy, latency, and resource use?
Local execution may provide several advantages:
- greater control over sensitive medical information;
- reduced dependence on external inference providers;
- continued operation during connectivity failures;
- lower inference latency for some workflows;
- predictable deployment costs;
- support for edge and resource-constrained environments;
- easier integration with local health-information systems.
However, running a model locally does not automatically make it private, safe, or clinically reliable. The complete system—including storage, logging, temporary files, application security, model behaviour, and device access—must still be evaluated.
HAIMed treats these concerns as part of the research rather than assuming local execution solves them automatically.
HAIMed currently studies five areas of medical information.
| Area | Information studied | Example capabilities |
|---|---|---|
| Medical knowledge | Medical questions and clinical text | Question answering and reasoning |
| Health records | Structured and synthetic FHIR data | Timeline understanding and summarization |
| Clinical documents | Laboratory reports and document images | Extraction and document understanding |
| Radiographs | Chest X-ray images | Classification and visual reasoning |
| Computed tomography | Ordered chest CT slices | Multislice abnormality classification |
These areas provide a foundation for studying whether a compact model can learn capabilities that transfer across medical tasks and modalities.
The project does not currently attempt to cover every medical specialty, imaging modality, disease, or clinical workflow.
Medical information rarely exists in one format.
A single episode of care may include:
- a patient’s history;
- structured observations;
- laboratory results;
- clinical notes;
- medication records;
- radiographs;
- CT studies;
- previous encounters.
HAIMed investigates how a model can connect these different forms of information instead of treating each one as a completely separate problem.
The long-term objective is a compact medical model capable of receiving the information available for a task, determining what it can reasonably do, and declining tasks that exceed its validated capabilities.
HAIMed uses “local-first” as a broad deployment principle.
Potential deployment environments include:
- personal workstations;
- clinic workstations;
- local hospital servers;
- edge servers;
- offline computing environments;
- supported laptops;
- supported mobile or specialized devices.
Not every model version will run on every device. Local deployment experiments will compare different model sizes, hardware targets, quantization levels, memory requirements, and performance trade-offs.
A platform will only be described as supported after it has been tested on representative hardware.
The project is currently establishing its data and evaluation foundation.
Current work includes:
- registering and reviewing medical-data sources;
- verifying licences and permitted uses;
- creating reproducible preprocessing pipelines;
- separating patients correctly across dataset splits;
- defining a shared multimodal training format;
- validating generated images and documents;
- building small pipeline-validation datasets;
- preparing baseline training and evaluation experiments.
The current CT experiment uses a deliberately constrained subset of CT-RATE to validate the complete processing pipeline before larger experiments are attempted.
This subset is intended to answer questions such as:
- Can the correct studies be identified reproducibly?
- Are patients isolated between training and evaluation splits?
- Can CT volumes be converted into valid ordered image sequences?
- Are labels mapped correctly?
- Can multimodal training examples be generated and validated?
- Can the same frozen dataset version be reproduced later?
The constrained subset is a pipeline-validation experiment, not the intended scale or capability limit of HAIMed.
The first chest CT experiments cover three findings:
- pleural effusion;
- lung nodule;
- cardiomegaly.
Each selected CT study is represented using a sequence of ordered axial images. Multiple CT window presentations are combined so the model can observe lung, soft-tissue, and broader anatomical information.
The initial objective is to validate the representation and training pipeline. It is not to claim clinically reliable performance from a small experimental subset.
HAIMed follows an evidence-before-scale approach:
- Identify an appropriate medical-data source.
- Verify its licence, provenance, and permitted use.
- Build a reproducible processing pipeline.
- Separate patients before creating dataset splits.
- Validate the resulting text, records, documents, and images.
- Establish an unmodified model baseline.
- Adapt the model using controlled experiments.
- Evaluate medical-task and safety performance.
- Optimize the model for local inference.
- Measure what capability is preserved or lost.
- Document limitations and failure cases.
- Scale only after the smaller experiment is trustworthy.
This process allows failed assumptions to be discovered before expensive training begins.
HAIMed will compare several approaches to model development.
Candidate models are evaluated before medical adaptation. This establishes what each model can already do and prevents improvements from being attributed incorrectly.
Models are trained on validated medical-text and multimodal examples to develop task-specific and cross-modal capabilities.
Methods such as LoRA and quantized LoRA will be evaluated as lower-cost alternatives to updating every model parameter.
Larger models may be used as teachers for smaller models where appropriate. Teacher-generated supervision must be identified clearly and evaluated rather than assumed to be correct.
Promising models may be optimized through:
- quantization;
- pruning;
- knowledge distillation;
- reduced image-token use;
- shorter context representations;
- optimized local inference runtimes.
Every optimized model must be compared with its original baseline. A smaller model is useful only when the capability and safety trade-offs are measured transparently.
HAIMed evaluates more than model accuracy.
Depending on the task, evaluation may examine:
- correctness;
- sensitivity and specificity;
- precision and recall;
- performance on imbalanced conditions;
- document-extraction quality;
- reasoning consistency;
- performance across modalities.
The project will examine whether a model:
- recognizes unsupported inputs;
- expresses uncertainty appropriately;
- abstains when required;
- produces unsupported medical claims;
- remains reliable when information is incomplete;
- becomes overconfident after optimization.
Dataset evaluation includes:
- patient-level leakage checks;
- duplicate detection;
- source and label verification;
- image and document validation;
- reproducibility checks;
- manual review of selected examples.
Deployment experiments may compare:
- model size;
- memory consumption;
- loading time;
- inference latency;
- generation speed;
- supported context size;
- hardware compatibility;
- offline operation;
- capability before and after compression.
HAIMed is intended for research, experimentation, and education around local medical AI.
It may support work involving:
- medical-language-model research;
- medical vision-language research;
- multimodal dataset engineering;
- model compression;
- local inference;
- uncertainty and abstention;
- privacy-aware system design;
- medical AI evaluation.
HAIMed is not intended for:
- independent clinical diagnosis;
- treatment recommendations;
- emergency decisions;
- patient monitoring;
- replacing qualified healthcare professionals;
- use as a medical device without appropriate validation and approval;
- presenting experimental outputs as verified medical facts.
Some datasets studied by HAIMed contain real, deidentified patient data and have access or redistribution restrictions.
This repository does not distribute:
- restricted medical images;
- gated source datasets;
- private patient-level records;
- access tokens;
- derived artifacts that cannot legally be redistributed.
Anyone reproducing an experiment must obtain data directly from its original provider and comply with its licence, access agreement, privacy requirements, and permitted-use restrictions.
The software licence for this repository does not override the terms of any dataset or pretrained model.
HAIMed is an early research project.
Current limitations include:
- small pipeline-validation subsets;
- limited medical-condition coverage;
- weak or automatically generated labels in some datasets;
- incomplete modality coverage;
- limited expert annotation;
- possible source and population bias;
- no prospective clinical evaluation;
- no regulatory review;
- no guarantee that model outputs are medically correct;
- no completed evidence that the current model can run effectively on every proposed device.
These limitations will be updated as experiments are completed.
| Area | Status |
|---|---|
| Research definition | Active |
| Dataset registry and governance | In progress |
| Synthetic health-record pipeline | In progress |
| Laboratory-document pipeline | In progress |
| Chest CT pipeline | In progress |
| Chest-radiograph pipeline | Planned |
| Shared multimodal format | In progress |
| Baseline model evaluation | Planned |
| Medical model adaptation | Planned |
| Compression and quantization | Planned |
| Local inference evaluation | Planned |
| On-device benchmarking | Planned |
| Safety and uncertainty evaluation | Planned |
| Model card | Planned |
| Technical report | Planned |
Establish reproducible medical-text, health-record, document, radiograph, and CT datasets with clear provenance, governance, and evaluation boundaries.
Evaluate candidate compact models before adaptation and establish comparable medical-task baselines.
Train selected models across medical text, structured records, documents, and images while measuring transfer and interference between tasks.
Compress and optimize promising models for local inference while measuring capability loss, latency, and resource consumption.
Evaluate uncertainty, abstention, robustness, bias, failure behaviour, and the effects of model compression.
Publish permitted code, configurations, evaluation results, deployment benchmarks, dataset documentation, model cards, and a technical report.
Detailed technical instructions are maintained separately from this overview.
The project documentation will cover:
- development setup;
- system architecture;
- dataset preparation;
- medical-image processing;
- synthetic health-record generation;
- training and fine-tuning;
- evaluation;
- local inference;
- dataset and model governance.
Contributions are welcome where they preserve the project’s reproducibility, governance, and safety principles.
Contributors should:
- never commit restricted medical data;
- document the provenance and licence of new sources;
- preserve patient-level dataset separation;
- add validation for new data formats;
- report limitations and failed experiments honestly;
- avoid unsupported clinical claims;
- update the relevant documentation.
A complete contribution guide will be added as the project matures.
A project-specific citation will be provided if a formal technical report is released.
Until then, research using this repository should cite the original datasets, pretrained models, libraries, and papers used in each experiment.
The repository’s licence applies only to original HAIMed code and documentation.
It does not replace or override:
- dataset licences;
- model licences;
- gated-access agreements;
- third-party software licences;
- medical-data restrictions.
Review the terms of every external source before using its data, models, or derived artifacts.
HAIMed builds on open work in medical language models, medical vision-language models, health-data interoperability, medical-image processing, model compression, edge inference, and responsible medical AI.
The project acknowledges the researchers, institutions, maintainers, and dataset providers whose work makes reproducible medical-AI research possible.