🏷️ Material Recognition fused audio+tactile+video · 76.0% test acc (chance 12.5%)	🧵 Tactile Super-Resolution clean 100Hz force curve from noisy low-rate input · −85% MSE, 0.996 corr
🔍 Cross-Modal Retrieval touch query → matching sound clip · 2× over chance Recall@1/@5	🔄 Cross-Modal Generation haptic recovery: tactile force curve generated from sound alone · 0.80 corr

VisTouch was constructed for Cross-Modal Semantic Communications (IEEE Wireless Communications, 2022) by controlling a robot arm (UR3 + RH56BF3 dexterous hand) to press and slide across everyday materials while a camera, a microphone, and a tactile force sensor record the same contact event simultaneously.

The full VisTouch research corpus spans 47 material categories and contains million-scale raw sensor observations across video frames, audio waveform samples, and tactile measurements. The companion paper reports 1000+ synchronized video–audio–haptic signal pairs over all 47 categories, supporting cross-modal semantic encoding, retrieval, and haptic signal recovery research.

This repository is the first curated public benchmark release: 2000 timestamp-aligned audio/tactile/video triplets from 8 representative material categories used in the paper's evaluation — brass · linen · paper · polyester · silk · spandex · stone · wood — with a predefined cross-pressure train/test split. Every released sample is a genuine sensor capture; no synthetic data is included. Future updates will progressively release more of the 47-category corpus, additional paths, views, and benchmark tasks.

_{Capture rig: dexterous hand + microphone + tactile sensor, fixed camera view.}

The data files are hosted externally — this repository ships with an empty dataset/ folder:

After downloading, place the contents inside the dataset/ folder at the repository root:

VisTouch/
└── dataset/
    ├── audio/      # per-material subfolders of .wav files
    ├── tactile/    # per-material subfolders of .csv files
    └── video/      # per-material subfolders of .avi files

All file paths in metadata/samples.csv (and every script) resolve relative to this layout, so no further configuration is needed.

VisTouch/
├── dataset/            # audio/ tactile/ video/  (empty in this repo — see Download above)
├── metadata/           # samples.csv · sessions.csv · segments.csv · classes.json
├── scripts/            # dataloader.py · classify_baseline.py · tasks/
└── docs/               # reports, alignment/quality docs, demo assets, logs/

from dataloader import VisTouchDataset, get_dataloader

ds = VisTouchDataset(classes=["silk", "stone"],          # any subset of the 8 materials
                     modalities=("audio", "tactile"),    # load only what you need
                     split="train")
loader = get_dataloader(ds, batch_size=8, shuffle=True)

cd scripts
python dataloader.py --classes silk stone --modalities audio tactile --split train
python classify_baseline.py --slice-mode cycle          # reproduce the classification baseline
python tasks/tactile_super_resolution.py                # reproduce any task baseline

Files follow VisTouch_{material}_f{force}_r{path}_v{view}_{segment}.{ext} (e.g. VisTouch_silk_f6_r1_v1_seg03.wav). Sample-level metadata, labels, and file paths live in metadata/samples.csv.

All baselines are intentionally lightweight (CPU-trainable in minutes) and serve as usability floors, not state-of-the-art. Trained weights ship in scripts/tasks/weights/; full training/test console records live in docs/logs/.

Every released signal passed an automated anomaly screen: rare, physically impossible ADC glitches in the tactile streams and isolated electrical pops in the audio were detected and repaired (824 tactile points and 18 audio samples across 2000 samples — everything else is byte-identical to the raw sensor recordings). Full findings: docs/logs/anomaly_check.log.

• Release the remaining material categories (47 total)
• Additional slide paths (r2, r3) and camera views (v2, v3)
• Deep baselines (spectrogram CNNs, video transformers, full waveform generation)

If you use VisTouch, please cite:

@article{li2022crossmodal,
  title   = {Cross-Modal Semantic Communications},
  author  = {Li, Ang and Wei, Xin and Wu, Dan and Zhou, Liang},
  journal = {IEEE Wireless Communications},
  volume  = {29},
  number  = {6},
  pages   = {144--151},
  year    = {2022},
  doi     = {10.1109/MWC.008.2200180}
}

Dataset files are released under CC-BY-4.0 (DATA_LICENSE); all code under MIT (LICENSE). "# VisTouch"