English | 中文 | Changelog | Contributing

The ZDEM Particle Tracker Application

A desktop research application that integrates the tools needed to inspect ZDEM discrete-element frame dumps and track one permanent particle ID across time steps. It includes streaming DAT parsers, batch 2D rendering (true-radius mesh discs), region policy (walls first), trajectory extraction with erosion vs file-error handling, displacement / path-length curves, and export hooks — without turning the UI into a generic multi-physics studio.

ZDEM particle tracking (what this solves)

ZDEM writes large text frames (all_<step>.dat) with tens of thousands of particles per step. Lab post-processing often needs a simple question answered quickly:

Where did particle ID = N go from step A to step B, and when (if ever) was it eroded?

Commercial / heavy tools (e.g. OVITO-class workflows) can visualise whole systems, but the everyday salt-tectonics / granular DEM loop is different: permanent ID ≠ file row index, walls define the box better than particle Y-bbox, GUI must not freeze on multi-GB campaigns, and displacement zero is the user-chosen start step.

This app is opinionated for that loop on Windows (primary) with headless CI on Linux.

MainViewer after loading a real lab sample path used by tests (~tens of thousands of particles, layered groups). Particles are not hand-drawn in the capture script.

Getting Started

These instructions will help you open a ZDEM experiment directory and track a single particle.

Prerequisites

• Python 3.11+
• Windows 10/11 for interactive VisPy / desktop use (primary target)
• Linux is supported for headless tests (QT_QPA_PLATFORM=offscreen)
• Optional: uv for fast, locked installs

Installation

Clone and install with uv (recommended):

git clone https://github.com/Phoenix0531-sudo/ZDEM_ParticleTracker.git
cd ZDEM_ParticleTracker
uv sync --extra dev

Or with pip / venv:

python -m venv .venv
# Windows:
.venv\Scripts\activate
pip install -r requirements.txt

Core stack: PySide6, NumPy, SciPy, PyQtGraph, Matplotlib (+ scienceplots), VisPy (default mesh discs). CI forces PyQtGraph via ZDEM_FORCE_PYQTGRAPH=1.

Quickstart

1. Run the app:

python main.py
# or: uv run python main.py

2. Choose an experiment directory that contains all_<digits>.dat (non-recursive scan).
3. Confirm start / end steps and experiment region (walls preferred; metadata fallback).
4. Select a particle by click (KD-tree) or by typing the permanent particle ID.
5. Trajectory extraction runs in a worker with progress + cancel; curves and tables update as frames advance.

# Headless / CI-style test run (no interactive window required)
set QT_QPA_PLATFORM=offscreen
set ZDEM_FORCE_PYQTGRAPH=1
uv run pytest -q tests

Screenshots and evidence

GUI hero captures prefer the local large ZDEM experiment path when available; the script falls back to the committed samples/ mini sequence (12 particles, 3 steps) so the repository remains clone-runnable without private lab dumps.

MainViewer — real ZDEM experiment capture (large data first; samples fallback)	Trajectory evidence — path length vs net |u| (return path → u=0, L=20)
Displacement vs step — ux / uy / |u| (zero = start frame)	Erosion policy — present / eroded / file_error (never confused)
Service overview — selection gate + region + kinematics	Evidence pack — 2×2 collage of the panels above

# GUI grabs from committed samples/ (or local SAMPLE_CANDIDATES for lab dumps)
uv run python scripts/capture_real_shots.py
# Differentiated service evidence figures (matplotlib, no interactive GUI)
uv run python scripts/generate_evidence_pack.py

Application architecture

A typical interactive session follows:

1. Scan the directory for ^all_(\d+)\.dat$ only (ignore .sav, vtk_inters_*.vtk, subfolders).
2. Parse metadata + particle basics (+ groups) in a streaming fashion; stop before huge contact blocks when possible.
3. Render particles as batched geometry (VisPy mesh discs by default; PyQtGraph / CPU path for CI).
4. Select one permanent ID present at the session start frame.
5. Track across the chosen step range: displacement relative to start, increments, path length; mark eroded vs file_error.
6. Export only when the user asks (CSV / PNG) — nothing is auto-written into the experiment folder.

Package layout (real modules):

main.py
zdem_particle_tracker/
  app.py
  parsers/          # DAT stream parse + scan
  rendering/        # VisPy / pyqtgraph / backend selection
  services/         # region, trajectory, export, quality, project config
  widgets/          # MainViewer, selection helpers
  workers/          # frame load / trajectory workers
  ui/               # side panels, about, legend
  utils/            # cache, colors, logging helpers
tests/              # parser, gate, region, trajectory, GUI smoke (subprocess)

Base ideas (algorithm-agnostic):

Concern	Approach
Identity	Permanent id only — never row index for tracking
Region	User lock > walls AABB > metadata (RegionDetector)
Velocity	v = Δx / Δstep (simulation step), not wall-clock seconds
Scale	NumPy arrays + one/few draw calls — no per-particle Qt items
GUI freeze	Workers + signals; LRU frame cache; cancelable trajectory

Performance

• Streaming / section-based DAT parse (do not load multi-GB contact tails for basic play).
• Viewport culling and optional decimation when draw count is huge; pick still uses full spatial index.
• Mesh buffer reuse while scrubbing frames.
• Linux CI isolates some Qt constructs in a subprocess (tests/qt_subprocess.py) to avoid process-wide OpenGL aborts.

Help and Support

1. Read this README and the in-app status messages (scan / load / trajectory / cancel).
2. Run uv run pytest -q tests with QT_QPA_PLATFORM=offscreen and ZDEM_FORCE_PYQTGRAPH=1.
3. Open a GitHub issue with: OS, Python version, a minimal DAT snippet or metadata, and whether the failure is parse / select / track / render.

Related lab tools in the same ecosystem:

Repo	Role
ZDEM_Salt_Kinematics	Salt geometry metrics
ZDEM_Area_Conservation	Area conservation / triangulation
ZDEM_Bond_Fracture	Bond damage series
ZDEM_Model_Editor	Model file visual editor

Contributing

Useful contributions:

• Parser robustness on real DAT variants (headers, scientific notation, missing property rows).
• Tests that stay offline (synthetic DAT) unless gated by env flags for lab samples.
• Docs / bilingual README fixes.
• Careful GUI paths that remain cancelable and thread-safe (no GUI from workers).

Please open an issue before large refactors of MainViewer. Keep permanent-ID tracking and region policy intact.

Scope / limitations (v1)

In

• ZDEM 2D all_*.dat sequences, walls, group / color modes
• Single permanent-ID tracking, erosion vs file-error distinction
• Displacement, increments, path length; curves and table
• Manual project config (.zdemtrack.json) and manual export

Out (by design for v1)

• Simultaneous multi-particle tracking
• Binary .sav reverse engineering
• vtk_inters_*.vtk contact network as primary source
• 3D solver UI, cloud sync, auto-export into experiment folders
• Treating PEPT / fluorescence multi-tracer linking as the core product (see pept for that domain)

Continuous Integration

GitHub Actions workflow .github/workflows/ci.yml:

1. Logic job — ruff critical rules + pytest excluding heavy GUI/render modules first.
2. Full offscreen job — installs XCB/EGL libs, QT_QPA_PLATFORM=offscreen, ZDEM_FORCE_PYQTGRAPH=1, full tests/.
3. pip-audit — advisory only (continue-on-error).

Badge at the top tracks the CI workflow on main.

License

MIT. See LICENSE.

Acknowledgements

README section layout is intentionally aligned with research-oriented particle-tracking projects such as pept (University of Birmingham Positron Imaging Centre): clear problem statement, install path, architecture, performance notes, support, and honest scope. Domain algorithms and ZDEM formats here are independent of PEPT.