Markers

Markers

A marker is an id and a title, followed by links and a status on the lines after it. Ids are Q (question), H (hypothesis), E (experiment), R (result) or C (claim), plus a number (H2, E10, R3b).

In notes (Markdown)

## H2: Learning-rate warm-up removes the gap
answers: Q1
status: open

A definition may be a heading or a list item. The links and attributes on the lines after it, up to the next heading, belong to it:

Line	Meaning
answers: Q1	a hypothesis (or anything) that answers a question
tests: H2	an experiment that tests a hypothesis
of: E1	a result produced by an experiment
supports: H1 / weakens: C1	what a result means for a hypothesis or claim
relies on: H1, R1	what a claim stands on
status: supported	open, supported, refuted, running … (supported gets a blue band, refuted a dashed box)
metrics: acc=0.91, loss=0.3	a result's numbers
sources: smith2020	sources by key
expects: E1.results acc > 0.85	on a claim: the number it stands on (>, >=, <, <=, ==); a warning when a result stops meeting it

In experiment configs

YAML, JSON or TOML, with a research: block:

research:
  id: E1
  title: Batch 32 vs 512 on small-8k
  tests: [H1]
  results: ../out/e1.json     # JSON (nested keys joined by ".") or CSV (its last row)

The results file becomes the result E1.results, with its numbers as metrics. A missing results file is a warning.

In scripts and notebooks

A comment: # research-diff: E2 tests H2, or # research-diff: E2: Warm-up sweep, then # research-diff: status: running on a later line.

In the paper (LaTeX)

% C1: Small batches give better accuracy on small datasets
% relies on: H1, R1
On the 8k split, batch 32 reaches 87.4\% … \cite{keskar2017large}.

The citations of the claim's paragraph become its sources.

In experiment trackers

Tag a run with the experiment's id and it becomes one of its results, with the run's last value of each metric:

Tracker	Tag
MLflow (local mlruns/ store)	the tag or param research-diff.experiment (or research_id): mlflow.set_tag("research-diff.experiment", "E1")
W&B (local wandb/ folders)	research_id in the run's config: wandb.init(config={"research_id": "E1"})

research-diff sync writes them to research/runs.json (commit it). The result's id is the experiment, the tracker and the run, e.g. E1.mlflow.abcdef12.

Warnings

• a claim that relies on nothing, or on a refuted hypothesis;
• a claim whose expects: a result no longer meets, or names a number that does not exist;
• an open hypothesis that no experiment tests;
• an experiment that tests no hypothesis;
• a link to an id defined nowhere;
• a results file that is not in the repository.

Where it looks

By default it looks in every .md, .yaml/.yml/.json/.toml, .py/.ipynb and .tex file, except a root README.md and CHANGELOG.md. A research-diff.toml narrows that:

notes = ["research/**/*.md"]
configs = ["configs/**/*.yaml"]
scripts = ["src/**/*.py"]
paper = ["paper/**/*.tex"]