federated-learning-attack-lab

Federated learning is sold as "private by design." This lab shows that the defense everyone reaches for to harden it — robust aggregation — provides exactly zero privacy, and that the one defense that does provide privacy is really two mechanisms people mistune as one.

Federated learning keeps raw data on the device and shares only gradients. Two things follow that this lab makes concrete, on a real federated logistic-regression trainer:

1. A gradient leaks its data. For the model people actually start with, a single-example update is a scalar multiple of the input — so whoever sees that update recovers the input direction exactly, no optimization required.
2. The two headline threats need two different defenses, and each is blind to the other.

$ flab matrix
defense             clean  poisoned  leakage
--------------------------------------------
none (mean)          0.80      0.48     1.00
robust (median)      0.80      0.80     1.00
robust (krum)        0.75      0.77     1.00
DP only              0.75      0.72     0.25
robust + DP          0.71      0.67     0.25

• none (mean): a single malicious client crushes the model (0.80 → 0.48) — and the honest clients' data leaks in full (1.00).
• robust (median / krum): poisoning is fully defended — poisoned returns to clean. But leakage is still a perfect 1.00. Robust aggregation buys zero privacy.
• DP only: leakage drops to 0.25 — privacy defended — at a utility cost (clean 0.80 → 0.75).
• robust + DP: the only row that defends both, and it pays the most utility (0.71).

Why robust aggregation leaks completely

Median, trimmed-mean, and Krum all work by comparing individual client updates and rejecting outliers. To do that, the server must see every client's raw update. That is fundamentally incompatible with hiding those updates — so a server running robust aggregation to stop poisoning is looking at exactly the per-client gradients that gradient inversion turns back into training data. Robustness and privacy aren't just separate; robust aggregation is architecturally at odds with privacy. Teams add it believing they've "hardened" federated learning, and they've bolted one door shut while leaving the other wide open.

The subtle finding: DP is two mechanisms, not one

"Just add differential privacy" hides a trap. DP here is clip, then add noise — and those are two different defenses against two different threats:

$ flab decompose
DP configuration              poisoned  leakage
-----------------------------------------------
clip-only  (C=0.5, sigma=0)       0.79     1.00
noise-only (C=50,  sigma=4)       0.61     0.00
clip+noise (C=0.5, sigma=4)       0.72     0.25

• The clip defends poisoning. clip-only keeps accuracy high (0.79) by bounding any single client's influence — but gives no privacy (1.00).
• The noise defends privacy. noise-only drives leakage to 0.00 — but with the clip loosened, poisoning comes back (0.61, worse than clip-only).

So if you treat DP as one knob and loosen the clip to recover utility, you silently reopen the poisoning door while your privacy metric still looks great. The clip and the noise must be tuned as the two independent controls they are.

Gradient inversion vs. batch size

$ flab leak
local batch size    mean inversion leakage
------------------------------------------
1                                     1.00
2                                     0.67
4                                     0.52
8                                     0.44
16                                    0.40

A single-example update leaks its input perfectly. Larger local batches dilute it (a batch update is a weighted sum of per-example gradients) — but small local batches are common in cross-device FL, and robust aggregation forces per-client visibility no matter the batch size.

What's modeled, honestly

The trainer is logistic regression on a synthetic, seeded, linearly-separable-ish task — deliberately the simplest setting, because it is exactly where "gradients are safe to share" is stated most confidently and is most wrong. The mechanisms shown (gradient→input recovery, robust-aggregation's visibility requirement, clip-vs-noise separation) are general; the numbers are specific to this model and task. Deep non-linear nets need a learned inversion attack and the numbers shift — see the threat model for where the claims stop.

Quickstart

git clone https://github.com/vinzabe/federated-learning-attack-lab && cd federated-learning-attack-lab
python -m pip install -e ".[dev]"

flab matrix         # the threat-by-defense matrix — read poisoned vs leakage
flab matrix --json
flab decompose      # DP's clip defends poison; its noise defends privacy
flab leak           # gradient inversion vs local batch size

Development

python -m pip install -e ".[dev]"
pytest --cov=flab          # 24 tests, ~99% coverage; each finding pinned as an invariant
mypy --strict src/flab     # clean (Python 3.12; numpy stubs require it)
ruff check src tests

License

MIT © vinzabe