CPU vs GPU Pulsed Update Produces Very Different Results

[Zhaoxian-Wu]

Observation

When running tile.update(x, d) on CPU and GPU with identical initial weights and inputs, the resulting weights differ substantially — not just at floating-point precision, but at a level comparable to or exceeding the weight change itself.

Measurement

64×64 tile, lr=0.01, all device-to-device variation disabled, averaged over 20 runs.

Metric: mean|w_cpu - w_gpu| / mean|Δw|

Device	dw_min	Batch=1	Batch=64	Batch=512
ConstantStep	0.01	147%	117%	116%
ConstantStep	0.001	63%	57%	57%
SoftBounds	0.01	150%	106%	72%
SoftBounds	0.001	63%	46%	25%

The cause is straightforward: CPU and GPU use independent random number generators, producing different pulse sequences from the same input.

Why This Matters

The simulator aims to predict how analog hardware behaves. But this means a researcher could get noticeably different training curves — and potentially different conclusions about a device configuration — depending on whether they run on CPU or GPU. Neither result is "wrong", but they are different realizations of the same stochastic process.

This raises a question about faithfulness vs. performance: the stochastic pulse model faithfully captures the noisy nature of real analog devices, and independent RNG is the natural consequence of parallel GPU execution. On the other hand, if two simulation runs of the same experiment give very different trajectories, it becomes harder to attribute differences to device parameters vs. simulation noise.

I'm just curious to hear how others think about the tradeoff between faithful stochastic simulation and reproducible results.

From the perspective of someone designing or evaluating analog AI hardware:

• Is this level of run-to-run variation (25–150% of the weight change per step) consistent with what we observe on real devices?
• When using simulation results to make device design decisions, do you typically average over many runs, or do you rely on single-run trajectories?

Reproduction

import torch
from aihwkit.simulator.configs.configs import SingleRPUConfig
from aihwkit.simulator.configs.devices import SoftBoundsDevice
from aihwkit.simulator.tiles.analog import AnalogTile

dev = SoftBoundsDevice(dw_min=0.001, dw_min_std=0.0, dw_min_dtod=0.0,
                       up_down=0.0, up_down_dtod=0.0,
                       w_max_dtod=0.0, w_min_dtod=0.0)
cfg = SingleRPUConfig(device=dev)

tile_cpu = AnalogTile(64, 64, cfg, bias=False)
tile_gpu = AnalogTile(64, 64, cfg, bias=False)
tile_cpu.tile.set_learning_rate(0.01)
tile_gpu.tile.set_learning_rate(0.01)
tile_gpu = tile_gpu.cuda()

w_init = torch.randn(64, 64) * 0.1
x = torch.randn(64, 64) * 0.5
d = torch.randn(64, 64) * 0.5

tile_cpu.tile.set_weights(w_init.clone())
tile_gpu.tile.set_weights(w_init.clone())
tile_cpu.update(x, d)
tile_gpu.update(x.cuda(), d.cuda())

w_cpu = tile_cpu.tile.get_weights()
w_gpu = tile_gpu.tile.get_weights()
delta_w = (w_cpu - w_init).abs().mean()
diff = (w_cpu - w_gpu).abs().mean()
print(f"mean|Δw| = {delta_w:.4e}, mean|cpu-gpu| = {diff:.4e}, rel = {diff/delta_w:.2%}")

[PabloCarmona]

Your point sounds quite interesting @Zhaoxian-Wu. Let me discuss it with the team and see what other users think. Thanks! 😄