Ah, OK. So this can happen when the indirect buffer is created with the non-nil clone argument. I've pushed the proposed workaround.

Do you have a particular situation in mind, though, where clone-indirect-buffer is used? Maybe diff-hl-mode should instead disable itself in clone-indirect-buffer-hook.