Is turning on diff-hl-mode in those files an actual problem? If so, can we identify a new class of buffers it should skip? Maybe there's a meaningful predicate we can add in turn-on-diff-hl-mode.