I am not sure where diff-hl gets turned on in those files and whether that is desirable or not.

Non-nil buffer-file-name is a precondition for having diff-hl-mode enabled from the globalized mode. diff-hl-changes also returns nil if buffer-file-name is nil.

So I think we're safe here.