I intend to implement a proper file-handler for visiting blobs, so I am well aware that the current solution in Magit is a huge hack, but this is how a blob is currently being visited. The following part causes diff-hl-mode to be turned on:

          (let ((buffer-file-name magit-buffer-file-name))
            (normal-mode t))

For now I have changed this to:

          (let ((buffer-file-name magit-buffer-file-name)
                (after-change-major-mode-hook
                 (remq 'global-diff-hl-mode-enable-in-buffers
                       after-change-major-mode-hook)))
            (normal-mode t))

Any suggestions? (I don't intend to make any risky changes until I actually implement the blob file-handler.)