When a background (or manual) fetch happens, the git graph is refreshed.
During this refresh, your selected commit stays selected, meaning the commit information panel should not update. However it refreshes as well, even when the selected commit hash hasn't changed.
in general this isn't an issue. But when you are browsing the changes, your currently selected file and position within that file both are reset to the beginning (top of first file). This is very annoying and unexpected.
I saw another issue about remembering the currently selected file when opening a commit through file history. This is obviously not the same, but they might (or not) have a similar solution, hence the reference.
If there is a technical blockage for this feature, fine by me. But then at least it is documented.