Internal changes detection does not work as expected

[malfet]

🐛 Describe the bug

See #110140 / D49673994 <- there is one more config file changed in the diff, than in PR, but https://github.com/pytorch/pytorch/pull/110140/checks?check_run_id=17194075737 reports success

Versions

CI

cc @seemethere @pytorch/pytorch-dev-infra