Our use case is having large multi-platform binaries (e.g. OpenSSL, WebRTC etc.), especially debug builds with debug symbols. We have multiple versions of those builds: v1, v2, v3... v100 - that quickly accumulates to gigabytes. If we put them into a git repo, SPM has to download it in its entirety (all 100 versions are inside the git clone directory) even though we just need a single version.
https has no authentication with SPM by default, so the repository has to become public;
if used without authentication GitHub sometimes limits the number of downloads, and produce errors (due to API limits);
Having SSH support solves both issues, and also reuses the same authentication mechanism as the main source code repo, which makes it easy to use: set up SSH key auth once and get the project with all the binary deps via Xcode (not having to pay for what we don't need, because it only downloads binaries that are needed).