Was due to Edge not supporting AV1 by default.
A URL of a video with this issue is at https://inv.zzls.xyz/watch?v=B-nluiQ1Hew. It still happens when having it not show annotations by default and with autoplay disabled. The error shown is "Playback cannot continue. No available working or supported playlists". At https://yewtu.be/watch?v=B-nluiQ1Hew, I get "The media could not be loaded, either because the server or network failed or because the format is not supported".

Microsoft Edge
Version 112.0.1722.48 (Official build) (64-bit)