MetadataLoadContext leaves .dll on disk locked until application exit

Cannot replace the dll that was just looked at by the mod checker. 
To repro: run the mod checker and leave it at "press any key", try to replace the .dll.

I tried to play with scope and GC which didn't help. Not feeling like digging deeper, sorry.