When zooming in or out, Modest Maps overlays multiple layers. When they are translucent, this results in an odd effect as shown in http://i.kkaefer.com/transparent_overlap-20110616-203545.png. Modest Maps should crop the tile so that only the most appropriate layer is visible.
To do this effectively we would need to change the way that "parent" tiles are shown when the "ideal" tile isn't available. Right now you're seeing artefacts because we simply scale up the tile to cover all four "child" tiles (or 16, or 64, etc.). Another way to do this would be to create temporary tiles at the ideal zoom level using a div and offset the best available image (or use a background image) scaled up to compensate for the missing region.
It's not a simple matter of cropping because as ideal tiles come in the old tiles need selectively hiding. This would mean punching holes in the old tile, not possible in a cross browser way without re-architecting as I describe. Could be interesting to re-think Modest Maps as only displaying ideal tiles and using whatever images are available in the cache to patch up the best 256px representation. Some clever use of divs and overflow:hidden, or some clever use of CSS background images should do it. Volunteers?
I thought about this feature too. In theory, it can be done using backgrounds on tile images with CSS3 background-size property (to scale tiles) and multiple backgrounds (for zoom-out), though it seems a lot of hassle reachitecting zooming code for quite a small benefit.