SingleLevelLoading animated layers with multiLevelLoads set not deleting higher levels.

[mosh1]

In develop branch.

Setting an animated layer with singleLevelLoading and multiLevelLoads set as below, the higher levels (in this cause targetLevel-2) don't get cleared after the target level loads. The result is two blended layers animating.

layer.singleLevelLoading = true
layer.multiLevelLoads = [-2]

[mousebird]

This is the on the globe, right?

[mosh1]

I see this with the 2D map. Haven't tested it on the globe.

[mousebird]

That makes sense. I'm not sure how single level loading works on the globe at the moment. I suspect it could be made to work for some cases fairly easily.

But I'll fix this in the map right now.

[mousebird]

Okay, I see what the problem is. This is a general overlap problem with multi-level loading. It works fine if the layer is opaque because you never see the lower level tiles beneath. It works less well for transparency.

I've been meaning to fix this for a while. I should have something in the next day or two.

[mousebird]

I think I might have it. Give it a try and let me know what you see.

[mosh1]

This seems to work much better. The only issue I see is that the higher level can take a while to clear, even when (visually at least) the lower levels seem to have loaded. I tested this with:

layer.multiLevelLoads = [0, -4]

Since each layer is animated they can take a while to load. How do you determine when to clear the higher level? Is it when the area covered by the higher level has a fully loaded level below it (but can encompass more than the displayed tiles on screen since higher level tiles are larger), or when enough of the tiles that are displayed below the level are loaded? Sorry if this isn't clear, can try to clarify if it's confusing.

[mousebird]

I was testing with the single frame loading, since it's just easier to test.

For multi-frame loading I'm going to guess that it will hold in that lower level until all the higher levels are fully loaded. In rare cases it may wait for everything to fully settle before it removes the lower level tile. It's smart enough to not consider off-screen tiles now.

I did paper over a loading problem by kicking off a reset when everything has settled. This fixed the problem, but it would only do so after everything has set. If you send me a video I can probably guess if that's happening.

[mosh1]

By a reset when everything is settled, is this reset done in the
// Note: Huge hack part of QuadDisplayLayer.mm?

I did a breakpoint there and it's repeatedly getting called.

Testing with layer.multiLevelLoads = [-3].

Things do seem to work well enough. Just wondering if that 'huge hack section' shouldn't be called frequently and it should be working better.

[mousebird]

It should always be called and it's benign. It's the fact that it has to be called that is a hack.

The QuadDisplay is complex state logic and ideally each tile should update itself properly on a load or unload, or viewpoint movement. However, something is getting out of sync and the easiest way to fix it is to flush the whole thing and start again. That happens every time the user moves anyway, so it's not expensive.

That comment is a reminder to myself that something has gotten out of sync and I really should track it down the next time I open up the hood and change something.

Performance-wise it should be fine. The only drawback is that a few of the lower res tiles may stay in longer than I'd like.

[mosh1]

I think I found a bug and might be explaining what I thought I was seeing earlier. If I just pan around it seems to do well and clears the lower res tiles. However if I zoom in, it doesn't seem to ever remove the lower resolution levels that were loaded before the zoom.

Video: https://dl.dropboxusercontent.com/u/1086233/MultiLevel%20loading%20issue.mov

Testing with layer.multiLevelLoads = [-3].

[mousebird]

Yeah, that's what that looks like. This is using the update I put in a few days ago?

[mosh1]

Yes, I updated latest develop branch from last night and still see it. I find it does eventually clear the lower res layer but only after a very long time (30s to a minute after when it should).

[mousebird]

Okay, I'll see if I can duplicate it.

[mosh1]

Were you able to duplicate it?

[mousebird]

Not yet, but I'd decided I should be fixing the root cause.
I'm a bit slow this week, but I'll get to this as soon as I can.

[mousebird]

Any chance you could try to duplicate it with the ComponentTester app? I have had no success.

Use the Quad Test Layer option.

[mosh1]

Yes, use the forecast.io test that uses MaplyMultiplexTileSource with a blank 2d map. Then zoom in and you'll see the top levels look like they hang around much longer than necessary. Since the grayscale tiles are too hard to see the bug you can use my radar tiling server. Here's the relevant section I modified in TestViewController.mm:

            } else if (![layerName compare:kMaplyTestForecastIO])
            {
                // Collect up the various precipitation sources
                NSMutableArray *tileSources = [NSMutableArray array];
//                for (unsigned int ii=0;ii<5;ii++)
//                {
//                    MaplyRemoteTileInfo *precipTileSource =
//                    [[MaplyRemoteTileInfo alloc]
//                     initWithBaseURL:[NSString stringWithFormat:@"http://a.tiles.mapbox.com/v3/mousebird.precip-example-layer/",ii] ext:@"png" minZoom:0 maxZoom:6];
//                    precipTileSource.cacheDir = [NSString stringWithFormat:@"%@/forecast_io_weather_layer%d/",cacheDir,ii];
//                    [tileSources addObject:precipTileSource];
//                }
//                
                // Use radar tiles
                for (int ii=50; ii >= 0;ii=ii-10)
                {
                    NSString *url;
                    if (ii == 0) {
                        url = @"http://test.radiantlabs.io/nexrad-n0q/";
                    } else {
                        url = [NSString stringWithFormat:@"http://test.radiantlabs.io/nexrad-n0q-m%02dm/",ii];
                    }

                    MaplyRemoteTileInfo *precipTileSource =
                    [[MaplyRemoteTileInfo alloc]
                     initWithBaseURL:url ext:@"png" minZoom:0 maxZoom:18];
                    precipTileSource.cacheDir = [NSString stringWithFormat:@"%@/weather_layer%d/",cacheDir,ii];
                    precipTileSource.cachedFileLifetime = 240;
                    [tileSources addObject:precipTileSource];
                }

                MaplyMultiplexTileSource *precipTileSource = [[MaplyMultiplexTileSource alloc] initWithSources:tileSources];
                // Create a precipitation layer that animates
                MaplyQuadImageTilesLayer *precipLayer = [[MaplyQuadImageTilesLayer alloc] initWithCoordSystem:precipTileSource.coordSys tileSource:precipTileSource];
                precipLayer.imageDepth = (int)[tileSources count];
                precipLayer.animationPeriod = 2.5;
                precipLayer.allowFrameLoading = false;
                precipLayer.singleLevelLoading = true;
                precipLayer.multiLevelLoads = @[@(-2)];
//                precipLayer.imageFormat = MaplyImageUByteRed;
                precipLayer.numSimultaneousFetches = 4;
                precipLayer.handleEdges = false;
                precipLayer.coverPoles = false;
//                precipLayer.shaderProgramName = [WeatherShader setupWeatherShader:baseViewC];
//                precipLayer.fade = 0.5;
//                [self quadImageFadeTest:precipLayer];
                [baseViewC addLayer:precipLayer];
                layer = precipLayer;
                ovlLayers[layerName] = layer;
            } else if (![layerName compare:kMaplyTestMapboxStreets])

[mousebird]

I've been setting up the test tile layer to try and duplicate the problem. I sort of did, but not in the way I expected.

Are you setting maxTiles to anything? If so, is it possible you're exceeding maxTiles? Try setting it to something large, like 512 and see if that helps.

Unfortunately it's counting phantom tiles, so it doesn't mean all that much in this case.

[mosh1]

I had set it to 512 already - just set it to 2048 and was able to replicate the issue.

[mousebird]

Is it possible that a tile didn't load in your example? Do you optimize out completely blank images perhaps?

I've modified my test layer to try and replicate what yours is doing, but with my ID numbers. It's hard to debug without them. The main difference is that it calls the async version of the fetch, which means things will return in random order.

Try the new "Quad Test Layer - Animated" in the ComponentTester app. See if you can replicate it there.

[mosh1]

I can't replicate it in "Quad Test Layer - Animated", even with a large random delay. So I'll close this bug unless I can get more info that it definitely points to an engine bug.

I'm not optimizing out blank images yet (but plan to do that at some point) in my test, so all tiles should be loading in my example. Perhaps some tiles aren't loading for another reason, I'll have to debug that. It's strange because it works appropriately if I pan around but not when I zoom in. Thanks!

[mousebird]

Oh, I'm willing to believe it's a bug on my side.
Try logging on a tile load failure and see if that's happening. That's my best guess at this point.

[mosh1]

Logged load failures and don't see any failures. Forced a few failures just to make sure logging is working correctly. Also when singleLevelLoading is disabled and all levels are loaded down, it works perfectly so if there were failed tiles causing it to stall then I'd assume we'd see the bug there too. Still investigating on why it works well on the "Quad Test Layer - Animated".

[mosh1]

I replicated the issue in "Quad Test Layer - Animated". I made a change in MaplyAnimationTestTileSource.m to increase the tile delay to better simulate slow network loads as well as see what's going on.

-                       usleep(drand48()* 0.115 * 1e6);
+                       usleep(drand48()* 3.0 * 1e6);

Video: https://dl.dropboxusercontent.com/u/1086233/Multilevel%20Loading%20Bug%20with%20Quad%20Animation%20Test.mov

In the video I zoom in from level 4. Level 5 loads in and then target level 8 loads. When level 5 is fully loaded, the level 4 tile hangs around until target level 8 is done loading. I do this a second time in the video and the behavior is consistent. I would expect level 4 to be cleared as soon as level 5 is loaded.

With more animation tiles and slower network the effect is more pronounced but this video demonstrates the issue.

[mousebird]

If I understand what I'm seeing, this is showing my fix in action.

At the end, the right tiles are loaded. So while it's true that the level 4 tile is hanging around too long, it's still going away at the end, right?

I thought what you had was a case where the low res tile never went away. That's slightly different.

Can you live with a tile sticking around too long sometimes? If not, I could add a periodic reset to catch things up every second or so.

[mosh1]

Sometimes the low res tile never goes away, but that's not reproducible all the time.

I would still want the level 4 tile to go away as soon as anything deeper is fully loaded. It hangs around much longer in my implementation than the example video because there are many frames in my tiles as well as being loaded over the network. Additionally my specific animation makes it much more jarring visually than the test case.

[mousebird]

Okay, I'll dig deeper into this next week.

If you can reproduce a case where the tile never goes away, that would be good. That's a different sort of bug, and a much more serious one than the tile taking a while to go away.

[mousebird]

So I tracked down the sticky tile problem and, I hope, fixed it. There should no longer be tiles sticking around longer than they ought.

That might be different from the permanent tile overlap problem. The sticky tile issue shouldn't cause a permanent problem. So we might have two different bugs here.

In any case, give the new source tree a try and let me know if you see any problems.

[mosh1]

Unfortunately I see the same issue.

Video: http://cl.ly/3b3j3W0w0k2b

You can see all the higher levels sticking around (three layers visible not including target level) until all the target levels are loaded.

To reproduce I modified your zoomTest in TestViewController.mm for a bit quicker zoom in. Changed MaxDelay in MaplyAnimationTestTileSource.m to:

static const float MaxDelay = 3.0;

// Test sequence for zoom
- (void)zoomTest
{
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)((2.0) * NSEC_PER_SEC)), dispatch_get_main_queue(),
                   ^{
                       [mapViewC setHeight:mapViewC.height/2.0];
                   });
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)((2.0+1.0) * NSEC_PER_SEC)), dispatch_get_main_queue(),
                   ^{
                       [mapViewC setHeight:mapViewC.height/2.0];
                   });
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)((2.0+2.0) * NSEC_PER_SEC)), dispatch_get_main_queue(),
                   ^{
                       [mapViewC setHeight:mapViewC.height/2.0];
                   });
//    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)((2.0+3.0) * NSEC_PER_SEC)), dispatch_get_main_queue(),
//                   ^{
//                       [mapViewC setHeight:mapViewC.height*2.0];
//                   });
}

To make the issue easier to see I changed the alpha of the test files from 0.5 to 0.2:

    if (_transparentMode)
    {
        backColor = [UIColor colorWithRed:0.0 green:0.0 blue:0.0 alpha:0.2];
        fillColor = [UIColor colorWithRed:red green:green blue:blue alpha:0.2];
    } else {
...

[mousebird]

This one seems to fix the problems you turned up. Give it a try and let me know what you see.

[mosh1]

Yep, that fixed it. SingleLevelLoading now works as expected with higher levels not staying around.

I can't seem to reliably reproduce the issue where they never disappear at all - I suspect that's an actual network load issue. So will close this one. Thanks!

[mousebird]

The tile loading logic is more of a journey than a destination. So let me know if it turns up again.